AviSynth+ thread Vol.2

[pinterf]

Originally Posted by wonkey_monkey

This expr behaves unexpectedly (at least to me, I might be missing something):

expr("sxr 0 < 128 192 M^ 255 ?")

"192 M^" should amount to a no-op as far as the stack is concerned, but instead of returning a white clip (which is what happens if you take out "192 M^"), a black clip is returned (it's returning the value of "sxr 0 <", as if the ternary operator and subsequent stack elements have been completely removed).

I think the problem might lie with constant folding and findBranches.

Thanks, good catch, fixed, see test8

[kedautinh12]

Thanks

[wonkey_monkey]

Excellent, thanks pinterf!

[FranceBB]

Well done, thanks!

[Dogway]

Thanks for the update!

Is a sort operator possible? I found myself the last days fighting with implementing some median algos (removegrain modes 2-4). I know I can't compete in speed but for completion in ExTools this is probably the last function I'm adding.

Tired to do permutations to no avail I searched some maths and found Sorting Networks, this is also proving hard to achieve with current Expr in-string tools.

Pseudo-code for 8 inputs (swap here means variable swap, 0 as nop sub)

A C < A C swap 0 ?
B D < B D swap 0 ?
E G < E G swap 0 ?
F H < F H swap 0 ?
A E < A E swap 0 ?
B F < B F swap 0 ?
C G < C G swap 0 ?
D H < D H swap 0 ?
A B < A B swap 0 ?
C D < C D swap 0 ?
E F < E F swap 0 ?
G H < G H swap 0 ?
C E < C E swap 0 ?
D F < D F swap 0 ?
B E < B E swap 0 ?
D G < D G swap 0 ?
B C < B C swap 0 ?
D E < D E swap 0 ?
F G < F G swap 0 ?

I thought on assigning new variables but Expr falls short with only A to Z. Maybe possible add AA, AB...?

I also found myself occasionally wanting to do the next:

"x y * sqrt A^ A[-1,0] B^ A[0,0] C^ ...."

Not asking, just thinking out loud, feel free to implement what's easy/useful.

[pinterf]

Originally Posted by Dogway

Thanks for the update!

Is a sort operator possible? I found myself the last days fighting with implementing some median algos (removegrain modes 2-4). I know I can't compete in speed but for completion in ExTools this is probably the last function I'm adding.

Tired to do permutations to no avail I searched some maths and found Sorting Networks, this is also proving hard to achieve with current Expr in-string tools.

Pseudo-code for 8 inputs (swap here means variable swap, 0 as nop sub)

A C < A C swap 0 ?
B D < B D swap 0 ?
E G < E G swap 0 ?
F H < F H swap 0 ?
A E < A E swap 0 ?
B F < B F swap 0 ?
C G < C G swap 0 ?
D H < D H swap 0 ?
A B < A B swap 0 ?
C D < C D swap 0 ?
E F < E F swap 0 ?
G H < G H swap 0 ?
C E < C E swap 0 ?
D F < D F swap 0 ?
B E < B E swap 0 ?
D G < D G swap 0 ?
B C < B C swap 0 ?
D E < D E swap 0 ?
F G < F G swap 0 ?

I thought on assigning new variables but Expr falls short with only A to Z. Maybe possible add AA, AB...?

I also found myself occasionally wanting to do the next:

"x y * sqrt A^ A[-1,0] B^ A[0,0] C^ ...."

Not asking, just thinking out loud, feel free to implement what's easy/useful.

Getting median is a bit easier for limited number of entries e.g. 3 or 5, and needs no explicite sorting, like is done here.
For example here:
Median of 3
https://github.com/pinterf/MedianBlu...r_sse2.cpp#L81
Median of 5
https://github.com/pinterf/MedianBlu..._sse2.cpp#L155

For these magnitudes (3, 5) they are not even complex ones and can be written specifically.

For larger radius we probably need a generic implementation with sorting but then Expr would starting to become an enormously complex filter (embedded MedianBlur ) rather than an expression evaluator.

Generic sorting algorithms are available for avx2 and avx512, (search for "avx2 avx512 simd sorting"); I can say that there are really nice solutions, but they are beyond my mental limits .

[pinterf]

Originally Posted by Dogway

I thought on assigning new variables but Expr falls short with only A to Z. Maybe possible add AA, AB...?

At the moment I have exactly 26 memory slots for them (A..Z), assigning letters to slots means one-to-one relation at the moment.

[Myrsloik]

Originally Posted by Dogway

Thanks for the update!

Is a sort operator possible? ...

Why are you not using real compilers at this point? WHY?

I mean even modern javascript engines will probably run better with that number of variables and have cleaner syntax.

Obviously the solution is to implement a secondary stack where you can push and pop overflow values. As a solution to the number of addressable input clips use stackvertical to get around it.

WE DON'T NEED NO STINKING COMPILERS!

[wonkey_monkey]

Originally Posted by Dogway

Thanks for the update!

Is a sort operator possible?

Pseudo-code for 8 inputs (swap here means variable swap, 0 as nop sub)

A C < A C swap 0 ?

The thing to remember about the ternary operator ("?") is that everything prior to that is already evaluated, so you've already swapped the variables and put 0 on the stack. In fact, what you're evaluating there is the value of C, because the stack currently contains:

(A C <) - 0 or 1
C
A
0

You could do the following:

A C < A C ? A C < C A ?

This will put either A C or C A on the stack depending on the comparison.

This isn't great because of the two comparisons. You could always store the first result into a variable:

A C < R@ A C ? R C A ?

Or you could be a bit cleverer and do:

 A C < A C ? dup A C + -

Then again it's probably most efficient just to do

A C min A C max

Or you might be able to do some swaps and dups to avoid loading the variables twice (I'm not sure how efficiently Expr recalls them).

Thank you for listening to my TED Talk.

[Dogway]

wonkey_monkey: nice TED hehe. I was going with the new var assignment starting from Z, and then thought what a sudoku I would end playing so asked for AA kind of vars. I will go with the most efficient method, ternaries are costly so min max probably. I'm not sure it might work but will test. Hopefully I can build a nice median and repair library as I did with ex_edge().

Myrsloik: It's just too much for me... and my 2013 SSD. I'm art oriented so my knowledge and disk space goes to DCC (3D, compo, video) and audio software mostly (no games). I fear that getting into nitty gritty programming can derail me too much, or simply make either my brain or SSD explode. I only do basic scripting in AVS, AHK and Python, some regex in there too and very basic GLSL. Will look forward how Expr evolves.

[pinterf]

New version
Avisynth+ 3.7.1 test build 9 (20210706)

- Expr: allow arbitrary variable names (instead of A..Z), up to 256 variables can be used. Do not use existing keywords.
  Variable names must start with '_' or alpha, continued with '_' or alphanumeric characters.

[kedautinh12]

Thanks

[FranceBB]

I just replaced the files on my installation and there's already a new version of AVS+.
You never take a break, Ferenc, thanks!!

[feisty2]

Originally Posted by Dogway

Myrsloik: It's just too much for me... and my 2013 SSD. I'm art oriented so my knowledge and disk space goes to DCC (3D, compo, video) and audio software mostly (no games). I fear that getting into nitty gritty programming can derail me too much, or simply make either my brain or SSD explode. I only do basic scripting in AVS, AHK and Python, some regex in there too and very basic GLSL. Will look forward how Expr evolves.

it is a common misconception to assume that writing a proper C++ plugin requires more work than dirty hacks like feature creep Expr.

to describe a 3x3 gauss blur filter, you have in Expr: "x[-1, -1] x[-1, 0] 2 * + x[-1, 1] + x[0, -1] 2 * + x 4 * + x[0, 1] 2 * + x[1, -1] + x[1, 0] 2 * + x[1, 1] + 16 /"

meanwhile to describe the same thing in C++: https://github.com/IFeelBloated/vapo...ssBlur.hxx#L22

you tell me which one is easier to write and to understand.

a temporal median filter that works for any temporal radius is also no more than a few lines in C++. but in this case, you can't even describe it (by ab)using Expr.

it's year 2021 and things have changed, drastically. writing a C++ plugin is scripting (given that it's a simple filter like 3x3 conv or temporal median), and C++ is a scripting language, if you will it to.

[StainlessS]

As Feisty says, some things are actually easier in C/CPP, and Doggy would make for a good coder. [So would Real.Finder].
The only real difficult bit about learning C is figuring out where to put all them damn semi-colons, after that its no more
difficult than AVS script.

[feisty2]

davidh*****, I wasn't replying to you. I was replying to Dogway continuing on Myrsloik's point. I suggest that you do not cut in on a conversion not centered around you, and stop acting like you own the place or something. It's a public forum, you do not have the right to stop me from talking just because you don't agree with my opinions.

[feisty2]

now back to your points

Oh, feisty, please give it a rest with the moaning. No-one's ever going to force you to use Expr. Clearly some of us find it very useful, even those like me who can write fully-fledged C++ plugins when we need to. I just don't understand why you're so offended that we have a different tool to use when the occasion rises.

I am a frequent user of Expr and it would be ridiculous for you to think that I am "offended" by Expr. However, I use Expr for the right purpose, namely expression evaluation.
what I am concerned about is that, if this feature creep Expr gets widely adopted, people would start abusing Expr for things that should be written in a proper programming language (Dogway is already asking for the sort functionality). the consequence of this is that it creates a shit mountain of unreadable and unmaintainable code, because: a) RPN itself is a rather "exotic" language, hard to decipher by human. b) Expr, even with all that feature creep, is still nowhere as expressive as a real programming language, so it is very likely that sometimes there's no direct way to describe what you want using Expr. and people would then "invent" weird tricks to do what they want indirectly, and those "tricks" can be extremely confusing to the reader of the code.

[feisty2]

Expr makes customised filtering incredibly quick and easy. I have scripts where I have to do dozens of Expr-friendly operations. It's ridiculous to write dozens of C++ filters, or even one multi-choice filter, when a single line of Expr per use will do.
...
And then when you're testing, you need to close VirtualDub or AVSpmod every time you recompile.

a C++ filter takes arguments, you simply make the WIP part, something like a not yet determined convolution kernel, a parameter of the filter and you're free to fill in all kinds of different arguments in your script.

And it's disingenous to imply that it universally doesn't require more work to write a C++ plugin (and Expr is not a dirty hack).

At the very best, with a .cpp template already saved somewhere, you've still got to create your project, add your source code, link avsynth.lib, make sure avisynth.h is included, fiddle about with optimisation switches, make sure you've got the right architecture targetted, that you're not accidentally doing a Debug build, add a build event to copy the DLL to your plugins folder... the list goes on.

that's your overcomplicated workflow. for me it's just one command line: g++ -shared -std=c++2b -lstdc++ -static -O3 -flto -march=native -finline-limit=1000000000000000000000000000 -funroll-all-loops -funsafe-loop-optimizations -o Filter.dll EntryPoint.cxx vapoursynth.lib

[feisty2]

Since you ask, the Expr one, definitely on the first count - it's far less code and far easier to debug

is this supposed to be a joke? RPN is easy to debug? RPN expressions are incredibly "nonlocal", in ways that there's no parentheses to identify the local sections of a complex formula. you'd have to imagine a stack in your head, and emulate the evaluation process with your imaginary stack to determine the operands of an operator. good luck with modifying a local term of a multi-line long expression.

x[-1, -1] x[-1, 0] 2 * + x[-1, 1] + x[0, -1] 2 * + x 4 * + x[0, 1] 2 * + x[1, -1] + x[1, 0] 2 * + x[1, 1] + 16 /

is hardly "far less code" than

auto GaussKernel = [](auto Center) {
	auto WeightedSum = Center[-1][-1] + Center[-1][0] * 2 + Center[-1][1] +
		Center[0][-1] * 2 + Center[0][0] * 4 + Center[0][1] * 2 +
		Center[1][-1] + Center[1][0] * 2 + Center[1][1];
	return WeightedSum / 16;
};

[feisty2]

First you complain about feature creep, now you're complaining that it doesn't have enough features. Make your mind up! Expr is never going to do everything and that's not why it exists.

it is clear that you want Expr to do everything, don't lie.

Originally Posted by wonkey_monkey

pinterf,

I've been thinking long and hard about Expr for the last few days, because it's so much quicker than the RPN compiler I've been working on for a while. It does lack a number of features that I have found very useful, though - not that I'm asking you to do anything, Expr is excellent enough as it is and you've already been kind enough to implement a few things at my suggestion, but I just thought it might be worth going over my ideas in public in case of it's any interest to anyone.

As I use RPN for a lot of things, a standalone RPN compiler is something of a necessity for me. But before I go off in my own direction - taking plenty of inspiration from Expr, if not actual code; anything I eventually release will be sure to be open source though - I just wondered about your potential interest in such a project, as in whether it would be something that could be usefully developed in parallel with future iterations of Expr (and/or used to create new programmable features of Avisynth).

On the one hand it seems silly to duplicate so much effort, when Expr already shares so much in common with my aims. But on the other hand, what I'm envisioning would also be quite different:

• A standalone RPN compiler, which has a concept of input and output "planes" but only as generic data sources, with width, height, (and possibly additional dimensions), type (byte, short, int, float, double) and pitch. Bit depth conversions and colour channel bias would be left for the calling function to inline as boilerplate RPN ops (more or less what Expr already does, just abstracted one more level)
• The ability to do per-clip, per-frame, per-line, and per-pixel calculations (storing variables). Per-clip would probably just be constant folding since there'd be no changeable input. Per-line might be easy to implement directly in Expr - it's just a matter of moving the start of the loop and making sure the stack is empty at the start of the loop. Per-frame is trickier. Per-pixel only applies if the output channels are calculated in an interleaved way, which is natural for packed RGB(A) but not so natural for YUV(A) (you have to jump around between the output planes)
• Internal multithreading
• Choice of float or double precision (my current compiler uses the x87 stack which offers 80-bit precision - not available with SIMD unfortunately)
• Trig functions implemented with SIMD (based on ssemath or some other library), floor, ceil, bitwise operators and so on
• Automatic type switching between int and float
• Option to return black out-of-bounds pixels instead of repeating edges
• Something more akin to true conditional execution - difficult with SIMD but not impossible to emulate - rather than just the basic a/b swap ternary operator
• Internal multi-dimensional arrays - build your own LUT with per-clip/per-frame calculations!
• Run-time calculable pixel offsets
• Support for other planes (Y in U, G in Y, etc) to be referenced (if they're the same size). Allow non-matching input clips as long as their non-matching planes aren't referenced
• Temporal offsets to access past/future frames
• Inlined Avisynth variables
• Support for comments (a simple regex can remove several styles)

The last four I plan to implement first as an Expr wrapper plugin, as a separate/proof-of-concept project.

Hmm... you know, now that I write this out it does seem like a bit of a monster. Still, it would be just as helpful to hear if it's of no real interest to you - no offence will be taken! I'd also welcome your thoughts on the subject, or if anyone else has any feature requests or hints.

PS Thanks for everything you do!

I don't see anything wrong with my argument which suggests that you attempt to make Expr capable of everything (thus the feature creep) and your attempt failed.