GRunT - Easier run-time scripting

[Gavino]

Avisynth's run-time environment is very powerful, supporting complex video processing that would be difficult or impossible to perform in a normal script. But it's not easy to use - the behaviour of run-time scripts (especially in combination) can be hard to understand and there are usability problems concerning scope and lifetime of variables.

"Cond. filters are quite complex - and sometimes I am even surprised of the outcome" - sh0dan

GRunT (Gavino's Run-Time ) is a plugin which addresses these and other problems, making the run-time system much easier to use.

Features:
- Simple, natural and robust way to pass variables into a run-time script from 'outside'
- A run-time script can be evaluated in its own independent scope
- Run-time functions can be called from a user function
- Run-time functions can be applied to any frame (relative to the current one)
- Additional variant of ConditionalFilter with single boolean expression
- Fixes a fairly serious bug in the run-time system
- Lightweight plugin extending the standard run-time environment, minimal time and memory overhead
- 100% backwards compatible with existing scripts

GRunT should be useful to anyone who uses the run-time filters, from those who make occasional use of ScriptClip to those who write complex functions based on run-time features (such as Restore24 or MRestore).

Details

The plugin provides extended versions of the following run-time filters:
- ScriptClip
- FrameEvaluate
- ConditionalFilter
- WriteFile
- WriteFileIf

The alternative names GScriptClip, GFrameEvaluate, GConditionalFilter, GWriteFile and GWriteFileIf may also be used. However, if running on version 2.57 or earlier of Avisynth, then only the alternative names may be used.
(This restriction is necessary for technical reasons - sorry about that.)

Each filter is 100% backwards compatible with its standard equivalent, but has two additional optional arguments:

• string args: the variables whose values are to be imported into the run-time script, written as a list of names separated by commas. The given variable names are evaluated in the current (compile-time) context, so can include function parameters or local variables.
  Each value becomes the initial value of the corresponding variable at each invocation of the run-time script.
• bool local: if true, the filter will evaluate its run-time script in a new variable scope, avoiding unintended sharing of variables between run-time scripts.
  Default is true if args is also specified, otherwise false (to preserve backwards compatibility).

A short example (based on the original) shows how this greatly simplifies passing function parameters into a run-time script.

function bracket_luma(clip c, float th1, float th2) {
  Assert(0 <= th1 && th1 < th2 && th2 <= 255, "Invalid thresholds!")
  ScriptClip(c, """
    avl = AverageLuma()
    avl <= th1 ? last.BlankClip() : avl >= th2 ? last.BlankClip(color=color_white) : last
  """, args="th1,th2", local=true)
}

This is much easier than the standard approach of dynamically building the runtime script using string concatenation, or passing the values via global variables.

"I really do not like this global variable business with ScriptClip ..." - stickboy

And because the run-time script is evaluated in its own scope, there is now no problem in calling bracket_luma more than once in the same script (previously the variables th1 and th2 of different instances could interfere with each other).

Elements of the args string can also take the form 'name=expression' - the expression is evaluated in the current context and is used to set the value of the named variable in the run-time script.
Example: args="x, y=n+1, c=c.Trim(2, 0)" will provide values for the variables x, y and c.
Here y need not even exist in the current environment (although x, n and c must).

The plugin also provides the following extensions to the run-time functions (eg AverageLuma):

• these functions can now be called inside a user function, when the user function is called from a run-time script
• each function has an new optional int argument, which can be used to get the value from another frame, relative to the current one. For example, AverageLuma(-1) returns the value for the previous frame. (No more assigning to current_frame ...)

For added convenience, there is a new variant of ConditionalFilter which takes a single boolean expression instead of three separate parameters as at present. This is useful when the condition to be tested is a compound one or is already available in boolean form. For example,

ConditionalFilter(c, c1, c2, \
 "AverageLuma(c1) > AverageLuma() && AverageLuma(c1) > AverageLuma(c2)")

where previously you would have to add (...,"=", "true")

Finally, the plugin fixes a fairly serious bug I discovered in the run-time system.
This fix is needed if you are running a version of Avisynth prior to build 080620 of 2.58.

More detailed documentation and examples are provided in the attached download.

EDIT 27-SEP-08: Updated to v1.0.1 to work with Avisynth 2.5.7

Comments and suggestions would still be very welcome.

[martino]

Oh my God! I love you!!! Finally I can do something on which I wasted 2 days trying to achieve (and countless headaches)... I think... xD

[Gavino]

Thanks, martino! I hope you find it solves your problem.

If it does, could you post your resulting script here as an example?
Alternatively, if you still have problems, explain what you're trying to do and I'll see if I can help.

[backtohell]

i am not an expert in avisynth but still will try your GRt

Thanks Gavino for sharing

[Gavino]

Previously, I wrote:

Originally Posted by Gavino

For added convenience, there is a new variant of ConditionalFilter which takes a single boolean expression instead of three separate parameters as at present. This is useful when the condition to be tested is a compound one or is already available in boolean form.

I have now noticed that the standard ConditionalFilter, as well as forcing you to use 3 parameters, does not even support all the boolean operators - you can only use "=", "<" or ">", and not "<=", ">=" or "!=". Of course, you can express ">=" by using "<" and switching round the 'then' and the 'else' clips. But GRunT's version (as well as being slightly shorter) allows you to express the condition in the way you choose, whichever is more convenient/natural to you.

[MOmonster]

@Gavino
I don´t get ScriptClip running.
I get allways the error:

ScriptClip does not have the named argument "show"

I tested your sample scripts and some of my own, but allways the same error (no error without your plugin).
What is wrong?

[Gavino]

@MOmonster: I'm sorry about this - it is a problem with GRunT and Avisynth v2.5.7 that I am still looking into.

At the moment, the workaround is to use v2.5.8, where it works fine.

[MOmonster]

Thanks for the fast answer.
I will do some testing when I adapt srestore to avisynth 2.5.8.

Edit:
Another question. Is it possible to shift values without using global values?

[Gavino]

Originally Posted by MOmonster

Is it possible to shift values without using global values?

What do you mean by 'shift values'?
Can you give me an example?

[MOmonster]

I use global vars for performance reasons.
Instead of something like this:

ScriptClip(last, """
prev = AverageLuma(last.loop(2,0,0))
curr = AverageLuma(last)
next = AverageLuma(last.trim(1,0))
...outputclip...
""")

I can write this:

ScriptClip(last, """
global prev = curr
global curr = next
global next = AverageLuma(last.trim(1,0))
...outputclip...
""")

[Gavino]

OK, I should have realised you meant that, having seen similar code in MRestore.

The variables in your example don't need to be global, as they are at the outer script-level scope (and hence persistent). But of course, if you need to see them in functions called from within the ScriptClip, they would have to be global (or passed as parameters).

If you are using GRunT, the example could also be written as

ScriptClip(last, """
prev = AverageLuma(-1)
curr = AverageLuma()
next = AverageLuma(1)
...outputclip...
""")

which would be slightly faster than the standard ScriptClip, though clearly still slower than your optimisation.
(As you know, the disadvantage of the optimised version is that it does not support seeking, since it relies on linear access.)

[MOmonster]

The variables don´t have to be global? I don`t know anymore why I think so.
The seeking problem can be solved with RequestLinear and some additional conditions, so speed is more important for me.

Thanks.

[Gavino]

Originally Posted by MOmonster

The variables don´t have to be global? I don`t know anymore why I think so.

Perhaps you started out thinking (long ago) that each ScriptClip runs in a separate variable scope.

They don't - that's why you can get unexpected interference between different ScriptClip instances, and why GRunT provides the local=true option.

[Didée]

Originally Posted by MOmonster

The variables don´t have to be global? I don`t know anymore why I think so.

Well, I can remember why ... iirc it was Restore24 which introduced this kind of variable-shifting in the conditional env.

In the 1st example with 3times AverageLuma, globals are not needed because all 3 variables are computed "now".
In the 2nd example, the globals are needed because only one variable is computed "now" (the "next" one), and the other two are derived from what the conditional environment had computed during processing the previous frame. If you need "now" the content of a variable from one frame before, then you're jumping out of ScriptClip's local context, and need a global variable to "save" that variable's content while stepping from one frame on to the next one.

The variant with independent calculation of all needed variables is of course slower (doing AverageLuma several times), and it's robust against seeking. The variant with globals is faster (only one time AverageLuma), but it's not robust against seeking and needs linear access.

When I did Restore24, the choice was easy:

a) RequstLinear() did not exist yet

b) GRunT did not exist yet

c) R24 uses not only the 3 variables 'previous', 'current', 'next'. Because of its kind of pattern reckognition, it uses six (or seven?) of such variables: previous(3), previous(2), previous(1), current, next(1), next(2). (Not sure anymore if the released versions used next(3), too.)

It's obvious why the variant with globals was chosen. The choices were to do only one time AverageLuma for every frame, or to do six (seven?) times AverageLuma for every single frame (with all except one of them being basically superfluous.)
So I went for the route that requires linear access, but does not do 500% (600%) of re-doing frame sampling operations that already had been done.


It's been quite some time since then. Knowledge/practice and available tools have noticeably developed in the meantime. I'm really looking forward to see Mrest..., erh, SRestore to do right what I had b0rked back then.

(Lord, make I don't ever have to touch Restore24 again!)

[MOmonster]

Originally Posted by Gavino

Perhaps you started out thinking (long ago) that each ScriptClip runs in a separate variable scope.

They don't - that's why you can get unexpected interference between different ScriptClip instances, and why GRunT provides the local=true option.

Yes, this and the need to initialise the variables are the reasons.
This script indeed works well:

ScriptClip (last, """ cfr=current_frame
now = cfr<2 ? 0 : next
next = AverageLuma(last)
last.subtitle(string(now)).subtitle(string(next),y=16)""")

I will test GRunT soon. Maybe next srestore release won´t use global variables anymore.

@Didee
Yes I remember, first Cdeint releases use practical the same variables as restore24.

[Gavino]

Originally Posted by Didée

In the 2nd example, the globals are needed because only one variable is computed "now" (the "next" one), and the other two are derived from what the conditional environment had computed during processing the previous frame. If you need "now" the content of a variable from one frame before, then you're jumping out of ScriptClip's local context, and need a global variable to "save" that variable's content while stepping from one frame on to the next one.

No, as I pointed out, and as MOmonster has demonstrated, there is no 'local' context in (the builtin) ScriptClip - it's all at outer script level.

[Didée]

Its probably related to the fact that R24 uses user defined functions within scriptclip -- Like

function ShiftBackVars() { do_the_shifting_stuff }
...
ScriptClip( c, "ShiftBackVars()" )

When referring to conditional variables in the do_the_shifting_stuff section, they have to be global, don't they?
(Sorry, I'm currently not in the mental "flow" of ScriptClip & Co. ... usually need some run-up when I have to.)

If so, well ... I would NEVER have done R24 without that style of using (lots of) defined functions, called from within the conditional envoronment. That way it was easily possible to get a reasonable level of "modularisation". When having to put all that stuff within one gigantic

ScriptClip(c, """
...
put several hundred lines of code here ...
... 
""" )

then Restore24 never would have come into existance. (Which would be a pity ... think the butterfly effect. )

Howeveritmightbe. The important point is you guys now are heading to do it right, without me needing to get knots in the brain.

[Gavino]

Originally Posted by Didée

When referring to conditional variables in the do_the_shifting_stuff section, they have to be global, don't they?

Yes, to be visible in a function, they do need to be global, as I mentioned above.
And if you're writing to them, you don't have the option of passing them as parameters.

I would NEVER have done R24 without that style of using (lots of) defined functions, called from within the conditional envoronment. That way it was easily possible to get a reasonable level of "modularisation".

GRunT helps in that respect because it allows calling of the run-time functions (such as AverageLuma) from within a user function.

[Gavino]

Version 1.0.1: Bug fix for Avisynth 2.5.7

As MOmonster reported above (#6), and also as reported here, there are problems using GRunT with Avisynth v2.5.7. In fact, I'm embarrassed to say that it just doesn't work on anything prior to v2.5.8.

I have produced a new version (GRunT 1.0.1) that solves the problem. The fix requires users running on 2.5.7 (and earlier) to use alternative names for the run-time filters (GScriptClip instead of ScriptClip and so on). The run-time functions, such as AverageLuma, are not affected.

Users already on 2.5.8 can continue to use the old names - the alternative names may also be used if preferred.

Go to the first post in this thread to download the new version.
Updated documentation, reflecting this change and minor editing, is also included.

[MOmonster]

Thanks for update, but I can´t download it.