Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 13th October 2013, 20:20   #41  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
OK everyone,
a running version is attached to 1st post. Don't blame me for any flaws please, work in progress.
martin53 is offline   Reply With Quote
Old 15th October 2013, 22:22   #42  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
M53, have changed 709 switch to > 600, is the better choice. RT_ updated, hope it works for you.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 16th October 2013, 12:01   #43  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
Hi martin53
Last version work properly. I see improvements for my VHS capture source.
I see some speed up.
yup.
yup is offline   Reply With Quote
Old 16th October 2013, 14:06   #44  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
from AWB
Code:
    # Calculate statistics from RGB clip
    # a) Average of Maximum for White Patch (global color evaluation)
    MRGBChanStats(cRGB, mask=cGamutMask, delta=delta, Chan=-1, flgs=$12, prefix="fRGB_", MaskMin=fGY_yMax-1)
    if (RT_VarExist("fRGB_Ave_0") && RT_VarExist("fRGB_Ave_1") && RT_VarExist("fRGB_Ave_2")) {
From later EDIT to MRgbChanStats, to make same as MYStats docs
Quote:
EDIT: Returns 0 if no pixels found in search area of mask within MaskMin and MaskMax.
Should use result of MRgbChanstats, testing for non 0 instead of RT_VarExist().

To be more specific, if no valid pixels found in masked area returns 0, the relevant channel pixel counts are set to 0
and no other global variables are altered in any way, ie will not be nullified nor deleted, if those variables already existed
before call, will remain in previous state.

Also note, I think we got crossed wires somewhere, I said I would alter RT_ 709 switch to be same as yours, based on height >= 600,
have now made that change in RT_Stats (also done but not released in RoboCrop, MYStats and one or two other places).
You have now changed yours to match my previous RT_stats, ie switched on width > 720.
ALL CHANGE PLEASE.
I shall keep same as you previously had it based on height >= 600, if its good enough for FireSledge (Cretindesalpes) its good
enough for me.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 16th October 2013, 21:13   #45  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
OK, fine with everything,
was confused a bit with the handling of variables and posted a question in the RT_Stats thread.
When existing variables are unaltered in the case of pixelcount==0, I need to set them to a neutral value in advance. Say I use chan=-1 then I can not know in advance whether all three channels will return data - although I'd think if one channel does, all channels should.
My observation was that when I initialize variables before I call RT_RgbChanStats, the function does not change them.
I observed some non-credibly low averages with real world footage, but was retarded with my tests by the variables thing.
Maybe I confused the system with local over global variables. Will check that.
However, the whole thing is a bit tricky to master, I think, for newbies, and maybe it should be redesigned a bit.

Will change back to height >= 600, no problem
martin53 is offline   Reply With Quote
Old 17th October 2013, 13:18   #46  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Will answer in RT_Stats thread. EDIT: Here:-http://forum.doom9.org/showthread.ph...05#post1648305
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 17th October 2013 at 13:50.
StainlessS is offline   Reply With Quote
Old 17th October 2013, 19:26   #47  |  Link
SamKook
Registered User
 
Join Date: Mar 2011
Posts: 216
Why not use width >= 900 to detect HD(which is what I personally use in some of my scripts)?
I don't have any concrete example in mind, but with some anamorphic 720p movies, I'm fairly certain that the height could drop below 600px, but the width should never drop much below 960px(4:3) for any HD sources and SD ones shouldn't be much over 854px(resized 16:9).

I mostly work with NTSC sources so maybe PAL ones would make my reasoning wrong though.
__________________
AMD Ryzen 9 5950X and EVGA RTX 3080 with G.skill 64Gb 3600 (2 16x2 kit) on Asrock X570 Taichi with Samsung 980 Pro 500Gb NVMe SSD running Win10 x64 on LG 34GN850-B 34.0" 3440 x 1440 160 Hz

Last edited by SamKook; 17th October 2013 at 19:29.
SamKook is offline   Reply With Quote
Old 11th November 2013, 07:00   #48  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
Hi martin53!

I try last script version. Source was 50Hz after QTGMC.
Code:
SetMemoryMax(1024)
global MeGUI_darx=4
global MeGUI_dary=3
AVISource("chapter13f.avi")#.Trim(0,10000)
AssumeTFF()
AutoGain(adjust_mode=1)
AWB()
unsharp(varY=310,varC=155,strength=0.15,U=3,V=3)
SeparateFields().SelectEvery(4,0,3).Weave().AssumeTFF()
AddBorders(16,12,16,12)
https://www.dropbox.com/s/i1tpg7o7wskks6s/encawb.mp4
From 1:30 min encoded only 28 min, ant at medium file changing color balance, see file at Dropbox.
May be problem related to Global variables?
yup.
yup is offline   Reply With Quote
Old 11th November 2013, 17:09   #49  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Only had time for a glimpse at your clip, but saw the sudden change at 8:39. Is the color after this time the original clip color? If so, the script is performing well up to this point, i feel
Before I answer your post, please allow me a note about the date stamp you use in this and the former example clip. Since the date stamp is created digitally, it is not part of the scene and might misguide the AWB estimation. You may supply a mask to the AWB script. The mask should be black in the area of the date stamp, then this area is excluded from the estimation.
Regarding your problem with the script and a longer clip, I share your suspicion about the variables. I was told that string memory is not cleaned up until script destruction, and therefore I minimise sring usage in RTE functions. Still, I don't know if this is also true with the GRunT plugin and local variable scope. And I have the suspicion that maybe the RTE script itself is a new string for every frame
I plan to rewrite the script as a plugin, but that will not happen in very near future. I saw that ultim is committedly working on AviSynth+. You might ask him if he's willing to implement string garbage collection during script serve stage (not possible with strict realtime requirements of course, but no problem with non-realtime processing scripts, and a great help).

As a short time workaround, I can only propose you split clips into parts of about 30', and join them after processing. It's what I do with the FSubstitute() script, which is even more complicated+long+demanding.
EDIT: For non-dynamic illuminant color situations, as your examples seem to me, I also suggest you use the ShiftCCT() script instead of AWB(). AWB() is able to adjust R, G, B independently, ShiftCCT() can only adjust along the temperature of a planckian radiator, and requires you to enter the originating color. But often, the result may still be pleasing.

Last edited by martin53; 11th November 2013 at 17:14.
martin53 is offline   Reply With Quote
Old 11th November 2013, 18:01   #50  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
ask him if he's willing to implement string garbage collection
Not sure if this could be possible, plugin writers call env->SaveString() to safely return a string (at least they are
supposed to) and the string is said to be a safe place to (EDIT: later) store whatever a plugin writer wishes, so long as he does
not overwrite outside of the original saved string boundaries. Would be hard not to break things without an entirely
differently defined new env->TempString() or whatever it might be called, perhaps free'ed on function exit.
(EDIT: Memory not re-usable by plugin writer, with memory total ownership passed to Avisynth.)
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 11th November 2013 at 18:16.
StainlessS is offline   Reply With Quote
Old 11th November 2013, 19:33   #51  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by martin53 View Post
I was told that string memory is not cleaned up until script destruction, and therefore I minimise sring usage in RTE functions. Still, I don't know if this is also true with the GRunT plugin and local variable scope. And I have the suspicion that maybe the RTE script itself is a new string for every frame
GRunT does not change the behaviour of ScriptClip regarding string usage, even when local scope is used, as string memory in Avisynth is orthogonal to the scope of variables and, as you say, is not released until script destruction.

The RTE script string (as a whole) is created only once when the containing script is loaded. However, that string itself is parsed afresh on every frame, which means that any identifiers and string literals within it are repeatedly added to the string heap.

Usually this is not significant, but for large run-time scripts, coupled with lots of source frames, it can add up. In fact, I discovered this was the source of a memory leak in SRestore (see here).

The solution is to move the code inside the run-time script to another function, reducing the run-time script itself to a simple function call. This effectively eliminates memory problems, and also gives a speed increase.

In other words, instead of
Code:
ScriptClip("""
... very long script ...
""")
use
Code:
function f(... some params ...) {
  ... previous script code ...
}
...
ScriptClip("f(...)")
Unless using GRunT, current_frame needs to be passed as a parameter to the function. (In GRunT, this is a global variable.)
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 11th November 2013, 22:54   #52  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by Gavino View Post
In other words, instead of
Code:
ScriptClip("""
... very long script ...
""")
use
Code:
function f(... some params ...) {
  ... previous script code ...
}
...
ScriptClip("f(...)")
But this is in conflict with this solution to create unique global variables for multiple instances of the script (because the replace function must be wrapped around the ScriptClip string). It is not possible to call one predefined function from all instances.
martin53 is offline   Reply With Quote
Old 12th November 2013, 00:55   #53  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
I see - basically, you need a distinct function for each instance.
That complicates the solution, but I think it can still be done as follows. (Warning - untested)

Replace:
Code:
GScriptClip(RT_StrReplace("GScript("+chr(34)+chr(34)+chr(34)+"""
    ... run-time script ...
"""+chr(34)+chr(34)+chr(34)+")", "%%%%%%", string(RTE_InstanceNumber)), local=true)
by:
Code:
Eval(RT_StrReplace("GScript("+chr(34)+chr(34)+chr(34)+"""
    function RTE_%%%%%%(...) {
      ... run-time script ...
    }
"""+chr(34)+chr(34)+chr(34)+")", "%%%%%%", string(RTE_InstanceNumber)))

GScriptClip(RT_StrReplace("RTE_%%%%%%(...)", "%%%%%%", string(RTE_InstanceNumber)), local=true)
This also moves the GScript call outside the RTE, avoiding what amounts to double parsing of the run-time script text on every frame.

EDIT: The replacement code can be further simplified to:
Code:
GScript(RT_StrReplace("""
    function RTE_%%%%%%(...) {
      ... run-time script ...
    }
""", "%%%%%%", string(RTE_InstanceNumber)))

GScriptClip(RT_StrReplace("RTE_%%%%%%(...)", "%%%%%%", string(RTE_InstanceNumber)), local=true)
__________________
GScript and GRunT - complex Avisynth scripting made easier

Last edited by Gavino; 12th November 2013 at 02:12. Reason: further simplification
Gavino is offline   Reply With Quote
Old 12th November 2013, 07:41   #54  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
Hi martin53!

I can see clip no problem in virtualDub from start tot end.
About mask for excluding date stamp, may be You can introduce area for processing or second clip for estimation color parameters, this approach exist in autogain from LaTo.
Waiting new version from You and try ShiftCCT.
yup.
yup is offline   Reply With Quote
Old 12th November 2013, 07:58   #55  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Gavino,
since you seem to know much about the identifiers memory hole, maybe you can give an answer to this question: Is it really neccessary to have different functions, or can the eval/GScript string evaluating functions be used outside the ScriptClip string - but inside the called function - without creating ever new identifiers, like:
Code:
function RTE_f(clip c, string instance, int i, ...) {
    c
    GScript(RT_Replace("""
        # runtime script payload body with global variables like:
        global RTE_f_%%%%%% = i
    """, "%%%%%%", instance))
    return last
}
...
#instance counter creation + global var init from other thread here
GScriptClip(RT_Replace("RTE_f(c, %%%%%%, i, ...)", "%%%%%%", instance), local=true, args=RT_Replace("%%%%%%, , i...", "%%%%%%", instance))
(Code is meant symbolically, not tested)

The core of my question is: Do also Eval() and GEcript() create a situation where the string is re-evaluated frame by frame, or can they be used to simplify the wrapping, and to avoid the creation of identical functions per instance.

Last edited by martin53; 12th November 2013 at 08:15.
martin53 is offline   Reply With Quote
Old 12th November 2013, 08:03   #56  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by yup View Post
Hi martin53!

I can see clip no problem in virtualDub from start tot end.
About mask for excluding date stamp, may be You can introduce area for processing or second clip for estimation color parameters, this approach exist in autogain from LaTo.
Waiting new version from You and try ShiftCCT.
yup.
The mask already is a second clip. Please take a look inside the AWB() script for details. If you had to punch the clip itself, you'd need several commands working on every frame. A mask can be constant over the whole clip. Also, a punched clip contains black areas (if black color is used for punching out). But black is then evaluated as clip color and influences the algorithm. These things are done inside the script, but only where they are allowed to.
To create the mask, you can use clp.BlankClip(color=$ffffff).Overlay(clp.Blankclip(width=..., height=...), x=..., y=...)

Last edited by martin53; 12th November 2013 at 08:09.
martin53 is offline   Reply With Quote
Old 12th November 2013, 10:42   #57  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by martin53 View Post
The core of my question is: Do also Eval() and GEcript() create a situation where the string is re-evaluated frame by frame, or can they be used to simplify the wrapping, and to avoid the creation of identical functions per instance.
Both Eval() and GScript() need to parse the provided string, so each time they are called, additional entries in Avisynth's string heap will be made for any identifiers and string literals found inside that string.

Therefore, although your solution will 'work', it will only reduce string memory consumption by approximately one half, since it still calls GScript() on every frame. What my proposed solution does is to generate and parse all the function bodies at compile-time, and hence only once (per instance).
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 12th November 2013, 20:06   #58  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Thanks for the clarification. I'll assemble a complete proposal for a syntax to fulfill the needs.

EDIT: please review. After approval, I'll update the other post too.
EDIT2: see here

Last edited by martin53; 13th November 2013 at 18:05. Reason: replaced example with link to example
martin53 is offline   Reply With Quote
Old 13th November 2013, 00:52   #59  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by martin53 View Post
EDIT: please review.
Yes, looks good.
I like the way you've tidied it up, moving the GScriptClip call inside GScript, requiring only a single call to RT_StrReplace.
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 13th November 2013, 09:17   #60  |  Link
yup
Registered User
 
Join Date: Feb 2003
Location: Russia, Moscow
Posts: 854
Hi martin53!
I am testing with mask which exclude area with date stamp.
Even when looking in VitrtalDub I see color shimmering from frame to frame.
I am try also other source (remember samplevhs.avi from my DropBox) and script work fine.
Different between 2 sources, first (first encawb.mp4) shooting during day time without artificial lighting (may be little cloudy weather and can note changing lighting condition), shooting non stop.
Second source (samplevhs.avi) shooting at evening time with fluorescent lamp lighting.
May be this info will be useful for debugging script.
yup.
yup is offline   Reply With Quote
Reply

Tags
awb, color, colorbalance

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 12:46.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.