Apply filters with local gradient

[nji]

Oh man - this looks far more complex than I had thought.

The less I'm able to contribute.

A thing I know about is that the collaboration between VD and AviSynth
is problematic sometimes when the script is very complex.
For example using MSU script for interpolating ... when doing
fastward/-backward in VD it crashes regulary. Have to jump to timeline.

About the code in general.
If I'm about to encircle strange effects I usually strip the code more and more
to see what's the cause.
So maybe disable multithread first.
If it still has unexpected effects (mem, time) maybe strip (comment)
lines of code one by one?
Especially mem usage should be really low for the alg ... I think.
The 256 masks can be handled not all at once but one after the other,
and for each iteration there are only 3 to 4 additional masks.
(==> There must be some strange thing happening).

Sorry I cannot tribute more than silly thoughts.

[StainlessS]

So maybe disable multithread first.

Is not multithreaded.

maybe strip (comment)

Not relevant.

The 256 masks can be handled not all at once but one after the other,
and for each iteration there are only 3 to 4 additional masks.

Total non-starter, and back to ~2 mins (or more) per frame.

I think problem in Avisynth, is hanging on to mem somewhere.

[nji]

Originally Posted by StainlessS

Total non-starter, and back to ~2 mins (or more) per frame.

I think problem in Avisynth, is hanging on to mem somewhere.

Another stupid thouht from my side:

So maybe the mem problem of AviSynth is the cause for the "run"time?
(Getting/ Letting mem usually takes long)

And then ... maybe worthwile to dig further to get a potential bug
Avisynth?

[StainlessS]

Removed script functions [ GreyRamp(), MyFunc() ], embed code directly into main level script.
Not much difference. (craps out about 1440 frames)

/*
    Req:
        Avs+
        SelectPixelByMask
        RT_Stats.
        requires OUT\ directory in current dir

        SelectPixelByMask(clip Data, clip Mask)
            Data    256 frames, Y8 or YV24, or RGB24, or RGB32.
            Mask    Y8
*/

# CONTRAST
StartCont = 0.5      # Mask 0
EndCont   = 2.0      # Mask 255
BaseName  = ".\OUT\MyPic_"
Matrix    = "Rec601"
myName    = "SelPix3: "
DEBUG     = True

###################
AviSource("D:\SELECTPIXELBYMASK\Parade.avi")
trim(60,0)                                        # trim off static start
BilinearResize(256,256)                           # Try not to swamp memory too much
ConvertToYV24()                                   # Sat will not work on Y8, And not RGB either (Tweak YUV only)
ORG=Last
FC=ORG.Framecount

Msg=MessageClip("Done")                           # Create now when mem avilable
###################
black = BlankClip(Length=1,color=$000000, width=1, height=256,Pixel_type="Y8")
white = BlankClip(Length=1,color=$ffffff, width=1, height=256,Pixel_type="Y8")
Mask=StackHorizontal(black,white).BilinearResize(512, 256).Crop(128,0,-128,-0).Loop(FC)

#Mask=Mask.BilinearResize(Width,Height)           # Rough ramp, single frame, same size as clip,  just for testing
#return Mask.info

BaseName=RT_GetFullPathName(BaseName)

StartTime=RT_TimerHP
For(n=0,FC-1) {
    current_frame=n
    Data=ORG.Trim(0,-1).Loop(256)       # 256 identical frames
    Data=Data.Animate (0, 255, "Tweak",
        \                          [* START Args          *]
        \ 0.0,                     [* Hue    default 0.0  *]
        \ 1.0,                     [* Sat    default 1.0  *]
        \ 0.0,                     [* Bright default 0.0  *]
        \ StartCont,               [* Cont   default 1.0  *]
        \ False,                   [* Coring Default True *]
        \                          [* END Args            *]
        \ 0.0,                     [* Hue    default 0.0  *]
        \ 1.0,                     [* Sat    default 1.0  *]
        \ 0.0,                     [* Bright default 0.0  *]
        \ EndCont,                 [* Cont   default 1.0  *]
        \ False                    [* Coring Default True *]
        \ )
    Out=Data.SelectPixelByMask(Mask).Loop(FC)
    Out=Out.ConvertToRGB32(matrix=Matrix).ImageWriter(file=BaseName, start=n,end=n, type="png")
    Out.RT_AverageLuma(w=1,h=1)                # Force write image file
    if(DEBUG) {
        Tim = RT_TimerHP
        RT_DebugF("%d] %f%% %.2fFPS",n,(n+1)*100.0/FC,1/(Tim-StartTime),name=myName)
        StartTime=Tim
    }
    Out=0
    Data=0
}

Return Msg

[nji]

Please excuse: I made a severe typo in my post before.
What I meant was:

If traced down to the line of AviSynth code which causes the mem problem,
and it should not grap/ keep as much,
then maybe it's an error of AviSynth?
(Which would make this investigation even more valuable)

[StainlessS]

Which would make this investigation even more valuable

Maybe so, but nothing that I can do about it.

[nji]

Originally Posted by StainlessS

Maybe so, but nothing that I can do about it.

No, of course.
That would be an issue for the AviSynth development team then.

But, if I get you right, it is not as far, that it is known if it's an error in your script or in a specific AviSynth statement?

EDIT:
A third option (that this alg can't be done without huge ressources of space and time)
cannot be true. That's what I do know for sure.

[zorr]

This piqued my interest. Unfortunately my code-fu is not very strong when it comes to Avisynth, but I was able to implement this in a VapourSynth script.

I haven't done any extensive testing but it seems to run ok, memory consumption is about 2,4GB when running with a 1080x576 source video. The 8-bit mask in this case had all 256 values in it, the script will skip calculating the effect for the missing mask values and will be more efficient when there are less mask values. Speed was about 1fps with 256 mask values and about 5-6 fps with 16 mask values, using a horizontal blur effect as the test function (I had another heavy task running in the background so these measurements are not very accurate).

If you're interested I can post the script (to VapourSynth section of this forum).

[nji]

Originally Posted by zorr

... I was able to implement this in a VapourSynth script.
...If you're interested I can post the script (to VapourSynth section of this forum).

Yes, please do.

@StainlessS:
Why does the Avisynth version crash/ take so much mem?

[StainlessS]

If you're interested I can post the script (to VapourSynth section of this forum).

I guess you could do, I'll take a peek when I have time.

Why does the Avisynth version crash/ take so much mem?

No idea, nothing that I'm doing [EDIT: I think], I dont allocate any RAM, is connected with the way avisynth is implemented.
EDIT: I do create a new frame on filter construction, but should be destroyed when destructing [when setting the relevant clips to 0 in script].

EDIT: Zorr,
Would your code work with all filters, eg could you for instance using a gray ramp, blur max 1.58 at frame left, null in middle and sharpen max 1.0 at frame right.
(or eg temporal smooth or sharpen, or MC filtering, or frame replacement filtering, or is it just a LUT).

EDIT: nji
I think that you mentioned OpenGL somewhere. IIRC, that stuff has some kind of pixel filtering operations, so it is likely exactly what you want
(dont know much at all about OpenGL, and not relally feeling inclined to duplicate OpenGL as some kind of Avisynth filter).

[zorr]

I posted my version here.

Zorr,
Would your code work with all filters, eg could you for instance using a gray ramp, blur max 1.58 at frame left, null in middle and sharpen max 1.0 at frame right.
(or eg temporal smooth or sharpen, or MC filtering, or frame replacement filtering, or is it just a LUT).

Yes, it applies the filter for the whole frame and then extracts the parts with correct mask value and composites all the variations with different mask values to a final result. The applied function could do anything and be as complex as needed.

[StainlessS]

Thanks Zorr, I cant spend any time examining your code right now, busy trying to recover some accidentally deleted files.
Got thousands (10,000) of text files to look through, busy making script to show first x number of lines of text files on video frame.

EDIT: I did consider trying out Expr() but shied off as I've never used it before. [I presume it will take Masktools expressions without change].

[nji]

Originally Posted by StainlessS;

nji
I think that you mentioned OpenGL somewhere. IIRC, that stuff has some kind of pixel filtering operations, so it is likely exactly what you want
(dont know much at all about OpenGL, and not relally feeling inclined to duplicate OpenGL as some kind of Avisynth filter).

I've never been in that "OpenGL corner" to know about its pixel filtering ops.
Maybe it exists.
Surely I will be able to implement even in plain C++ etc.

But the point is I'm needing it on app level of movie manipulation.
And - as I tried to point out in this thread - it seems quite interesting
to see about the difference of exact pixel filtering vs. mishmash that
is proposed all the time.

The only thing I knew about AviSynth so far that was it's just that
inter-filter manipulation which makes its strength.
Sorry this thread convinced me of the contrary.
Thanks anyway!

[StainlessS]

Avisynth was never meant to be abused in the way that I've been trying to pervert it, trouble is I dont know how to abuse it well enough [nor know if it is possible to to do it without error].

Anyways, I've did repeat of script in post #44, but with the trim line chaged to "trim(60,-800) # trim off static start, 800 frames only".
[try to ensure script completes]
Also force freed destframe in dll destructor(not really necessary but just to be sure) , enabled debug stuff in dll and capped output to file in Debugview.

00000087    47.04938507 SelectPixelByMask: Calling AddFunction SelectPixelByMask                   # Add dll function to avisynth (pre script execution)
00000088    47.04946518 SelectPixelByMask: returned from AddFunction

00000198    0.14413995  SelectPixelByMask: Creator IN
00000199    0.14419755  SelectPixelByMask: Creator Calling Constructor                             # Creating filter
00000200    0.14424321  SelectPixelByMask: Constructor IN/OUT
00000201    0.14428234  SelectPixelByMask: Creator returned from Constructor, OUT
00000202    0.14447002  SelectPixelByMask: GetFrame IN
00000203    0.44192630  SelectPixelByMask: GetFrame OUT
00000204    0.52262574  SelPix3: 0] 0.125000% 2.63FPS                                              # in script
00000205    0.52272862  SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT    # destframe force freed, filter destroyed (called when out=0 in script)
00000206    0.52439749  SelectPixelByMask: Creator IN
00000207    0.52443445  SelectPixelByMask: Creator Calling Constructor
00000208    0.52446741  SelectPixelByMask: Constructor IN/OUT
00000209    0.52449894  SelectPixelByMask: Creator returned from Constructor, OUT
00000210    0.52462900  SelectPixelByMask: GetFrame IN
00000211    0.80134016  SelectPixelByMask: GetFrame OUT
00000212    0.84081995  SelPix3: 1] 0.250000% 3.14FPS
00000213    0.84093916  SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT
00000214    0.84261560  SelectPixelByMask: Creator IN
00000215    0.84266162  SelectPixelByMask: Creator Calling Constructor
00000216    0.84270108  SelectPixelByMask: Constructor IN/OUT
00000217    0.84273803  SelectPixelByMask: Creator returned from Constructor, OUT
00000218    0.84287393  SelectPixelByMask: GetFrame IN
00000219    1.11795390  SelectPixelByMask: GetFrame OUT
00000220    1.15778732  SelPix3: 2] 0.375000% 3.15FPS

# ...

00006558    356.57012939    SelPix3: 795] 99.500000% 2.63FPS
00006559    356.57028198    SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT
00006560    356.57205200    SelectPixelByMask: Creator IN
00006561    356.57211304    SelectPixelByMask: Creator Calling Constructor
00006562    356.57214355    SelectPixelByMask: Constructor IN/OUT
00006563    356.57217407    SelectPixelByMask: Creator returned from Constructor, OUT
00006564    356.57232666    SelectPixelByMask: GetFrame IN
00006565    356.99841309    SelectPixelByMask: GetFrame OUT
00006566    357.03823853    SelPix3: 796] 99.625000% 2.14FPS
00006567    357.03839111    SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT
00006568    357.04010010    SelectPixelByMask: Creator IN
00006569    357.04016113    SelectPixelByMask: Creator Calling Constructor
00006570    357.04019165    SelectPixelByMask: Constructor IN/OUT
00006571    357.04022217    SelectPixelByMask: Creator returned from Constructor, OUT
00006572    357.04037476    SelectPixelByMask: GetFrame IN
00006573    357.52993774    SelectPixelByMask: GetFrame OUT
00006574    357.57003784    SelPix3: 797] 99.750000% 1.88FPS
00006575    357.57019043    SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT
00006576    357.57192993    SelectPixelByMask: Creator IN
00006577    357.57199097    SelectPixelByMask: Creator Calling Constructor
00006578    357.57202148    SelectPixelByMask: Constructor IN/OUT
00006579    357.57205200    SelectPixelByMask: Creator returned from Constructor, OUT
00006580    357.57220459    SelectPixelByMask: GetFrame IN
00006581    357.93356323    SelectPixelByMask: GetFrame OUT
00006582    357.97360229    SelPix3: 798] 99.875000% 2.48FPS
00006583    357.97375488    SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT
00006584    359.09152222    SelectPixelByMask: Creator IN
00006585    359.09158325    SelectPixelByMask: Creator Calling Constructor
00006586    359.09161377    SelectPixelByMask: Constructor IN/OUT
00006587    359.09164429    SelectPixelByMask: Creator returned from Constructor, OUT
00006588    359.09185791    SelectPixelByMask: GetFrame IN
00006589    359.38601685    SelectPixelByMask: GetFrame OUT
00006590    359.42672729    SelPix3: 799] 100.000000% 0.69FPS
00006591    359.42684937    SelectPixelByMask: Destructor IN, setting destframe=NULL Destructor OUT

1) Rising mem usage


2) Script shows "Done" messageclip, finished. Pause then close VDub (window closes, remains in memory.)


3) Remains in memory, maybe 3 stages of avisynth freeing memory


4) End of 3rd stage of freeing memory, Vdub process gone. Little bit extra mem freed.


Note Up time and Commit(GB) [EDIT: Pagefile expanding, I saw it in earlier run at 35GB]

Looks like memory cannot be freed until brought from pagefile into physical memory, I guess thats to do with linked lists.

[nji]

... thanks again! :-)

Well - as told above - I'm completely new to AviSynth and by that
am not able to assess if the task of gradient driven filter application
is only possible by "abusing" AviSynth.

However if you look at the Vaporsynth script that was provided by zorr:
It's very small and neat.

Again ... if you're sure the error is by AviSynth (and not by your script)
maybe strip it to the minimum and post it to AviSynth development subforum?

[zorr]

EDIT: I did consider trying out Expr() but shied off as I've never used it before. [I presume it will take Masktools expressions without change].

Yes Expr is a lot like MaskTools.

I tried to do a pretty much direct and simplified conversion of my VapourSynth version. It's using your GreyRamp() and limited 256x256 resolution and only processing the luma channel... yet it seems to run out of memory or stack space. I can get it to process 20-30 mask values but anything more and it craps out in multitude of ways, yet the memory usage is actually very low.

The version below is processing a modest number of mask values, you can try more by changing the mask_value of the applyFunc call or the recursion test value on the last line of applyFunc. Some of these issues may be because I only have an ancient x86 version of Avisynth installed. The applyFunc should be run inside ScriptClip in order to use animated masks but that made it fail as well.

src = AVISource("source.avi")
src = ConvertToYV16(src)
src = Crop(src, 0,0,256,256)

mask = GreyRamp()
mask = ConvertToY8(mask)

final = BlankClip(src)
result = applyFunc(mask, src, final, 128)

return result

Function applyFunc(mask, src, final, int mask_value) {
	m = mask.mt_lut("x "+String(mask_value)+" = 255 0 ?")
	gradient_clip = blurx(src, mask_value/255.0)
	final = mt_lutxyz(final, gradient_clip, m, "z 255 = y x ?")
	return mask_value < 140 ? applyFunc(mask, src, final, mask_value+1) : final	
}

Function blurx(clip, float ratio) {
	return FastBlur(clip, ratio*5.0)
}