Apply filters with local gradient

[nji]

I recently ran into the task of restricting filters
(brightness, contrast, saturation, hue)
by a local mask.
With a kind support this was solved by an AviSynth script.
https://forum.doom9.org/showthread.p...43#post1893743
And as I normally do in VirtualDub2 I found a workaround there too.

No I would like to do the same filters in a "local gradient way"
(Portions of the area need some more contrast, others much more.
Saturation etc. dito).

It looks like no chance to fix that with VD2.

Being an absolute newbie to AviSynth I searched for infos.
I found something like:
http://avisynth.nl/index.php/Filter_SDK/GradientMask
http://avisynth.nl/index.php/GraMaMa
But have no idea if this is the right spot...

Any help appreciated.

[poisondeathray]

It depends on the complexity of the gradient.

Is it linear and simple, or complex pattern ? Are the edges sharp or feathered (gradual transition) in 1 or more directions ? Does gradient area change over time ?

If it's simple, it might be possible to do in avisynth directly

If it's complex you will probably want to do at least the gradient mask part in other programs , possibly other operations - such GUI's that can keyframe (e.g. resolve, blender, natron, hitfilm). The filter parameters can also be keyframed directly also, so >2 levels of control

To apply the filters through the gradient in avisynth, you can use overlay() or mt_merge()

[nji]

It's the most simple case:

No change over the time.
The gradient is given by a b/w mask, which determines amount of application.
Application should be one of: brightness, contrast, hue, saturation, ... perhaps sharpen.

[manono]

Originally Posted by nji

The gradient is given by a b/w mask, which determines amount of application.

A=Last
B=A.Filter Here
B=B.Another Filter If Needed
Mask=ImageSource("Mask.bmp")
Overlay(A,B,0,0,Mask)

[nji]

Originally Posted by manono

A=Last
B=A.Filter Here
B=B.Another Filter If Needed
Mask=ImageSource("Mask.bmp")
Overlay(A,B,0,0,Mask)

Intuitive I "felt" this is not correct.

Thinking about it ... I would like to declare,
that overlaying as above is only correct
if the filter is a linear function.
For all other filters the overlaying does something "similar".

I really hope there is someone to confirm
or to contradict that.

And if it should be right:

Which of the mentioned filters are linear?

And - most important:
is there really no way to do it even with AviSynth?
Gradient application of filters should be very often needed.
Or is the approximation by overlaying good enough?

Help!

[poisondeathray]

Originally Posted by nji

Intuitive I "felt" this is not correct.

Thinking about it ... I would like to declare,
that overlaying as above is only correct
if the filter is a linear function.
For all other filters the overlaying does something "similar".

I really hope there is someone to confirm
or to contradict that.

And if it should be right:

Which of the mentioned filters are linear?

And - most important:
is there really no way to do it even with AviSynth?
Gradient application of filters should be very often needed.
Or is the approximation by overlaying good enough?

Help!

The mask can be a non linear gradient , so the application of filter(s) can be distributed along non linear gradient

Or are you asking something else ?

[manono]

I, for one, have no idea what he's asking. He's the one that said:

Originally Posted by nji

The gradient is given by a b/w mask, which determines amount of application.

So, if the mask is properly feathered (faded out), then it'll do the job of blending the filtered portion of the frame back into the unfiltered portion. And, as StainlessS says, the Opacity setting of Overlay can also help. Has he even tried yet to see if the script does what he wants?

[StainlessS]

Posts in a VDub2 thread.

Originally Posted by shekh

First thought: this is implementation nonsence: filters are very expensive to manipulate parameters per-pixel in generic way.
You won't be happy to run this.

Originally Posted by nji

(... well, I won't argue with the boss.

But still...

Although I'm in SWE I at first place I don't look at implementation
but on what is needed, helpful, useful.
And it's just the movie why I started this thread
where a gradient of parameters is needed.
As - with the support of the kind members above -
the local restriction of filters is solved...
in my example it turned out, that the original film material
is "bleeched" by contrast, brightness and saturation
in a local gradient too!
I fiddled around with the filter stream/ pipeline "trick" (see above)
with all kind of masks and blending modes ... to no success.
But ... doing an older movie restoration ... this is will
be a very common needed task.
So how to do that?
(Probably will do a separate task, in VD, and if unsuccessful
in AviSynth subforum).

Second:
It may be "very expensive" ...
But your fine master blend does exactly that, ... in time gradient?
I personal don't care if something takes long to process.
As long it is done at all.

Please don't get me wrong.
I'm not argueing, but just put my thinking, to learn...

???

Greetings!

Originally Posted by shekh

Normally filters are optimized to process massive amount of pixels at once.
Varying parameters each frame means the filter is processing w*h amount of pixels.
Varying each pixel ... means the filter is processing 1 pixel and then a shitload of work is required to change setup.

Something intermediate that can be done: run the filter N times to produce N resulting frames and then interpolate them.
However, few examples:

contrast. Filtering with average contrast is quite similar to average blending with full contrast. Would you even notice?
rotate. In this case increasing N is hopeless, it really ought to be per-pixel. Btw the result could be funny but I hardly imagine practical use

+++++ more.

[in say RGB] To alter eg Saturation of a single source pixel using say lumaY value in a single plane gradient GradY to control output,
would involve for each R,G,B channel [something like],

OutR[x,y] = R_Magic(InR[x,y],InG[x,y],InB[x,y],GradY[x,y])
OutG[x,y] = G_Magic(InR[x,y],InG[x,y],InB[x,y],GradY[x,y])
OutB[x,y] = B_Magic(InR[x,y],InG[x,y],InB[x,y],GradY[x,y])

Something like above for each and every pixel, and where X_Magic involves multiple terms in the calculations,
it will not be simple[and very very slow], as already suggested by Shekh.
[EDIT: For eg hue, would also need to alter behaviour depending upon which segment of the color wheel the input colour lives in]

Dont know if avs+ Expr() filter would be able to do as above [maybe using input channels extracted to eg Y8], and then results re-combined to RGB.
Also, maybe one of Wonkey_Monkey's weird rpn type filters could do it [I aint got a clue how to use either Expr or them there wonkey things].

Good luck to anyone that attempts something like above.

EDIT: And to do same for sharpening [as suggested somewhere], would also have to sample pixels surrounding [x,y] of each channel.

Doubt if anyone is likely to volunteer for any such simple project.

EDIT: Here the quote that mentions Sharpen:

Originally Posted by nji

It's the most simple case:

No change over the time.
The gradient is given by a b/w mask, which determines amount of application.
Application should be one of: brightness, contrast, hue, saturation, ... perhaps sharpen.

[nji]

Yes, actually I am "asking something else"

Are my thoughts above correct, or wrong?
If right:
Which of the filters actually are linear?
(R, G, B seems yes.
H, S, L seems not - if in RGB-format, but if you turn to HSL-format then yes, ???.
Contrast, Sharpen are not linear, ...)

Non-linear gradient of the mask does only equal the point-wise-application of a filter,
if the gradient is "like" the filter ...
This means: Not correct to approximate a non-linear filter like sharpen.

So: How is that common task done ... everywhere?

(Nothing but questions ...)

[poisondeathray]

Originally Posted by nji

Yes, actually I am "asking something else"

Are my thoughts above correct, or wrong?
If right:
Which of the filters actually are linear?
(R, G, B seems yes.
H, S, L seems not - if in RGB-format, but if you turn to HSL-format then yes, ???.
Contrast, Sharpen are not linear, ...)

Non-linear gradient of the mask does only equal the point-wise-application of a filter,
if the gradient is "like" the filter ...
This means: Not correct to approximate a non-linear filter like sharpen.

So: How is that common task done ... everywhere?

(Nothing but questions ...)

Probably not what you're asking, but even if they appear to operate linearly, technically, NONE of the filters are actually linear when you just use them - because there is a gamma function "baked" in to everything with most common video formats . 1+1 does not equal 2.

In order for most video formats to be linear, you have to "degamma" or linearize it. Basically reverse the transfer function. Then all operations can be linear when operating in a linear space. 1+1 =2

(There are some exceptions, such as resampleHQ which resizes in linear light)



Or are you asking something else ?

[StainlessS]

All overlay does is BLEND TWO frames together with [EDIT: per pixe] control via mask and [EDIT: Frame global control] via Opacity arg [there are a few other ops in addition to blend, eg lighten/darken, but still a sort of blend],
taking both mask pixel color and opacity into account, will result in eg 25% of pixel from Frame_1 and 100-25% of same pixel from Frame_2 [ie blended].

[StainlessS]

He wants to eg change Hue of a pixel, with control on change via mask pixel value. [EDIT: I think]

[To tell the truth, I aint sure if its a crazy idea or genius, I suspect crazy]

EDIT: Same for other mentioned ops, like Sharpen, with control via mask pixel value.
I guess that simple Overlay blend with Pre-Sharpened frame should work OK [to some degree].

Presumably would need offset for Hue adjust, for +ve/-ve adjustment.

EDIT: He basically wants a Per Pixel filter, where the filter operation for the individual pixel is controlled via a mask pixel.

EDIT: Above based on this post from VDub forum:

Originally Posted by nji

... in addition to that the idea for that meta filter:

Take the b/w mask not only for switching on/off locally the manipulated filter
but ... well ... take the mask's values as degree of the set parameters.
I.e. degrees of the set parameters, not degree of the blending.
By this gradients could nicely be done.
Thinking about it ... yes, it's like master blend filter of the parameters,
but not in time but in space.
And - like there - would work only for some filters.
Cool?
Or complete nonsense?

[poisondeathray]

"Point-wise application of the filter" - If you meant "spatially" linear...

"sharpen" is slightly different because more than 1 pixel is changed. The "perception" of "sharpness" boils down to local contrast enhancement.

If I have 1 black pixel. You can't "sharpen" that. You need adjacent pixels for contrast.

But things like H,S,V can change per pixel if in a full color format (or "interpolated" to a full color format -because most consumer video formats will be 4:2:0 - the color information will actually be 1/2 width, 1/2 height). So you don't have a 1:1 pixel mapping for the color information and the luminance. If you convert it to (a new, interpolated) 4:4:4 . Then you do have (a new) 1:1 pixel mapping between Y,Cb,Cr planes . Then hue of individual pixels can be changed, and individual pixels controlled by the mask . Same with R,G,B. RGB is full color. Each R,G,B plane is the same dimension (no subsampling)

Or were you asking something else ?

[StainlessS]

PDR,
Some points that did not at all occur to me.

Then hue of individual pixels can be changed, and individual pixels controlled by the mask

Confirm, that is a theoretical observation, and not something that already exists for single pixel fixes ?

[poisondeathray]

Originally Posted by StainlessS

Confirm, that is a theoretical observation, and not something that already exists for single pixel fixes ?

Bad wording

To clarify - if you change hue, say with tweak() or similar, yes you can control application of that layer per pixel by using a mask

But the effect itself applied to the filter layer does not control individual pixels directly because of subsampling .





But I have no idea if this is what he is asking...

[nji]

Thank you, all of you, I really appreciate the support given to me.
Not a hint of the rudeness that unfortunately is common in most
other forums.

And please, don't begin to insinuate bad things from my side
like "crazy idea" or "probably hasn't even tried"
as reaction when it becomes difficult to understand.

I can well asure to you that I only post a question after having
tried and thought by myself, and an unsuccessful search in the net.
(And, of course, I tried the "blending solution").

As I have the strong impression that you're really interested
I like to explain the thinking again:

As told, I recently stumbled upon the problem of
having to apply a filter not as usual uniforum but in a "patchy" way.

As I'm coming from the "math side" my thinking was that I
need a mechanism to ... well ... do a point-wise changing
of the parameters of the specific filter (say sharpen).

It didn't came to me to gradient blend a full sharpened layer,
as my "implicite intuition" hindered my on thinking that way.
(When I have to close a hole in a white wall, I don't look
around to see what "does the job". "Oh, the toothpaste!" )

As I was pointed often to the blending thing by you,
I tried it, and the result "looks" satisfying.
(Like the toothpaste BTW )
So the problem is solved?

Question only academic?
No, I don't think so.
If, for example, you by some means know that the frame needs
pointwise-parameter-sharpen-parameter-changing,
as the disturbance process have been in that way...
wouldn't you want to revert it correctly?
And not only in way "it looks OK"?

2nd:
I've to admit that I'm interested in which cases the
blending thing does the job exactly.
As I said, it depends on the filter.
I suspect the less linear the filter is,
the larger the difference from correctness will be.
But I do not know.
And I'm a little lost about that.
And of course I cannot test it, as I don't know a method
to generate the pixel-wise parameter change
and compare it with the blending.

If you take away the time consuming aspect it should be
quite easy to implement:
Call the specific filter for every single pixel with
the parameter value that is given by the mask value.

And of course the problem is not in time.
So I had look at image manipulation progs and libs
without success.

[nji]

Originally Posted by nji

...
If you take away the time consuming aspect it should be
quite easy to implement:
Call the specific filter for every single pixel with
the parameter value that is given by the mask value.

Subject to my thinking about "linear filter" is correct
(sadly nobody confirmed or convincingly contradicts)
the mathematical correct implementation could be "sped up"
by not calling the "inner filter" for every pixel of the manipulated frame
but for ever value of the gradient mask:

Call the "inner filter" for each value v that arises from the gradient mask.
In the resulting frame set only the pixels (maybe by masking)
that have the value v in the gradient mask.
Cool, don't you agree?
OK, still 256 times for every frame.
But a solution at last (that doesn't exist anywhere).
And allows comparing to the "blend workaround".

---
"We don't need to worry about the crazy ones,
the normal ones are the problem!"

[poisondeathray]

Originally Posted by nji

As I'm coming from the "math side" my thinking was that I
need a mechanism to ... well ... do a point-wise changing
of the parameters of the specific filter (say sharpen).

"Point wise changing ?" Or did you mean procedurally adjusting the filter values ? . e.g. say saturation 1.02 vs. 1.03 . Or sharpen 1.2 vs. 1.3. Or maybe 1.25 would be better - and preview it to "see" it ? ie. How do you know what values to use or what is the better "answer" or settings to use?

If this was the question, you can use sliders to control settings in avspmod like a GUI. You can also preview in avspmod with tabs and flip with number keys quickly so each swapped and superimposed (e.g. compare different versions, different settings how they look)

You can view different output "nodes" . It's similar to the "chains" or branches you were talking about in vdub. You're viewing different chains or different points in the chain. Say you wanted to view layer A separately with just the filter only, maybe put that in tab2 . Maybe tab3 has the filter settings stronger. Maybe tab4 has the overlay at opacity 100%. Maybe tab5 has the overlay with a feathered mask instead of a sharp binary mask to make it blend better. Tab1 has the "original" so you can see it quickly to compare

I tried it, and the result "looks" satisfying.

"satisfying", but not "great" ?

Why wasn't it better than satisfying ? Possibly it means you have to adjust the mask, and or filter, or add other filters. Wild guesses - There can be many other reasons the results were not ideal, and possibly many ways to improve on the current results.

You're looking a the 1 filter, 1 mask scenario - You can also combine multiple masks with blend modes and multiple filters

Question only academic?
No, I don't think so.
If, for example, you by some means know that the frame needs
pointwise-parameter-sharpen-parameter-changing,
as the disturbance process have been in that way...
wouldn't you want to revert it correctly?
And not only in way "it looks OK"?

Not at all. Make it better than OK.

Compositing is used every day , frequently, many different programs. eg . photographers use similar techniques all the time in photoshop, the underlying concepts are the same here, maybe slightly different for video and colorspace

Say you want to sharpen the clouds in the background, but nothing else. Or denoise shadows below a car. Or change the lipstick color of some beautiful woman. It's all done through masks

I've to admit that I'm interested in which cases the
blending thing does the job exactly.
As I said, it depends on the filter.
I suspect the less linear the filter is,
the larger the difference from correctness will be.
But I do not know.

It depends on what the source problem is, exactly. And how do you know what is "correct" exactly ? Your old video is probably lossy compressed and "not exact" in the first place

But if you "know" exactly what the problem is - you can be as complete and tedious as setting individual pixel values. e.g. pixel in x=34, y=45 has a YCbCr value of 134,65,74. That's 100% exact.

The "commonly" encountered scenarios that you won't be able to "fix" very easily in these scenarios is if there is channel clipping somewhere. e.g lets say part of the sky is blown out. Or some shadow details are crushed. Those are not as easily "fixable" by these methods such as a simple filter application +/- mask. You have to use other advanced methods




I have no idea if this is on the right track...it's just the wording here is very confusing... at least to me

[nji]

@PDR:
I'm sorry but I can't better explain than in my last post.
Sorry.
Maybe it's because I'm no native speaker.
(Well - actually I am. But not for English )

A short addendum to my last long post:

As said in a post above using a non-linear mask
would surely make it mathematical correct.
But how do you know the exact mask-values?
The more non-linear the filter operates on a specific pixel
the more "non-gradient" will the mask point be.

2nd HSL:
I'm pretty sure R,G and B changing is linear.
So the blending workaround will do perfect.
But what about H,S and L?
If you have look of the color of a pixel as its "state",
then the filter changes it from stateA to stateB.
Having the state in RGB representation, changing the
say lightness does not look to be linear. Or H or S.
But if you do another representation (HSL)
it will be the same states of course.
But applying the change of H, S or L seems to be linear now.
BOING!!
"The situation in general is worrying"
(The inspector in "Die neun Leben des Tomas Katz")

[poisondeathray]

Originally Posted by nji

As said in a post above using a non-linear mask
would surely make it mathematical correct.
But how do you know the exact mask-values?
The more non-linear the filter operates on a specific pixel
the more "non-gradient" will the mask point be.

I think I understand what you're saying roughly, in the mathematical context

But in practice, with an old movie or video source, you're not going to have a exact gradient anyways

And if clipping ever occurs , anywhere, all bets are off

2nd HSL:
I'm pretty sure R,G and B changing is linear.
So the blending workaround will do perfect.
But what about H,S and L?
If you have look of the color of a pixel as its "state",
then the filter changes it from stateA to stateB.
Having the state in RGB representation, changing the
say lightness does look to be linear. Or H or S.
But if you do another representation (HSL)
it will be the same states of course.
But applying the change of H, S or L seems to be linear now.

Changing can be linear, but the actual end results will technically not be linear because of gamma error