Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 3rd July 2021, 05:07   #421  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
btw the existing StripeMask has a memory leak. I wonder why that never came up before. I cannot reproduce the memory leak in v2 beta in Avisynth. In VapourSynth, memory usage goes up from 1.3GB to 1.8GB over time but that could just be the way it handles the cache.

I've been playing with parameters to try to match previous results; the bar is pretty high, my best efforts barely come to 90% of the previous results!

Working with Luma-only (ShufflePlanes) works better than converting to Grey8. Btw, why do ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY) and resize.Point(format=vs.GRAY8) give different results? The latter gives a slightly darker output.

This is the pre-filtering that worked the best so far... yet it's not as good as the previous results! The linear gamma conversion is a bit different. I guess I got very lucky on the last released version (except that the old version assumes PC range and would behave different with Full range input)

Code:
video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1/2.2, min_in=16, max_in=235, min_out=65, max_out=190)
The question is: what makes it that I can't fully match the accuracy of the last version? It was using thr=23. Now I nearly match it with thr=15, range=125 (considerably compressing the data range)

Before I was using ColorYUV to change gamma. Perhaps the difference is the way Gamma is applied, over Full or PC range? I think it was applying full-range gamma on a PC-range clip... somehow improving results. What impact did it really have? Anything to learn here in terms of better tweaking the input before processing?

Edit: I suppose converting to GRAY8 takes chroma into account during conversion. If I use ShufflePlanes, then coloured stripes with similar Luma would fail to be detected.

Note: StripeMask could very well run on an image with Width and Height reduced by half to increase performance.

Last edited by MysteryX; 3rd July 2021 at 07:15.
MysteryX is offline   Reply With Quote
Old 3rd July 2021, 12:12   #422  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,250
blur(0.6) produces the following result on a single white pixel (in RGB32, anyway; it's slightly different in RGB24 for some reason):

Code:
 8  29  8
29 110 29
 8  29  8
So it does spread to the corners, and these would be the best numbers if you want the closest match.

EDIT: My mistake, that's in Avisynth. Vapoursynth blur() might be different.
__________________
My AviSynth filters / I'm the Doctor

Last edited by wonkey_monkey; 3rd July 2021 at 13:59.
wonkey_monkey is online now   Reply With Quote
Old 3rd July 2021, 21:16   #423  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
I could use some help here. Here's a new DLL vesion.

The download link contains a sample video clip. Can you try to get a more accurate stripe detection, which means, more of the fence being detected, and less false positives. You can compare to Avisynth StripeMask from before. Best I got so far is about 90% the quality of before for some reason. You can play with levels, contrasts, gamma, perhaps sharpening. Havsfunc has SmoothLevels which perhaps could be useful. Resize to Gray8 or ShufflePlane? This is time-consuming trial-and-error task that requires a lot of testing. This version has an additional parameter "skip". When set, it bypasses the pre-filter calls so you can do it manually.

Let me know what you find out!

Also for the crash about invalid core -- I made a small change. Can you test whether you still get a crash?

As for ContinuousMask, I fixed it to process all planes (instead of leaving garbage on chroma planes). It should behave more appropriately now. There really isn't much to debug in there... Perhaps I could use a Threshold parameter so that it processes pixels above a certain value (other than 0), if it gets useful for anyone else.

Last edited by MysteryX; 3rd July 2021 at 21:52.
MysteryX is offline   Reply With Quote
Old 3rd July 2021, 23:41   #424  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
Quote:
Originally Posted by MysteryX View Post
You can compare to Avisynth StripeMask from before. Best I got so far is about 90% the quality of before for some reason.
Just to be clear - so this difference is not just between the Avisynth and Vapoursynth versions but with the current and previous Avisynth versions as well?

Quote:
Originally Posted by MysteryX View Post
This is time-consuming trial-and-error task that requires a lot of testing.
Sounds like a perfect job for Zopti. Since we have a good result (the old Avisynth version) we can compare the result with it, I think a simple diff will do instead of some fancy SSIM or GMSD metric. If you want even better results you could create a mask manually (even for just one frame).

I'll try to come up with a Zopti script and do some testing. First I will need to recreate the good result and save it as lossless video.
zorr is offline   Reply With Quote
Old 4th July 2021, 00:00   #425  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
Previous version was using ColorYUV to change gamma. It was never converting to Gray8 and was only using the Luma plane, so that's the equivalent of ShufflePlanes. I don't know whether there are other "unknown" factors or errors coming into play.
MysteryX is offline   Reply With Quote
Old 4th July 2021, 10:50   #426  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
Is this the correct code for the good result (using FrameRateConverter v1.3)?

Code:
src = FFVideoSource("MotionTest2.mp4")
src = src.BicubicResize(src.width / 2, src.height / 2)
mask = src.StripeMask(thr=23)
return mask
zorr is offline   Reply With Quote
Old 4th July 2021, 11:23   #427  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
The VapourSynth version is returning a mask with two gray values, is that expected?



The Avisynth version returns a mask with luma 0 or 255, VS version returns a mask with luma values 0, 100 or 200.

EDIT: Ah, that's controlled by the parameters str and strf. So I guess we should use the default values to match the Avisynth version?

Last edited by zorr; 4th July 2021 at 15:27.
zorr is offline   Reply With Quote
Old 4th July 2021, 15:34   #428  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
StripeMask now has parameter range. What's it for and what is the usable value range? Should it always be max_out - max_in?
zorr is offline   Reply With Quote
Old 5th July 2021, 01:47   #429  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
The lesser-gray is adding the next frame into it. Temporal stabilizer. Str sets the color of matching areas, strf(orward) sets the color of next frame data.

BTW......... could I improve performance 2x by caching the data of next frame instead of calculating it twice? I guess so. Or perhaps initialize the plugin twice with 1-frame offset and merge them.

Btw reducing size by half is only for performance reason because I don't need the clip to be that big; can reduce it to various sizes to test with various block sizes. Also see the impact of processing on half-resolution or quarter-resolution vs full-resolution (with half or quarter block sizes).

Range. If you want Full range, set 255. If you want PC range, set 220. To further compress the data range, I've set it to 125. Note that if skip=1, range isn't doing anything because it's only used for pre-filtering.

The ZIP contains the VapourSynth script that is working good,

Code:
import os
import sys
dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(dir)

import vapoursynth as vs
core = vs.get_core()

core.std.LoadPlugin(path=dir + r"\FrameRateConverter-x64.dll")
file="MotionTest2.mp4"
video = core.ffms2.Source(source=file, format=vs.YUV420P8)
video = video.resize.Bicubic(video.width / 2, video.height / 2)

video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1/2.2, min_in=16, max_in=235, min_out=65, max_out=190) # range=125
video = core.frc.StripeMask(video, blksize=16, str=200, strf=100, thr=15, range=125, skip=1)

video.set_output()
btw I'm curious what would be the effect of sharpening.

Last edited by MysteryX; 5th July 2021 at 01:55.
MysteryX is offline   Reply With Quote
Old 6th July 2021, 00:58   #430  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
I've got some interesting results.

Short version: You can use thr=23, min_out=0, max_out=208 and gamma=1/2.078 to get a pretty good result. I measured similarity against the older Avisynth version and average SSIM is 0.987 per frame (1.0 is perfect score).

Long version: Naturally I used Zopti to find the optimal values. First I needed a "reference video" so I ran StripeMask with FrameRateConverter v1.3 and saved the resulting mask as lossless video. I also created a pre-scaled lossless version of the original video to avoid unnecessary work during optimization.

I created this VapourSynth / Zopti script:

Code:
import os
import sys
from zoptilib import Zopti

dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(dir)

import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin(path=dir + r"\FrameRateConverter-x64.dll")

zopti = Zopti(r'results.txt', metrics=['ssim', 'time'])

file="MotionTest2_scaled.avi"
video = core.ffms2.Source(source=file, format=vs.YUV420P8)

goodmask = core.ffms2.Source(source="StripeMask_good.avi")
goodmask = core.std.ShufflePlanes(goodmask, planes=0, colorfamily=vs.GRAY)

gamma = 220/100.0	# optimize gamma = _n_/100.0 | 1..300 | gamma
min_out = 65		# optimize min_out = _n_ | 0..100 ; max:max_out 1 - | min_out
max_out = 190		# optimize max_out = _n_ | 100..255 ; min:min_out 1 + | max_out
thr = 15		# optimize thr = _n_ | 1..100 | thr

video = core.std.ShufflePlanes(clips=video, planes=0, colorfamily=vs.GRAY)
video = video.std.Levels(gamma=1/gamma, min_in=16, max_in=235, min_out=min_out, max_out=max_out) # range=125
video = core.frc.StripeMask(video, blksize=16, thr=thr, range=max_out-min_out, skip=1)

frame = 20
video = video[frame]
goodmask = goodmask[frame]

zopti.run(goodmask, video)
So there are four optimized parameters: gamma, min_out, max_out and thr. I didn't include min_in or max_in because those should be correct for this video.

Gamma range is 1 - 300 divided by 100 so effectively 0.01 - 3.0 in steps of 0.01. max_out and max_in have dependencies, max should always be larger than min so I set a min limit for max_out and max limit for min_out.

I only measured SSIM on one frame to save time and just to see if even one frame can be matched well (if we can't do that then doing it for the whole clip is hopeless). The frame number 20 contained interesting mask features so that's the one I chose. I later found out that if the result matches well with this one frame it also matches well with the rest of the frames.

I ran a couple of Zopti tests with
Code:
zopti StripeMaskTest.vpy -alg mutation -iters 1000 -pop 48 -threads 6
and then set the iterations to 10000 and population to 30. With VapourSynth you don't need that many threads to saturate all cores, with Avisynth I usually run -threads 16 or 24 (and 16 usually only when there isn't enough memory to run more).

The best result was 0.9778275 so already quite promising and I verified visually that it's pretty close. You can output the best result script with
Code:
zopti -mode evaluate -scripts best
Next I wanted to see some heatmaps to better understand where the good values are. The first one was for max_out and thr.



There's a nice linear area of good values. In theory you could find a good match with any thr or max_out value as long as you adjust the other one. The red dot is the best found result.

The next one is min_out and max_out.




This revealed to me what you probably already knew - that the absolute values of min_out and max_out don't matter, only their distance (which is the range). Ok, this means I can run the next tests with min_out=0 and leaving only gamma, max_out and thr to optimize.

What about gamma + min_out?



That's odd... there's kind of checkerboard pattern going on. Changing gamma slightly makes the result go up and down. Let's take a closer look at the gamma alone:



Oh wow, no wonder you had trouble finding a good match, the gamma behaves quite erraticly! I wonder what's going on, is the up-down swing also there in the brightness of the image? I don't know yet.

Meanwhile I ran some tests with 100 000 iterations and using more focused search ranges. I decreased the gamma step to 0.001. I had trouble finishing those runs as the execution would sometimes stop with error. Once I got

Code:
Script exceeded memory limit. Consider raising cache size.
terminate called after throwing an instance of 'std::out_of_range'
  what():  _Map_base::at
even though I had plenty of memory left (about 40 gigs). Usually the script just reports
Code:
Output 1 frames in 0.62 seconds (1.61 fps)
or similar. Zopti has a tendency to find rare bugs so this could be a sign of that.

Anyway, two complete and one almost-complete runs later they all found the best result 0.98015726 so I think that's as good as it gets. I think giving gamma even more decimals could improve things a bit so that's something I'll try.

I looked at some heatmaps of these 100000 iteration runs and they look like they're straight out of a horror film:



I have no idea why there are those vertical stripes but I know they use different thr than the best found value. First I thought they are just some randomness but the stripes were found in every one of the three runs at the same locations. Investigation continues...
zorr is offline   Reply With Quote
Old 6th July 2021, 03:49   #431  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
If you took the v1.3 results as reference, those aren't perfect at all. Perhaps you could manually draw the areas in Photoshop as a perfect reference mask, and see how close the tests get. Comparing to v1.3, there are random fluctuations of areas that are included or excluded that would randomize the test results. You need a fixed and clean reference.

As for performance, it could be optimized by doing this in a script function:
- Create 2 clips with 1-frame offset
- Output with 255 (pure black & white)
- Apply each mask over a blank clip of the desired colour
- Merge the 2 masks together, taking the highest value of each clip

Could someone write that function? It should double the performance.

Did you try sharpening just out of curiosity? Anything else that could be tried?

P.S. your last test, it's a galactic war between two fleets. The Dark Fleet tried to take over the galaxy.

The gamma could be altering the way stripe patterns are being detected or not past certain thresholds. Your white bars kind of look like the stripes we're trying to detect. If playing with the gamma considerably alters the results, there's also the option of running the mask different times with various gamma values and merging them. Or a parameter: run with 1, 2 or 3 passes.

Last edited by MysteryX; 6th July 2021 at 06:13.
MysteryX is offline   Reply With Quote
Old 6th July 2021, 10:00   #432  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,250
Quote:
Oh wow, no wonder you had trouble finding a good match, the gamma behaves quite erraticly!
Rounding errors/artefacts?
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is online now   Reply With Quote
Old 7th July 2021, 10:23   #433  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
This function gives just over 25% performance gain with strf
Code:
def StripeMask2(clip, blksize = None, blksizev = None, overlap = None, overlapv = None, thr = None, range = None, comp = None, compv = None, str = None, strf = None, lines = None, skip = None):
    if not isinstance(clip, vs.VideoNode):
        raise vs.Error('StripeMask: This is not a clip')

    mask = clip.frc.StripeMask(blksize=blksize, blksizev=blksizev, overlap=overlap, overlapv=overlapv, thr=thr, range=range, comp=comp, compv=compv, str=str, strf=0, lines=lines, skip=skip)
    if strf > 0:
        maskf = mask.std.DeleteFrames(frames=[0]).std.DuplicateFrames(frames=[clip.num_frames-2])
        return core.std.Expr(clips=[mask, maskf], expr=f"x {str} y {strf} 0 ? ?")
    else:
        return mask

Last edited by MysteryX; 7th July 2021 at 10:43.
MysteryX is offline   Reply With Quote
Old 8th July 2021, 00:29   #434  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
I ran one more test to see how much even more decimals for gamma helps. Not much, best result was 0.98015726 with gamma 1/2.0778.

Quote:
Originally Posted by wonkey_monkey View Post
Rounding errors/artefacts?
I decided to take a closer look at how the gamma behaves. This is the average brightness of the source frame 20 with gammas between 1/2.05 and 1/2.1.



So while it's not buttery smooth it doesn't look like it's doing anything radical. If we draw a straight line from gamma 1/2.05 to gamma 1/2.1 and calculate the offset it looks like this:



Not pretty but the errors are small.

Quote:
Originally Posted by MysteryX View Post
If you took the v1.3 results as reference, those aren't perfect at all. Perhaps you could manually draw the areas in Photoshop as a perfect reference mask, and see how close the tests get.
Yes this is just a starting point, at least we now know that there is nothing fundamentally wrong with the VS version as it's capable of emulating the Avisynth version.

Doing one or more masks manually and letting Zopti play with the parametes is probably a good next step. With Zopti you can develop algorithms on the idea level not having to bother with the gruesome testing of whether this or that idea actually delivers. Of course setting up the test scripts and running them still takes time but it's still tremendously helpful.

Quote:
Originally Posted by MysteryX View Post
Comparing to v1.3, there are random fluctuations of areas that are included or excluded that would randomize the test results. You need a fixed and clean reference.
Yes there are some differences, curiously the VS version usually makes the mask one pixel wider than Avisynth. I'm not sure what you mean by fixed and clean reference. Using the same clip and frame counts as fixed, right?

Quote:
Originally Posted by MysteryX View Post
Did you try sharpening just out of curiosity? Anything else that could be tried?
Didn't try sharpening, that (and blurring) should be tested as well. My first idea on how to improve the mask was to remove the smaller areas using the RemoveSmallSpots-function I used with Delta Restore. And likewise fill the small holes of the mask using a similar FillSmallHoles -function. I also tried offsetting the image half blksize and taking another StripeMask and combining it with the original. This seems to stablilize the mask somewhat just like using strf. Here's the script (including aforementionded functions) I played with:

Code:
src = FFVideoSource("MotionTest2.mp4")
src = src.BicubicResize(src.width / 2, src.height / 2)

blksize = 16
thr=20
offset = 8
spotSize = 14
holeSize = 20
finalHoleSize = 22

src2 = src.AddBorders(offset,0,0,0)

mask = src.StripeMask(thr=thr, blksize=blksize)
mask2 = src2.StripeMask(thr=thr, blksize=blksize)
mask2 = mask2.Crop(offset,0,0,0)
mask2 = mask2.RemoveSmallSpots(spotSize).FillSmallHoles(holeSize)

mask = mask.RemoveSmallSpots(spotSize).FillSmallHoles(holeSize)
mask_final = mt_logic(mask, mask2, "and")
mask_final = mask_final.RemoveSmallSpots(spotSize).FillSmallHoles(finalHoleSize)

return src.Overlay(mask_final, opacity=0.5)


function FillSmallHoles(clip mask, int holeSize) {
	return mask.mt_invert().RemoveSmallSpots(holeSize).mt_invert()
}

function RemoveSmallSpots(clip mask, int spotSize) {
	mask_orig = mask

	# remove too small spots
	for (i = 1, spotSize) {
		mask = mask.mt_inpand()
	}
	mask = mt_hysteresis(mask, mask_orig, chroma="-128") 
	return mask
}
Perhaps the area around the found mask should be allowed to match the stripes with a lower threshold, that way some of those cracks in the mask could be filled. This can be implemented by running another StripeMask with lower threshold and anding it with the original which has been expanded about half blocksize.

Quote:
Originally Posted by MysteryX View Post
P.S. your last test, it's a galactic war between two fleets. The Dark Fleet tried to take over the galaxy.


Quote:
Originally Posted by MysteryX View Post
there's also the option of running the mask different times with various gamma values and merging them. Or a parameter: run with 1, 2 or 3 passes.
Yes, good ideas that could be tested.
zorr is offline   Reply With Quote
Old 8th July 2021, 05:12   #435  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
Quote:
Originally Posted by zorr View Post
Yes there are some differences, curiously the VS version usually makes the mask one pixel wider than Avisynth.
Mask areas are wider, or the output clip is wider?

Quote:
Originally Posted by zorr View Post
I'm not sure what you mean by fixed and clean reference. Using the same clip and frame counts as fixed, right?
I mean, creating a mask manually that covers 100% of the right areas with no false positive.

Quote:
Originally Posted by zorr View Post
I also tried offsetting the image half blksize and taking another StripeMask and combining it with the original. This seems to stablilize the mask somewhat just like using strf.
strf stabilizes in time (after all, the mask hides all generated frames between this frame and the next), whereas this will help fill spacial holes. It's actually a really good idea! Cutting the image in half, and working on a single grey plane, the performance hit will be minimal.

Small spots will be removed by ContinuousMask. mt_expand fills small holes.

Last edited by MysteryX; 8th July 2021 at 06:47.
MysteryX is offline   Reply With Quote
Old 9th July 2021, 00:07   #436  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
Quote:
Originally Posted by MysteryX View Post
Mask areas are wider, or the output clip is wider?
The mask, and just the right edge. It's actually two pixels, and sometimes the bottom edge is one pixel wider. Like this:



Quote:
Originally Posted by MysteryX View Post
I mean, creating a mask manually that covers 100% of the right areas with no false positive.
Oh yes. And that's what I did. Well, perhaps not 100% right areas as that depends on how MaskTools sees the frame (where the stripes are causing trouble). But I did roughly select the fence, if StripeMask can do that it's going to be a significantly improved result. I chose 4 frames, numbers 5, 20, 32 and 72. This is the mask in frame 5:



I've started running Zopti on this new challenge, first using the same three params gamma, max_out and thr. Initial results indicate that this is a much harder problem.

Quote:
Originally Posted by MysteryX View Post
It's actually a really good idea! Cutting the image in half, and working on a single grey plane, the performance hit will be minimal.
Not sure I follow but I'm glad I was able to help!

Quote:
Originally Posted by MysteryX View Post
Small spots will be removed by ContinuousMask. mt_expand fills small holes.
Ok, we just have to test the ContinuousMask as well. Actually we should follow the whole procedure in the .avsi and include new parameters from there.

mt_expand will also make the mask larger, if you only want the holes filled then it must be combined with mt_hysteresis. But that comes with a performance penalty (I haven't measured but since mt_hysteresis is a fill algorithm it's probably not super fast) and might not be worth it.

Last edited by zorr; 9th July 2021 at 00:28. Reason: can't spell
zorr is offline   Reply With Quote
Old 9th July 2021, 03:44   #437  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,283
Quote:
Originally Posted by zorr View Post
The mask, and just the right edge. It's actually two pixels, and sometimes the bottom edge is one pixel wider. Like this:
Is that between VapourSynth and Avisynth or between v1.3 and v2.0? I did fix a bug where sometimes the area to mark wasn't closed properly and a huge bar would appear; so I did a change to the closing of the masked area. That could be the pixel difference.

I'm not sure it's worth testing the whole procedure. The rest of the AVSI code is to give the shape I want for a mask. I do want it larger so that artefacts are properly hidden.

What I really need is just the right thr, range and gamma parameters. If doing 2 or 3 passes with different values gives a better mask, then need to find which set of params to use. Also do these parameters need to be adjusted for various videos, or are the right values right for everything?

We're better to include more and have a bit more false positive here; the small spots get removed with ContinuousMask, and the most important is for stripe areas to definitely be masked because they're UGLY otherwise.

In terms of fine-tuning the analysis, I think we've got enough data to implement something.

Unless someone wants to write machine-learning AI for stripe detection? If it's not over-kill.

Last edited by MysteryX; 9th July 2021 at 05:38.
MysteryX is offline   Reply With Quote
Old 12th July 2021, 01:28   #438  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
Quote:
Originally Posted by MysteryX View Post
Is that between VapourSynth and Avisynth or between v1.3 and v2.0? I did fix a bug where sometimes the area to mark wasn't closed properly and a huge bar would appear; so I did a change to the closing of the masked area. That could be the pixel difference.
Yes between v1.3 and 2.0. And that's a very logical explanation.

Quote:
Originally Posted by MysteryX View Post
What I really need is just the right thr, range and gamma parameters.
Here's some results, using the default StripeMask parameters except for thr. The best SSIM found was 0.3176762 so much lower than before, especially since the maximum possible is now 4.0. I did two 100k iteration runs.

This is the best mask for each of the four test frames (target is overlaid in red).



The best found combo was gamma 1/1.811, max_out 247 and thr 16.

I'm leaving for a little trip for a couple of days, I'll adress the rest of your comments after that.
zorr is offline   Reply With Quote
Old 14th July 2021, 22:42   #439  |  Link
zorr
Registered User
 
Join Date: Mar 2018
Posts: 331
Quote:
Originally Posted by MysteryX View Post
I'm not sure it's worth testing the whole procedure. The rest of the AVSI code is to give the shape I want for a mask. I do want it larger so that artefacts are properly hidden.
...
We're better to include more and have a bit more false positive here; the small spots get removed with ContinuousMask, and the most important is for stripe areas to definitely be masked because they're UGLY otherwise.
Perhaps the whole procedure is not needed but anything that ends up doing more than just cosmetic changes (like blurring) to the final mask should be included, otherwise we're measuring the wrong thing. For example if the ContinuousMask is left out then we won't know how much larger the mask needs to be for it to be optimal for ContinuousMask.

I actually tried ContinuousMask but seems like it's not designed to remove large areas completely. Then I did some tests with RemoveSmallSpots to see how including it will change the optimal gamma, max_out and thr. The best result was 0.32462016 with gamma=1/1.763, max_out=225, thr=13 and spotsize=20. So removing spots afterwards seems to change the optimal values.

Here's what the optimal mask looks like with RemoveSmallSpots:



It looks much cleaner than the previous attempt.

After this I also wanted to see if changing the defaults of parameters blksize, overlap and comp will give better results. This resulted in SSIM of 0.43589097 using gamma=1/1.345, max_out=230, thr=14, blksize=27, overlap=26, comp=5 and spotsize=38.

Something I observed: optimal overlap is always the largest possible (blksize-1) and optimal comp is the largest supported 5.

After that I also included filling the small holes in the mask using FillSmallHoles and the best result was 0.45771345 using gamma=1/1.034, max_out=204, thr=9, blksize=24, overlap=23, comp=5, spotsize=61 and holesize=43.

But while the SSIM is now much better the mask looks way too large, it seems SSIM is not measuring this the way we want:



So I think the next step is to replace SSIM with something better. As you mentioned it's very important to cover all of the striped areas we are better off measuring that directly - how large percentage of the target area is covered. But in addition we need another measurement to determine how much larger the mask is than what is needed to cover the target (otherwise it would be easy to get perfect score by making the mask fill 100% of the screen), this one should be minimized. Zopti does multiobjective optimization by default so this is not a problem - the results will be a pareto front with increasing target mask cover and increasing overflow area and we just need to pick the one that is the best compromise. I have already added custom props to ZoptiLib so that the similarity metric(s) can be calculated in the script.

Quote:
Originally Posted by MysteryX View Post
If doing 2 or 3 passes with different values gives a better mask, then need to find which set of params to use. Also do these parameters need to be adjusted for various videos, or are the right values right for everything?
These are also interesting to test and we'll get to those once the similarity metric has been fixed. We'll need some more test clips in order to determine if one set of values works well. I think one of those could be the one I'm doing MVTools tests with.

Quote:
Originally Posted by MysteryX View Post
Unless someone wants to write machine-learning AI for stripe detection? If it's not over-kill.
I'd like to try frequency domain filters such as DeFreq to create a stripe mask. The filter is not meant for creating masks but the output can be turned into a mask quite easily.
zorr is offline   Reply With Quote
Old 15th July 2021, 05:23   #440  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 577
@zorr, this is FrameRateConverter.avsi support HBD
https://github.com/Dogway/Avisynth-S...Converter.avsi
kedautinh12 is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:42.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.