MVTools-pfmod

[flossy_cake]

Originally Posted by DTL

May be you need mean of MVs length in the current pairs of frames analysed ?

Yes, I think so. I am just trying to get a number which describes how much motion there is between 2 frames. Bigger number = more motion, smaller number = less motion. I need the result to be accessible in an Avisynth script variable so I can make conditional decisions on it in the rest of my script - I am not sure if MVTools was designed for this. Looking at the wiki, it seems MVTools stores all the motion vector information in its own bespoke format which can only be used with other modules of the MVTools package. Or perhaps it can be output to a file on disk and then read that information back in from file in realtime.

[DTL]

You can simply make 2 frames equal (Trim(1,1).Loop()) and after MAnalyse put MStoreVect() and look into output clip in 2D viewer with RGB values display for each sample. Try to use some simple sample format like RGB24. After some header in the first line you will see 0,0,0 for each dx,dy,SAD because frames are equal and no motion.

Now take first (top-left) block of size set in MAnalyse and shift to 1 sample left, right, top, bottom and see how RGB data changes. The blocks scanned from left to right and from top to bottom of frame. So you can found where header data ends and simple contigous list of VECTOR data starts and where 1st block data starts. The data is of 32bit integers so takes 4x8bit values. So you collect RGB triplets from frame returned by MStoreVect in script environment and decode it into 2D array of blocks with their VECTOR structure data (dx, dy, SAD). And use dx and dy to calculate each MV length (or squared to be faster and not take square root of slow op) - Length^2 = dx^2 + dy^2. And calculate mean value of MV length over all blocks in frame (width/blocksizeH x height/blocksizeV).

The analysis data header is fixed and the w and h size of the encoded frame is selected by some algorithm. You can skip header fixed and the rest should be 3x32bit VECTOR data for all blocks. If clip sample format is RGB24 these 32bit values are shifted over RGB24 samples and if set to RGB32 - you may be (or not) lucky to get each dx, dy and SAD into RGB32 sample data or each data will have equal start shift by 8bits.

If someone can access sample data from frame (Eval ? or some other function) - he can make simple enough decoder script to get each block VECTOR data from clip produced my MStoreVect(). And make math calculations with dx,dy and SAD data.

The only helper filter for script writers may be created to make 1st line with header data fixed and start 2nd (or other line if blocks number in line is too small) line in RGB32 format from dx data of 1st block in the frame and so on, So the width of a pseudo-clip frame will be number of blocks in line x 3x32bit data (x3 RGB32). So for 1920 width clip and 8x8 block size blocks number per line is 240 and width of MStoreNiceVect will be 240x3=720 and each RGB32 4bytes samples value will keep dx, dy, SAD values for blocks (VECTOR structure for each block). And number or rows (skipping header-row(s)) will be equal to V-number of blocks. It will make script-decoder more simple.

[anton_foy]

request:
A "Threshold" option.
Very often denosing/degraining needs to be stronger in the darker parts (heavier nosie) and less in the brighter ones (lighter noise).
It would be incredibly benificial IMHO to have this option as a limiter of strength when setting a threshold. From default 1.0 to 0.0 or 255-0 or so.
Preferably a soft threshold like a roll-off curve rather than a hard pointy one.

This can already be achived externally but it is very slow and if made internally in mvtools I think it could be much faster in speed.

example:

bright=smdegrain(tr=3,thsad=200)
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,255)
mt_merge(dark, bright, mask)

or a simpler limiting example:

original=last
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,200) #merge back some denoised together with original by adding 200 on the last param for better visual appearence.
mt_merge(dark, original, mask)

By now I hope I made my idea clear.
A threshold parameter (max 1.0 min 0.0 or 255-0 to adjust how bright it should limit degraining) and a thresholdlimit/thresholdstrength parameter (max 1.0 min 0.0 to adjust strength of degrain in the thresholded areas).
Strength of degraining/denoising can be argued if to blend with the original (simplest but less visually appealing) or to somehow "choke" the strength like rgtools repair or the "limit" parameter in mvtools (although the current parameter limit=1 seems not to lower the degraining very much?).

Pinterf what do you think about this?

Thanks!

[DTL]

Short answer (long may be later):

I think you can start from moving 1 level of AVS usage lower at first: Not level of user-provided script function (SMDegrain) but level of manual script writing using plugin functions.

Most quality (per-sample level) can run only at after-MDegrainX processing. So it is not very benefitical in performance to implement user-script in plugin.

Current suggestion:

Put this idea:
bright=smdegrain(tr=3,thsad=200)
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,255)
mt_merge(dark, bright, mask)

In manual script (or ask Dogway to add to SMDegrain):

MDegrainN can run at provided tr-value (if not - it can be easily fixed).

So you better in performance is simply run 1 MAnalyse and several MDegrainN and combine in several layers with masking.

Some like:

super=MSuper()
mv=MAnalyse(tr=max(my_tr1, my_tr2),..)

dg1=MDegrainN(super, mv, tr=my_tr1, thSAD=my_thSAD1)
dg2=MDegrainN(super, mv, tr=my_tr2, thSAD=my_thSAD2)

mask=MyCustonMaskFunc(...)
mt_merge(dark, dg1, dg2)

It will be better in performance because of 1 MAnalyse with max_tr only. Typically MAnalyse is most of time-taking process.

[anton_foy]

Originally Posted by DTL

Short answer (long may be later):

I think you can start from moving 1 level of AVS usage lower at first: Not level of user-provided script function (SMDegrain) but level of manual script writing using plugin functions.

Most quality (per-sample level) can run only at after-MDegrainX processing. So it is not very benefitical in performance to implement user-script in plugin.

Current suggestion:

Put this idea:
bright=smdegrain(tr=3,thsad=200)
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,255)
mt_merge(dark, bright, mask)

In manual script (or ask Dogway to add to SMDegrain):

MDegrainN can run at provided tr-value (if not - it can be easily fixed).

So you better in performance is simply run 1 MAnalyse and several MDegrainN and combine in several layers with masking.

Some like:

super=MSuper()
mv=MAnalyse(tr=max(my_tr1, my_tr2),..)

dg1=MDegrainN(super, mv, tr=my_tr1, thSAD=my_thSAD1)
dg2=MDegrainN(super, mv, tr=my_tr2, thSAD=my_thSAD2)

mask=MyCustonMaskFunc(...)
mt_merge(dark, dg1, dg2)

It will be better in performance because of 1 MAnalyse with max_tr only. Typically MAnalyse is most of time-taking process.

Aaah so clever! Of course this is a great idea thank you. So to implement this into mvtools internally would not be faster you think?

Edit:
I did not know this was possible:

mv=MAnalyse(tr=max(my_tr1, my_tr2),..)

Great, I think actually this will help many in retaining more of the original detail where grain/noise is not as heavy in the highlights.

[DTL]

"this was possible:
Quote:
mv=MAnalyse(tr=max(my_tr1, my_tr2),..)"

Well - it is at least must be possible by logic of mvtools plugins but currently may still not work as it should. But it is really very small fix and the most probable pinterf will agree it.

Current status: If tr of MAnalyse != tr of MDegrainN it may cause MVs frame request mismatch because the requesting pseudo-frame number from MDegrainN to MAnalyse is about current_frame x 2 x tr. So the logic in MAnalyse in multi=true decode required current + ref frame pair to search from N frame requested + multi-flag + tr-provided. If tr of MAnalyse and tr of MDegrainN not match - it currently looks cause not correct processing.

But as I see pinterf already add analysis_data structure to MVclip and if it have (easy to add) the MAnalyse tr-value and take it in MDegrainN - the MDegrainN can encode proper N-frame number to request if even tr-value of MAnalyse and MDegrainN not equal.

" to implement this into mvtools internally would not be faster you think?"

It will be somehow faster because of more processing with single memory read-write data. But to implement it - the completely separate processing mode in MDegrainN need to be added. To make dual-tr and dual-thSADs blocks processing and some blending/mixer based on the internal or better external mask clip provided. Also 2.7.45 MDegrainN have only separated processing functions for luma and chroma planes (so need to double number of processing functions). Or attempt to make even more complex one_for_all (templated for better performance ?) processing function of single and dual params processing depending of the current running MDegrainN mode.

It require to add lots of new control params to MDegraiN like:
1. Enable/disable new mode
2. Input mask clip (Y8 ? Y16 ?) param
3. tr2
4. thSAD_second (+Chroma)
5. thSAD2_second (+Chroma)
6.... - really many of possible MDegrainN params need to be doubled (also added in possibly ported features from my post-2.7.45 builds).

So it currently much more easier to implement it as 1 MAnalyse + 2 (or more MDegrainN) and custom mask engine + some blending engine in scripting. It is a completely new mode to test and typically only if most users will like to use it for (months ? years ? decades ?) it possibly may be put to compiled form (also with possible minimul new params or some selected params by users practic accumulated in months/years/decades of scripted form usage).

The only simple fix reqiured to run several MDegrainN with different tr-value using single MAnalyse source. It is really benefitical in performance if (!?!) AVS core will cache already requested MVs frames to 2 or more MDegrainN filters running after single MAnalyse MVs-source. But AVS cache around MAnalyse source may be one more unknown variable - it may be limited to some tr-values only and depend on Prefetch() local and/or global script settings ? I not sure how many MV-frames after MAnalyse will be cached depending on AVS core settings and may be other config params somewhere. But may be for tr=3 all required MV-frames will be cached at most of use cases.

" a simpler limiting example:
Code:
original=last
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,200) #merge back some denoised together with original by adding 200 on the last param for better visual appearence.
mt_merge(dark, original, mask)"

This idea may be easier to add as new post-processing engine in addition to 'limit' current post-processing. So it may be programmed as separated module and not touch internal MDegrainN core (of any version). But it may be somehow (may be much) worse in quality in compare with dual-MDegrainN runs with different sets of params. Also may be better to add new input clip param of mask from external mask source (so user will not be limited with internal masking only). But also it may be not significantly faster (additional postprocessing pass really again read-write all frame memory) and only may save some RAM in AVS caching.

[anton_foy]

It require to add lots of new control params to MDegraiN like:
1. Enable/disable new mode
2. Input mask clip (Y8 ? Y16 ?) param
3. tr2
4. thSAD_second (+Chroma)
5. thSAD2_second (+Chroma)
6.... - really many of possible MDegrainN params need to be doubled (also added in possibly ported features from my post-2.7.45 builds).

Sure there could be alot of params depending on this but as I figure only "tr" would suffice. Maybe "thSAD" aswell but I do not think it is needed. But even the limiting by merging the original with mdegrain by a brightness threshold would be great. As you say, the best quality would be two different mdegrain calls merged, yet I have to try the benchmarks to see if it will be alot slower than only one mdegrain call.

Edit: and there is already mmask in mvtools, so why not lumamask

[DTL]

"Maybe "thSAD" aswell but I do not think it is needed."

It is typically still work as major protection from too bad blends (causing blurring) where transform compensation is not perfect. So at the lower-nosied medium and high code values it is expected to have both lower tr and lower thSAD - so it will save more details from typicaly possible blurring.

[anton_foy]

Originally Posted by DTL

"Maybe "thSAD" aswell but I do not think it is needed."

It is typically still work as major protection from too bad blends (causing blurring) where transform compensation is not perfect. So at the lower-nosied medium and high code values it is expected to have both lower tr and lower thSAD - so it will save more details from typicaly possible blurring.

Okay fair enough

[DTL]

One more idea with possibly fastest implementation in processing and simple enough in design:

If quality of block-based weighting of current (source, full input noise and details) frame in output blending result will be acceptable - it can be added reading of simple one more input mask clip (in full-frame size or blocks number H and V size only) and simply additionally correct central/current block weight in blending weights after DegrainWeight() call and before normal_weights() call. The correction may be inverse multiplication of all ref-frames blocks between current weight after DegrainWeight() call and zero weights. So in mask clip range 255..0 may be mapped to 1.0..0 float additional multiplier to all ref-frames weights except current frame weight. (May be with additional check if current frame weight !=0). So if mask is 0 - all ref frames weight will be put to zero and norm_weights() will set current frame block only to output (non-changed input). It will make close to zero penalty to performance and keep total implementation very simple.

Also it will make MDegrainN more flexible to use in complex scripts - it someone can make mask of badly blurred areas (blocks) - this mask can be applied as additional input to fix weights of badly transform-compensated ref frames and cure these areas from blurring at next MDegrainN processing (or at final output if mask will be ready before it).

It will be close to

original=last
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,200) #merge back some denoised together with original by adding 200 on the last param for better visual appearence.
mt_merge(dark, original, mask)

processing but block-based (not sample-based).

(Registered as feature number 44 in my list of post-2.7.45 build features).

[anton_foy]

Originally Posted by DTL

One more idea with possibly fastest implementation in processing and simple enough in design:

If quality of block-based weighting of current (source, full input noise and details) frame in output blending result will be acceptable - it can be added reading of simple one more input mask clip (in full-frame size or blocks number H and V size only) and simply additionally correct central/current block weight in blending weights after DegrainWeight() call and before normal_weights() call. The correction may be inverse multiplication of all ref-frames blocks between current weight after DegrainWeight() call and zero weights. So in mask clip range 255..0 may be mapped to 1.0..0 float additional multiplier to all ref-frames weights except current frame weight. (May be with additional check if current frame weight !=0). So if mask is 0 - all ref frames weight will be put to zero and norm_weights() will set current frame block only to output (non-changed input). It will make close to zero penalty to performance and keep total implementation very simple.

Also it will make MDegrainN more flexible to use in complex scripts - it someone can make mask of badly blurred areas (blocks) - this mask can be applied as additional input to fix weights of badly transform-compensated ref frames and cure these areas from blurring at next MDegrainN processing (or at final output if mask will be ready before it).

It will be close to

original=last
dark=smdegrain(tr=7,thsad=500)
mask=levels(32,1.0,190,0,200) #merge back some denoised together with original by adding 200 on the last param for better visual appearence.
mt_merge(dark, original, mask)

processing but block-based (not sample-based).

(Registered as feature number 44 in my list of post-2.7.45 build features).

Great idea. So that could minimize the need for time consuming 2nd+ generation of mdegrain by instead merging it as you explain?

[DTL]

It is single pass MDegrainN as it run today. But it still require some external engine to generate mask clip. Simple mask may be inverted Y channel (and for best performance first downsize to H and V block numbers size). Having block-number sized mask clip also skips possible questions about sample value to use if several samples per block provided. So may be put strict requirement for mask clip to be sized of H and V blocks number. So single Y8 sample per block to use.

In the above examples the 2nd MdegrainN is not 2nd generation - is 2nd parallel running with single MVs source. But that method expected to provide better quality.

[anton_foy]

Originally Posted by DTL

It is single pass MDegrainN as it run today. But it still require some external engine to generate mask clip. Simple mask may be inverted Y channel (and for best performance first downsize to H and V block numbers size). Having block-number sized mask clip also skips possible questions about sample value to use if several samples per block provided. So may be put strict requirement for mask clip to be sized of H and V blocks number. So single Y8 sample per block to use.

In the above examples the 2nd MdegrainN is not 2nd generation - is 2nd parallel running with single MVs source. But that method expected to provide better quality.

Yes I see but still gains better quality and you agree the blending/merging is a great feature to implement?
Yet it can even be 2 mdegrain calls stacked to get 2nd generation while a brighter is only using 1st gen to use for the brights?

[DTL]

"Yes I see but still gains better quality and you agree the blending/merging is a great feature to implement?"

I agree it can be useful feature to implement.

"Yet it can even be 2 mdegrain calls stacked to get 2nd generation while a brighter is only using 1st gen to use for the brights?"

Yes - we can serialize 2 mdegrains with single MV source and different params and use 1st and last output as inputs to some weighted mixer. There are many ways to implement required processing and it is better to test first in easy to design scripting form the good or better in quality.

[Fjord]

@DTL: Thanks for your development work on MVTools!

Will your multi-generation MV approach work with the existing v2.7.45 MVTools? Or are new processing and/or parameters in your v2.7.46_xxx work-in-progress essential for multi-generation MVs to work?

Your "RefineMV()" function includes new parameters in calls to MDegrainN() and MAnalyse() - such as wpow, adjSADzeromv, adjSADcohmv, ... I can see the variable types in interface.cpp on github, and mention of some of them in your new_features_list.ods spreadsheet, but I haven't yet found any description of these new parameters (purpose, valid values, default value). Do you have some basic description for your new parameters somewhere that I have missed?

I have only tried your latest release v2.7.46_e.02.zip (dated 20230811), with understanding this only contains "compatible" changes to v2.7.45 version. Apparently _e.02 does not include the new parameters used in your multi-generation MV approach, since I get avisynth script errors like "MAnalyse does not have a named argument "optSearchOption".

I have 2k 10-bit YUV422 source videos (high-noise/grain film scans), where BlkSize 32 and 64 are actually useful. But from reading your posts I understand you have only implemented 8-bit operations with 8x8 Blksize. Is that correct? If so I need to stay with MVTools v2.7.45. (and that is why I ask if multi-generation MV approach will work with pinterf's release.) My application uses MCompensate() where refined MVs might help.

[DTL]

"Will your multi-generation MV approach work with the existing v2.7.45 MVTools? "

Possibly not because it is based on MAnalyse with separated inputs for current and ref clips (frames).

In august 2023 I made some special builds tagged as 2.7.46-e.xx - they are based on the latest sources from the pinterf repository (with some commits after 2.7.45 release) and included very few of new and important features. Latest is e.02 - https://github.com/DTL2020/mvtools/releases . So expected to be stable close to 2.7.45 release and have some few possibly not critical to stability features. It has SuperCurrent input for MAnalyse so can be used in multi-generation approach as described.

"Or are new processing and/or parameters in your v2.7.46_xxx work-in-progress essential for multi-generation MVs to work?"

Most important feature is the dual-input MAnalyse filter. Other features of post-2.7.45 builds may add to quality (like MVLPF processing) but not act as the key component of the multi-generation refining idea.

"Your "RefineMV()" function includes new parameters in calls to MDegrainN() and MAnalyse() - such as wpow, adjSADzeromv, adjSADcohmv, ... I can see the variable types in interface.cpp on github, and mention of some of them in your new_features_list.ods spreadsheet, but I haven't yet found any description of these new parameters (purpose, valid values, default value). Do you have some basic description for your new parameters somewhere that I have missed?"

I made some update of HTML documentation but it still not include all latest features added - https://github.com/DTL2020/mvtools/b.../mvtools2.html . Also typically every new added param described in the release description where it was added. So one way to found at least short description is to look into every release description also some extended description sometime added in the posts in the thread with new releases announce - https://forum.doom9.org/showthread.php?t=183517

"I have only tried your latest release v2.7.46_e.02.zip (dated 20230811), with understanding this only contains "compatible" changes to v2.7.45 version. Apparently _e.02 does not include the new parameters used in your multi-generation MV approach, since I get avisynth script errors like "MAnalyse does not have a named argument "optSearchOption"."

You can skip all new params except supercurrent clip for MAnalyse.

"only implemented 8-bit operations with 8x8 Blksize. "

Some of the many new features have only implementation or SIMD faster implementation or were tested with 8x8 block size and 8bit. To display all new features and compatibility with different blocksize/bitdepth combinations someone needs to build a table of current state.
The only required for multi-generation approach feature of dual-input MAnalyse (added SuperCurrent clip input) expected to be compatible with any blocksize/bitdepth.

Expected smallest working RefineMV() function with e.02 build is

Function RefineMV(clip mvclip, clip super_ref, clip src)
{
g_next=MDegrainN(src, super_ref, mvclip, tr, thSAD=my_thSAD, thSAD2=my_thSAD2, mt=false, thSCD1=my_thSCD)

super_g_next=MSuper(g_next,chroma=true, mt=false, pel=my_pel)

return=MAnalyse(super_g_next, SuperCurrent=super_ref, multi=true, delta=tr, search=3, searchparam=2, trymany=my_trymany, \
overlap=0, chroma=true, mt=false, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=my_global)
}

And usage:
my_src=last
s=Super(my_src)
mv1=MAnalyse(s)
mv2=RefineMV(mv1, s, my_src)
and so on

"(high-noise/grain film scans), where BlkSize 32 and 64 are actually useful"

I think 'large' blocksizes really works better on film scans because typically they are not only grainy but also very soft. So blocksize 8x8 have too few details to use in good search. And the higher the filmscan resolution the more softer the digital result (and larger blocksize recommended). But for refining of motion after first search stages with large blocksize - the refining MVs with MRecalculate to lower blocksize may be recommended.

[Fjord]

Thanks DTL for the detailed reply. I will look at your revised MVTools2.html and file history for info on new features and parameters.

Thanks for the minimal multi-generation MV script, for your v2.7.46_e.02 (20230811) release. I will give it a try.

[DTL]

"and file history for info on new features and parameters."

It looks I was lazy to put description in single history file - so only available in github releases description (in each version release). https://github.com/DTL2020/mvtools/releases . Will try to collect descriptions from each release in single history file later.

In the script example my_* params are script 'globals' - I found them useful when adjusting MAnalsye for degraining. Also overlap may be set to > 0 for better quality (and even lower performance). I left overlap=0 from my newer versions where I use interpolated overlap in MDegrainN. Really best quality is with full 'real' 4x overlap in MAnalyse (when overlap = bloksize/2).

With high quality settings like trymany=true and overlap=4 (and blksize default of 8x8) even 2 generatins of MVs (single RefineMV call) takes about 10 days of processing 2 hours FullHD film at i5-9600K CPU. And onCPU MAnalyse gives better quality in compare with hardware from GTX1060 card.

Also remember the MAnalyse in refining stages with partially degrained one of 2 input clips gives lower SAD values so the last MDegrainN (or even each next RefineMV call if several present) need to have lower thSAD values for MDegrainN to keep from more blurring. Here the new Auto-thSAD feature for MDegrainN expected to make good help. See example in https://forum.doom9.org/showthread.p...42#post1990642

[Fjord]

@DTL - I copied all your release descriptions into a Word document for my own reference. I sent you a PM with a link to it.

[DTL]

I added file to github.

Current research question for multi-generation MVs refining (at least using MDegrainN only - not with external denoisers as prefilters):

Is usage of several generations with slow increasing of denoise params produces better MVs in comparison with simple 2 stages with full power denoise params ?

It requires lots of test runs and checks for quality.

Example pseudo-scripts:
1. 2 stages full power denoise in each:

my_src=last
s=Super(my_src)
mv1=MAnalyse(s)
mv2=RefineMV(mv1, s, my_src) // tr=12, thSAD_a=1.3 for MAnalyse/MDegrainN in RefineMV()
final_denoise=MDegrainN(my_src, s, mv2,..) // tr=12, thSAD_a=1.3 for MDegrainN in last output denoise

2. example of several stages with slow incresing of denoise params in refining stages

my_src=last
s=Super(my_src)
mv1=MAnalyse(s)
mv2=RefineMV(mv1, s, my_src) // tr=1, thSAD_a=0.9 for MAnalyse/MDegrainN in RefineMV()
mv3=RefineMV(mv2, s, my_src) // tr=3, thSAD_a=0.9 for MAnalyse/MDegrainN in RefineMV()
mv4=RefineMV(mv3, s, my_src) // tr=6, thSAD_a=1.0 for MAnalyse/MDegrainN in RefineMV()
mv5=RefineMV(mv4, s, my_src) // tr=12, thSAD_a=1.3 for MAnalyse/MDegrainN in RefineMV()
final_denoise=MDegrainN(my_src, s, mv5,..) // tr=12, thSAD_a=1.3 for MDegrainN in last output denoise

better to supplement RefineMV() script with user-provided arguments of tr and thSAD* (_a) params.

Current idea for many generations refining - the first generations of internal degrain with low degrain params (low tr and low thSAD) may lower degrade/blur details so may save more MVs from errors in first stages and provide more correct intermediate denoised clip for next stages of MAnalyse. But this idea check may require lots of test runs.