Avisynth+

[FranceBB]

Originally Posted by pinterf

Is it the same Avisynth version (my second test build) which you had troubles with?

Nope, it's AviSynth+ 3.5 (r3106, 3.5, i386) (3.5.0.0)

Originally Posted by pinterf

Next test please

https://github.com/pinterf/AddGrainC...ses/tag/v1.8.1

Original parameters restored, simd noise blocks use now unaligned load (possible solution to FranceBB's crash?)

You did it. Now it works fine.
Thanks.

[MeteorRain]

Had some time looking at mvtools and masktools, they are pretty huge and hard to start. I may try to run QTGMC and MCTD and find the hottest path and see if there's anything I can improve.

Masktools, it's possible I'll start from F2's float version and improve from there. So, likely you'll see neo_masktools in a few weeks if everything goes right.

Another thing that I'm looking is dfttest. dfttest-avs has some WIN32 APIs, and there's vapoursynth-dfttest ported by holy. Maybe it's a good idea to combine them into one that would support AVS on Linux. As usual, I'll find a chance to remove YUY2 support, if there's any. And will change the plugin name if it breaks backward compatibility.

[pinterf]

You mean, mvtools-float? It is full floating point I think, even the hottest path: the SAD calculation.
Latest avs mvtools can do float as well, I separated the motion vector clip format from the format of the clip to process. Do motion vector search on 8 or 10-16 bits and use the results on whatever bit depth clip you have. As mv search is the bottleneck, speed penalty is minimal.
Avs mvtools has finer granularity in block sizes 2-3-6-8-12-16-24...64, vs stayed at original sizes but it has even 128 sized blocks. Avs mdegrain supports 16 bit output from 8 bit inputs.
Vs versions have much nicer code, my next move would be drop yuy2 support, drop stacked (probably there are people who don't even know what it is ) support and re-do the internal MT. Just because this complex codebase is non-maintanable. Four or five years ago I had to keep compatibility with the ancient software infrastructure, but by 2020 things has been changed.

[Stereodude]

If you get really bored > 8-bit support for MCTD would be nice.

[MeteorRain]

Thanks, pinterf, for the information. I may take a look at the differences between the 2 before deciding what to do. My plan was to take mvsf as a starting point, gradually adding back functions from the avs version. Considering float version having a higher precision I wonder if it makes sense to have a uint16 internal process variation -- more precise than uint8 but faster than float or double?

At this point, internal MT (C++PSTL), AVX2(Intrinsics), HBD(Template functions) and dual synth interface(wrapper) are the main reasons I'm refactoring these code. I'd like to see MCTD running on AVS+ Linux with zero glitch and at a reasonable speed. A side effect for dual synth interface is of course hopefully reducing the chance of market fragmentation.

BTW I just rewrote the temporal median into dual synth interface early this week. Benchmark speed is amazing with AVX2.

Stereodude, that's definitely what I'm approaching.

[pinterf]

It is even very hard to decide whether 8 or 16 bit clip based motion vector search is better. SAD calculation and comparison is the key, I think it is overkill to use double SADs (note that a 32 bit float may have even less precision at these scales than 32 bit int, double should be used. In classic integer SAD mvtools versions 8 bit SAD code is handcrafted asm which is borrowed from the x264/265 project and I think it has reached its optimization limit. Since the typical blocksize is 8x8 or 16x16 avx2 is not even considered there. Motion vectors however are of greater precision in F2's version, and can cover of greater length, classic mvtools is using s7.8 short precision for them. Still I don't think that it's a deal-breaker difference. But the outer filters that are using the result of the core motion vector search can be optimized, but since the bottleneck is mv search, moving a part from SSE2 to AVX2 can result in <1% benefit if any and even less if looking at a whole qtgmc process. And another difference between integer SAD based vs and avs mvtools: vs version is always using 64 bit for summing up SAD (mainly because of supporting 128x128 blocksize @16bit @3 planes needed it), avs mvtools is using 64 bits ("bigsad_t") when it is needed. Thus avs code is a bit more complex, but is possibly quicker at 8 bit clips.

But since you are not tied to the historical things and I can see clearly that you like the "art" side of programming rather than living with ancient hacks, if I were you I'd take F2's code, as you have said. I can imagine that it would even take less time than to understand and kick away a hack then another hack and another in an infinite loop.

[tormento]

Things are getting interesting here.

[MeteorRain]

Originally Posted by pinterf

But since you are not tied to the historical things and I can see clearly that you like the "art" side of programming rather than living with ancient hacks.

My rule is if there's an ancient alternative running for old folks I'd be happy to ditch the historical part. For example, for those who still need ancient fft3dfilter options they can keep using that, for those who don't, they can migrate to neo_ variant. The fact that my update still keeps pure C and SSE/SSE2 tells a lot about my point of supporting old folks -- if there's no significant overhead.

I'll take your information as a food for though. Let's see if I'll be able to work out something.

[pinterf]

Pure C is essential as a reference. How I hate reverse engineering those many hundred lines inline assembler (which is then able to run faster from C with modern compilers) Other than that I'd target only sse4.1, avx2 and avx512 options

[MeteorRain]

IKR! IT() is one of them and good god that's pure MMX code core functions. Took me days just to extract C code out of MMX, so much pain there.

[tormento]

Somethings buzzes in my mind since yesterday afternoon: what sense has SetMemoryMax now, at least on x64 systems?

We have all at least 8GB of RAM, wouldn't be easier to let AVS+ allocate the RAM it needs with no useless declarations?

[LigH]

Imagine running several processes in parallel ...

[real.finder]

Originally Posted by MeteorRain

IKR! IT() is one of them and good god that's pure MMX code core functions. Took me days just to extract C code out of MMX, so much pain there.

you can try a new challenge http://forum.doom9.net/showthread.ph...73#post1882073

also Reel.Deel said that

the default for the original VerticalCleaner is 2, in RgTools the default is 1.

also here http://avisynth.nl/index.php/RgTools/VerticalCleaner

so RgTools should do same (should back to 2)

[tormento]

Originally Posted by LigH

Imagine running several processes in parallel ...

Well, tell me a good value for SetMemoryMax and how is it respected.

I use SetMemoryMax(8000) simply because with 16GB of ram, I really don't know what value to put.

From Wiki:

In Avisynth+ this limit for default Memory Max is 1024MB for 32 bits and 4096MB on the x64 version

Well, 8000 seems to be valid too.

@pinterf?

[StainlessS]

SetMemoryMax, Max amount for frame cache, how many frames you reckon you is gonna need cachin' for your mega massive script ?
8000, is probably way overkill [I would think].

Anyways, I aint never played with 4K never mind 8K, but why not try test with (8000) cache, and your script, and then again with (4000, or maybe 2000), any speed diff ?

Setting too high a cache does nothing good, IanB has made many/(well some) posts about overkill where people allocated ridiculous amounts of mem, just
because they felt the need to justify purchase of their huge RAM sticks.

Several Processes In Parallel, might want to limit current script to NOT use all available RAM, because you want to also run another few scripts at same time,
limiting current script usage ensures that subsequent scripts also have some ram to play with.

EDIT Give your script loads of RAM with typical max RAM usage script, and see what RAM usage AvsMeter shows, then set to a little above what AvsMeter shows.

EDIT:

Originally Posted by IanB

Where are people getting the crazy idea that a big SetMemoryMax is ever a good idea.

I have searched the forum numerous times looking for any fud and misinformational posts and I do not find any, all I find are the occasional post advising to turn it down. and lots of posts with people using insanely high values and having problems.

SetMemoryMax just sets the size of the Avisynth frame cache!

It does not control any other memory usage!

The frame cache only needs to be just large enough to hold the temporal requirements of the script. If a script has no temporal requirements, i.e. a frame that is required more than once in some part of the script then the cache is completely useless and may prevent some other memory usage from being able to malloc the memory that it needs to work.

A 1920x1080 YV12 video frame is ~3meg, as RGB32 it is 8meg, you can hold 64 such frames in 512meg. YV12 720x480 frames are only 0.5meg you can hold 1024 in 512 meg.

It takes some resources to manage the frame cache so doing a SetMemoryMax(2048) for a SD video and having 4000 odd buffers in play is just stupid and will actually slow the script processing down a little bit.

Advise :- SetMemoryMax, Turn it down!

[tormento]

Originally Posted by StainlessS

Anyways, I aint never played with 4K never mind 8K, but why not try test with (8000) cache, and your script, and then again with (4000, or maybe 2000), any speed diff ?

To tell you the truth, I am currently limited by video memory, as I extensively use both CUDA and OpenCL. I have to limit multithreading to the number of threads that don't saturate my video card which, unfortunately, has 3GB of DDRx only.

On 4k I have to use 1 thread only, as a single frame process can saturate it. On 2k I can go up to 6 thread.

I will make some trials with AVSMeter and tell you actual used resources.

I know the post of IanB but it was referred to ancient AviSynth. I don't know what happened since AviSynth+ and if his considerations are still valid, higher color spaces given.

[StainlessS]

I don't know what happened since AviSynth+ and if his considerations are still valid

Well HBD frames are of course bigger (size x2 for 16 bit and x4 float) compared to sizes given above by IanB, Avs+ is actually less greedy on cache requirements than Avs Std,
so for same frame size would require less cache than avs std. 32 bit still limited in available memory, I assume you are using x64 else SetMemoryMax(8000) would not even work.
By the way, SetMemoryMax() on its own returns the size that avs is using.

Maybe something like

m=SetMemoryMax()
blankclip
Subtitle(String(m,"MemMax = %.0f"))

would show the default for your memory size. [with old x86 avs std, was I think half available|free RAM limited to max 500MB].

EDIT: You might even try a SetMemoryMax(8000) before above script, to see if Avs+ actually sets the value requested, or limits it assuming that you are daft.
(It probably does not limit it, assuming that you are not daft)

[StainlessS]

Default Memory Max on my machine with 12GB RAM, x86 1024MB, x64 3055MB [seems a bit high but I guess deliberate in case of HBD, probably fine if you got plenty RAM].
x64, 3055MB Looks to be about Available RAM / 3 [9295 / 3 = 3098MB]

Output from AvsInit.avsi (x86)

00000251    0.00000000  [3108] AvsInit:
00000252    0.00005326  [3108] AvsInit: Auto load plugins script ENTRY
00000253    0.00010580  [3108] AvsInit:
00000254    0.00377458  [3108] AvsInit_ShowInfo:
00000255    0.00380937  [3108] AvsInit_ShowInfo: AvsInit_Version     = 1.10
00000256    0.00385103  [3108] AvsInit_ShowInfo: RT_Stats Version    = 2.00Beta12
00000257    0.00389197  [3108] AvsInit_ShowInfo: SysInfo Version     = 0.115000
00000258    0.00393473  [3108] AvsInit_ShowInfo: VersionString       = AviSynth+ 3.5 (r3072, master, i386)
00000259    0.00397531  [3108] AvsInit_ShowInfo: SetMemoryMax        = 1024
00000260    0.00401733  [3108] AvsInit_ShowInfo: GScript Available   = AVS+
00000261    0.00405900  [3108] AvsInit_ShowInfo: OS Bitness          = 64
00000262    0.00410175  [3108] AvsInit_ShowInfo: Avisynth Bitness    = 32
00000263    0.00414487  [3108] AvsInit_ShowInfo: WorkingDir          = C:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\
00000264    0.00418545  [3108] AvsInit_ShowInfo: ProcessName         = C:\Program Files (x86)\DAUM\PotPlayer\PotPlayerMini.exe
00000265    0.00422784  [3108] AvsInit_ShowInfo: ParentProcessName   = PSPad.exe
00000266    0.00426914  [3108] AvsInit_ShowInfo: OSVersionString     = Windows 7 (x64) Service Pack 1.0 (Build 7601)
00000267    0.00431045  [3108] AvsInit_ShowInfo: OSVersionNumber     = 6.100000
00000268    0.00435139  [3108] AvsInit_ShowInfo: CPUName             = Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz / Yorkfield (Core 2 Quad) 6M
00000269    0.00439305  [3108] AvsInit_ShowInfo: Cores               = 04:04 (Phy:Log)
00000270    0.00443436  [3108] AvsInit_ShowInfo: CPU Extensions      = MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1
00000271    0.00447747  [3108] AvsInit_ShowInfo: Total Memory        = 12221MB
00000272    0.00452023  [3108] AvsInit_ShowInfo: Avail Memory        = 9161MB'
00000273    0.00456153  [3108] AvsInit_ShowInfo: Screen Res          = 1920x1080
00000274    0.00460175  [3108] AvsInit_ShowInfo: Screen BitsPerPixel = 32
00000275    0.00464414  [3108] AvsInit_ShowInfo: Time                = Saturday 11 April 2020 11:29:21[GMT Standard Time]
00000276    0.00468544  [3108] AvsInit_ShowInfo: User TEMP Dir       = C:\Users\LiquoriceRoot\AppData\Local\Temp
00000277    0.00472674  [3108] AvsInit_ShowInfo: ComSpec             = C:\Windows\system32\cmd.exe
00000278    0.00476805  [3108] AvsInit_ShowInfo: Computer Name       = PLEX-P2
00000279    0.00480899  [3108] AvsInit_ShowInfo: User Name           = LiquoriceRoot

EDIT: And x64

00000543	0.00000000	[1952] AvsInit: 	
00000544	0.00003333	[1952] AvsInit: Auto load plugins script ENTRY	
00000545	0.00006703	[1952] AvsInit: 	
00000546	0.00352278	[1952] AvsInit_ShowInfo: 	
00000547	0.00355792	[1952] AvsInit_ShowInfo: AvsInit_Version     = 1.10	
00000548	0.00359596	[1952] AvsInit_ShowInfo: RT_Stats Version    = 2.00Beta12	
00000549	0.00363364	[1952] AvsInit_ShowInfo: SysInfo Version     = 0.115000	
00000550	0.00366661	[1952] AvsInit_ShowInfo: VersionString       = AviSynth+ 3.5 (r3072, master, x86_64)	
00000551	0.00369886	[1952] AvsInit_ShowInfo: SetMemoryMax        = 3055	
00000552	0.00373111	[1952] AvsInit_ShowInfo: GScript Available   = AVS+	
00000553	0.00376444	[1952] AvsInit_ShowInfo: OS Bitness          = 64	
00000554	0.00379705	[1952] AvsInit_ShowInfo: Avisynth Bitness    = 64	
00000555	0.00382893	[1952] AvsInit_ShowInfo: WorkingDir          = C:\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\	
00000556	0.00386154	[1952] AvsInit_ShowInfo: ProcessName         = C:\NON-INSTALL\VDUB\VDUB2\VirtualDub64.exe	
00000557	0.00389379	[1952] AvsInit_ShowInfo: ParentProcessName   = explorer.exe	
00000558	0.00392639	[1952] AvsInit_ShowInfo: OSVersionString     = Windows 7 (x64) Service Pack 1.0 (Build 7601)	
00000559	0.00395828	[1952] AvsInit_ShowInfo: OSVersionNumber     = 6.100000	
00000560	0.00399125	[1952] AvsInit_ShowInfo: CPUName             = Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz / Yorkfield (Core 2 Quad) 6M	
00000561	0.00402386	[1952] AvsInit_ShowInfo: Cores               = 04:04 (Phy:Log)	
00000562	0.00405683	[1952] AvsInit_ShowInfo: CPU Extensions      = MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1	
00000563	0.00408907	[1952] AvsInit_ShowInfo: Total Memory        = 12221MB	
00000564	0.00412132	[1952] AvsInit_ShowInfo: Avail Memory        = 9295MB'	
00000565	0.00415393	[1952] AvsInit_ShowInfo: Screen Res          = 1920x1080	
00000566	0.00418617	[1952] AvsInit_ShowInfo: Screen BitsPerPixel = 32	
00000567	0.00421987	[1952] AvsInit_ShowInfo: Time                = Saturday 11 April 2020 11:38:25[GMT Standard Time]	
00000568	0.00425211	[1952] AvsInit_ShowInfo: User TEMP Dir       = C:\Users\LiquoriceRoot\AppData\Local\Temp
00000569	0.00428400	[1952] AvsInit_ShowInfo: ComSpec             = C:\Windows\system32\cmd.exe	
00000570	0.00431624	[1952] AvsInit_ShowInfo: Computer Name       = PLEX-P2	
00000571	0.00434921	[1952] AvsInit_ShowInfo: User Name           = LiquoriceRoot
00000572	0.00438146	[1952] AvsInit_ShowInfo:

EDIT: SetMemoryMax(8000) on my system does indeed set memory Max to 8000.

[Groucho2004]

Originally Posted by tormento

I will make some trials with AVSMeter and tell you actual used resources.

I know the post of IanB but it was referred to ancient AviSynth. I don't know what happened since AviSynth+ and if his considerations are still valid, higher color spaces given.

The proper way to do it with AVS+:

Put your SetMemoryMax() line on top of the script.
Put "SetLogParams("stdout", LOG_WARNING)" on the next line.
Run AVSMeter(64) and watch out for this warning:
"Caches have been shrunk due to low memory limit. This will probably degrade performance. You can try increasing the limit using SetMemoryMax()"

Increase the value for SetMemoryMax() until the warning disappears.

[Selur]

Trying to using ScriptClip with Subtitle to see the current frame number.
I see the frame number, but the align I gets ignored, it's always at the default align=7, even if I use align=4 in my code:

LoadCPlugin("I:\WORKSP~2\Hybrid\debug\32bit\AVISYN~1\ffms2.dll")
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
# loading source: F:\TestClips&Co\files\interlaceAndTelecineSamples\interlaced\YUV422_sample.avi
#  input color sampling YUY2
#  input luminance scale tv
FFVideoSource("F:\TESTCL~1\files\INTERL~1\INTERL~1\YUV422~1.AVI",cachefile="E:\Temp\avi_1deca8d5573c8cb8cd8489678b764f38_853323747_1_0.ffindex",fpsnum=25,colorspace="YUY2")
# current resolution: 720x576
# filtering
# color modifications
Levels(0,1.50,255,0,255)
# adding frame number
ScriptClip("Subtitle(string(current_frame),align=4)")
# Reinterlacing since interlaced output is requested
ConvertToYV12(matrix="Rec601", interlaced=true)
PreFetch(8)
return last

I also tried:

ScriptClip("""Subtitle(string(current_frame),align=4)""")

but I don't see where I go wrong.

-> Can someone tell me how to properly use ScriptClip to show the current frame number in a way that the alignment works?

Thanks

Cu Selur

Ps.: using Avisynth+ 32bit r3106