Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
9th April 2020, 18:41 | #5401 | Link | ||
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
|
Quote:
Quote:
Thanks. |
||
10th April 2020, 01:15 | #5402 | Link |
結城有紀
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
|
Had some time looking at mvtools and masktools, they are pretty huge and hard to start. I may try to run QTGMC and MCTD and find the hottest path and see if there's anything I can improve.
Masktools, it's possible I'll start from F2's float version and improve from there. So, likely you'll see neo_masktools in a few weeks if everything goes right. Another thing that I'm looking is dfttest. dfttest-avs has some WIN32 APIs, and there's vapoursynth-dfttest ported by holy. Maybe it's a good idea to combine them into one that would support AVS on Linux. As usual, I'll find a chance to remove YUY2 support, if there's any. And will change the plugin name if it breaks backward compatibility. |
10th April 2020, 05:25 | #5403 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
You mean, mvtools-float? It is full floating point I think, even the hottest path: the SAD calculation.
Latest avs mvtools can do float as well, I separated the motion vector clip format from the format of the clip to process. Do motion vector search on 8 or 10-16 bits and use the results on whatever bit depth clip you have. As mv search is the bottleneck, speed penalty is minimal. Avs mvtools has finer granularity in block sizes 2-3-6-8-12-16-24...64, vs stayed at original sizes but it has even 128 sized blocks. Avs mdegrain supports 16 bit output from 8 bit inputs. Vs versions have much nicer code, my next move would be drop yuy2 support, drop stacked (probably there are people who don't even know what it is ) support and re-do the internal MT. Just because this complex codebase is non-maintanable. Four or five years ago I had to keep compatibility with the ancient software infrastructure, but by 2020 things has been changed. Last edited by pinterf; 10th April 2020 at 05:27. |
10th April 2020, 06:30 | #5405 | Link |
結城有紀
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
|
Thanks, pinterf, for the information. I may take a look at the differences between the 2 before deciding what to do. My plan was to take mvsf as a starting point, gradually adding back functions from the avs version. Considering float version having a higher precision I wonder if it makes sense to have a uint16 internal process variation -- more precise than uint8 but faster than float or double?
At this point, internal MT (C++PSTL), AVX2(Intrinsics), HBD(Template functions) and dual synth interface(wrapper) are the main reasons I'm refactoring these code. I'd like to see MCTD running on AVS+ Linux with zero glitch and at a reasonable speed. A side effect for dual synth interface is of course hopefully reducing the chance of market fragmentation. BTW I just rewrote the temporal median into dual synth interface early this week. Benchmark speed is amazing with AVX2. Stereodude, that's definitely what I'm approaching. |
10th April 2020, 07:30 | #5406 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
It is even very hard to decide whether 8 or 16 bit clip based motion vector search is better. SAD calculation and comparison is the key, I think it is overkill to use double SADs (note that a 32 bit float may have even less precision at these scales than 32 bit int, double should be used. In classic integer SAD mvtools versions 8 bit SAD code is handcrafted asm which is borrowed from the x264/265 project and I think it has reached its optimization limit. Since the typical blocksize is 8x8 or 16x16 avx2 is not even considered there. Motion vectors however are of greater precision in F2's version, and can cover of greater length, classic mvtools is using s7.8 short precision for them. Still I don't think that it's a deal-breaker difference. But the outer filters that are using the result of the core motion vector search can be optimized, but since the bottleneck is mv search, moving a part from SSE2 to AVX2 can result in <1% benefit if any and even less if looking at a whole qtgmc process. And another difference between integer SAD based vs and avs mvtools: vs version is always using 64 bit for summing up SAD (mainly because of supporting 128x128 blocksize @16bit @3 planes needed it), avs mvtools is using 64 bits ("bigsad_t") when it is needed. Thus avs code is a bit more complex, but is possibly quicker at 8 bit clips.
But since you are not tied to the historical things and I can see clearly that you like the "art" side of programming rather than living with ancient hacks, if I were you I'd take F2's code, as you have said. I can imagine that it would even take less time than to understand and kick away a hack then another hack and another in an infinite loop. |
10th April 2020, 12:36 | #5408 | Link | |
結城有紀
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
|
Quote:
I'll take your information as a food for though. Let's see if I'll be able to work out something. |
|
10th April 2020, 12:48 | #5409 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
Pure C is essential as a reference. How I hate reverse engineering those many hundred lines inline assembler (which is then able to run faster from C with modern compilers) Other than that I'd target only sse4.1, avx2 and avx512 options
|
10th April 2020, 15:31 | #5411 | Link |
Acid fr0g
Join Date: May 2002
Location: Italy
Posts: 2,582
|
Somethings buzzes in my mind since yesterday afternoon: what sense has SetMemoryMax now, at least on x64 systems?
We have all at least 8GB of RAM, wouldn't be easier to let AVS+ allocate the RAM it needs with no useless declarations?
__________________
@turment on Telegram |
10th April 2020, 18:09 | #5413 | Link | ||
Registered User
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
|
Quote:
also Reel.Deel said that Quote:
so RgTools should do same (should back to 2)
__________________
See My Avisynth Stuff |
||
10th April 2020, 18:31 | #5414 | Link |
Acid fr0g
Join Date: May 2002
Location: Italy
Posts: 2,582
|
Well, tell me a good value for SetMemoryMax and how is it respected.
I use SetMemoryMax(8000) simply because with 16GB of ram, I really don't know what value to put. From Wiki: In Avisynth+ this limit for default Memory Max is 1024MB for 32 bits and 4096MB on the x64 version Well, 8000 seems to be valid too. @pinterf?
__________________
@turment on Telegram Last edited by tormento; 10th April 2020 at 18:34. |
11th April 2020, 00:39 | #5415 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
SetMemoryMax, Max amount for frame cache, how many frames you reckon you is gonna need cachin' for your mega massive script ?
8000, is probably way overkill [I would think]. Anyways, I aint never played with 4K never mind 8K, but why not try test with (8000) cache, and your script, and then again with (4000, or maybe 2000), any speed diff ? Setting too high a cache does nothing good, IanB has made many/(well some) posts about overkill where people allocated ridiculous amounts of mem, just because they felt the need to justify purchase of their huge RAM sticks. Several Processes In Parallel, might want to limit current script to NOT use all available RAM, because you want to also run another few scripts at same time, limiting current script usage ensures that subsequent scripts also have some ram to play with. EDIT Give your script loads of RAM with typical max RAM usage script, and see what RAM usage AvsMeter shows, then set to a little above what AvsMeter shows. EDIT: Quote:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 11th April 2020 at 08:01. |
|
11th April 2020, 09:34 | #5416 | Link | |
Acid fr0g
Join Date: May 2002
Location: Italy
Posts: 2,582
|
Quote:
On 4k I have to use 1 thread only, as a single frame process can saturate it. On 2k I can go up to 6 thread. I will make some trials with AVSMeter and tell you actual used resources. I know the post of IanB but it was referred to ancient AviSynth. I don't know what happened since AviSynth+ and if his considerations are still valid, higher color spaces given.
__________________
@turment on Telegram |
|
11th April 2020, 10:36 | #5417 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
so for same frame size would require less cache than avs std. 32 bit still limited in available memory, I assume you are using x64 else SetMemoryMax(8000) would not even work. By the way, SetMemoryMax() on its own returns the size that avs is using. Maybe something like Code:
m=SetMemoryMax() blankclip Subtitle(String(m,"MemMax = %.0f")) EDIT: You might even try a SetMemoryMax(8000) before above script, to see if Avs+ actually sets the value requested, or limits it assuming that you are daft. (It probably does not limit it, assuming that you are not daft)
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 11th April 2020 at 10:46. |
|
11th April 2020, 11:34 | #5418 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Default Memory Max on my machine with 12GB RAM, x86 1024MB, x64 3055MB [seems a bit high but I guess deliberate in case of HBD, probably fine if you got plenty RAM].
x64, 3055MB Looks to be about Available RAM / 3 [9295 / 3 = 3098MB] Output from AvsInit.avsi (x86) Code:
00000251 0.00000000 [3108] AvsInit: 00000252 0.00005326 [3108] AvsInit: Auto load plugins script ENTRY 00000253 0.00010580 [3108] AvsInit: 00000254 0.00377458 [3108] AvsInit_ShowInfo: 00000255 0.00380937 [3108] AvsInit_ShowInfo: AvsInit_Version = 1.10 00000256 0.00385103 [3108] AvsInit_ShowInfo: RT_Stats Version = 2.00Beta12 00000257 0.00389197 [3108] AvsInit_ShowInfo: SysInfo Version = 0.115000 00000258 0.00393473 [3108] AvsInit_ShowInfo: VersionString = AviSynth+ 3.5 (r3072, master, i386) 00000259 0.00397531 [3108] AvsInit_ShowInfo: SetMemoryMax = 1024 00000260 0.00401733 [3108] AvsInit_ShowInfo: GScript Available = AVS+ 00000261 0.00405900 [3108] AvsInit_ShowInfo: OS Bitness = 64 00000262 0.00410175 [3108] AvsInit_ShowInfo: Avisynth Bitness = 32 00000263 0.00414487 [3108] AvsInit_ShowInfo: WorkingDir = C:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\ 00000264 0.00418545 [3108] AvsInit_ShowInfo: ProcessName = C:\Program Files (x86)\DAUM\PotPlayer\PotPlayerMini.exe 00000265 0.00422784 [3108] AvsInit_ShowInfo: ParentProcessName = PSPad.exe 00000266 0.00426914 [3108] AvsInit_ShowInfo: OSVersionString = Windows 7 (x64) Service Pack 1.0 (Build 7601) 00000267 0.00431045 [3108] AvsInit_ShowInfo: OSVersionNumber = 6.100000 00000268 0.00435139 [3108] AvsInit_ShowInfo: CPUName = Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz / Yorkfield (Core 2 Quad) 6M 00000269 0.00439305 [3108] AvsInit_ShowInfo: Cores = 04:04 (Phy:Log) 00000270 0.00443436 [3108] AvsInit_ShowInfo: CPU Extensions = MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1 00000271 0.00447747 [3108] AvsInit_ShowInfo: Total Memory = 12221MB 00000272 0.00452023 [3108] AvsInit_ShowInfo: Avail Memory = 9161MB' 00000273 0.00456153 [3108] AvsInit_ShowInfo: Screen Res = 1920x1080 00000274 0.00460175 [3108] AvsInit_ShowInfo: Screen BitsPerPixel = 32 00000275 0.00464414 [3108] AvsInit_ShowInfo: Time = Saturday 11 April 2020 11:29:21[GMT Standard Time] 00000276 0.00468544 [3108] AvsInit_ShowInfo: User TEMP Dir = C:\Users\LiquoriceRoot\AppData\Local\Temp 00000277 0.00472674 [3108] AvsInit_ShowInfo: ComSpec = C:\Windows\system32\cmd.exe 00000278 0.00476805 [3108] AvsInit_ShowInfo: Computer Name = PLEX-P2 00000279 0.00480899 [3108] AvsInit_ShowInfo: User Name = LiquoriceRoot Code:
00000543 0.00000000 [1952] AvsInit: 00000544 0.00003333 [1952] AvsInit: Auto load plugins script ENTRY 00000545 0.00006703 [1952] AvsInit: 00000546 0.00352278 [1952] AvsInit_ShowInfo: 00000547 0.00355792 [1952] AvsInit_ShowInfo: AvsInit_Version = 1.10 00000548 0.00359596 [1952] AvsInit_ShowInfo: RT_Stats Version = 2.00Beta12 00000549 0.00363364 [1952] AvsInit_ShowInfo: SysInfo Version = 0.115000 00000550 0.00366661 [1952] AvsInit_ShowInfo: VersionString = AviSynth+ 3.5 (r3072, master, x86_64) 00000551 0.00369886 [1952] AvsInit_ShowInfo: SetMemoryMax = 3055 00000552 0.00373111 [1952] AvsInit_ShowInfo: GScript Available = AVS+ 00000553 0.00376444 [1952] AvsInit_ShowInfo: OS Bitness = 64 00000554 0.00379705 [1952] AvsInit_ShowInfo: Avisynth Bitness = 64 00000555 0.00382893 [1952] AvsInit_ShowInfo: WorkingDir = C:\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\ 00000556 0.00386154 [1952] AvsInit_ShowInfo: ProcessName = C:\NON-INSTALL\VDUB\VDUB2\VirtualDub64.exe 00000557 0.00389379 [1952] AvsInit_ShowInfo: ParentProcessName = explorer.exe 00000558 0.00392639 [1952] AvsInit_ShowInfo: OSVersionString = Windows 7 (x64) Service Pack 1.0 (Build 7601) 00000559 0.00395828 [1952] AvsInit_ShowInfo: OSVersionNumber = 6.100000 00000560 0.00399125 [1952] AvsInit_ShowInfo: CPUName = Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz / Yorkfield (Core 2 Quad) 6M 00000561 0.00402386 [1952] AvsInit_ShowInfo: Cores = 04:04 (Phy:Log) 00000562 0.00405683 [1952] AvsInit_ShowInfo: CPU Extensions = MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1 00000563 0.00408907 [1952] AvsInit_ShowInfo: Total Memory = 12221MB 00000564 0.00412132 [1952] AvsInit_ShowInfo: Avail Memory = 9295MB' 00000565 0.00415393 [1952] AvsInit_ShowInfo: Screen Res = 1920x1080 00000566 0.00418617 [1952] AvsInit_ShowInfo: Screen BitsPerPixel = 32 00000567 0.00421987 [1952] AvsInit_ShowInfo: Time = Saturday 11 April 2020 11:38:25[GMT Standard Time] 00000568 0.00425211 [1952] AvsInit_ShowInfo: User TEMP Dir = C:\Users\LiquoriceRoot\AppData\Local\Temp 00000569 0.00428400 [1952] AvsInit_ShowInfo: ComSpec = C:\Windows\system32\cmd.exe 00000570 0.00431624 [1952] AvsInit_ShowInfo: Computer Name = PLEX-P2 00000571 0.00434921 [1952] AvsInit_ShowInfo: User Name = LiquoriceRoot 00000572 0.00438146 [1952] AvsInit_ShowInfo:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 11th April 2020 at 11:57. |
11th April 2020, 11:39 | #5419 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
Put your SetMemoryMax() line on top of the script. Put "SetLogParams("stdout", LOG_WARNING)" on the next line. Run AVSMeter(64) and watch out for this warning: "Caches have been shrunk due to low memory limit. This will probably degrade performance. You can try increasing the limit using SetMemoryMax()" Increase the value for SetMemoryMax() until the warning disappears.
__________________
Groucho's Avisynth Stuff |
|
11th April 2020, 17:23 | #5420 | Link |
Registered User
Join Date: Oct 2001
Location: Germany
Posts: 7,277
|
Trying to using ScriptClip with Subtitle to see the current frame number.
I see the frame number, but the align I gets ignored, it's always at the default align=7, even if I use align=4 in my code: Code:
LoadCPlugin("I:\WORKSP~2\Hybrid\debug\32bit\AVISYN~1\ffms2.dll") SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE) # loading source: F:\TestClips&Co\files\interlaceAndTelecineSamples\interlaced\YUV422_sample.avi # input color sampling YUY2 # input luminance scale tv FFVideoSource("F:\TESTCL~1\files\INTERL~1\INTERL~1\YUV422~1.AVI",cachefile="E:\Temp\avi_1deca8d5573c8cb8cd8489678b764f38_853323747_1_0.ffindex",fpsnum=25,colorspace="YUY2") # current resolution: 720x576 # filtering # color modifications Levels(0,1.50,255,0,255) # adding frame number ScriptClip("Subtitle(string(current_frame),align=4)") # Reinterlacing since interlaced output is requested ConvertToYV12(matrix="Rec601", interlaced=true) PreFetch(8) return last Code:
ScriptClip("""Subtitle(string(current_frame),align=4)""") -> Can someone tell me how to properly use ScriptClip to show the current frame number in a way that the alignment works? Thanks Cu Selur Ps.: using Avisynth+ 32bit r3106 Last edited by Selur; 11th April 2020 at 17:27. |
|
|