Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
26th December 2014, 20:10 | #1 | Link |
Registered User
Join Date: Jan 2013
Posts: 12
|
FFmpegSource2 Incorrect Timing
Firstly, Happy Holidays all! This is my first post after following and searching the forum for a couple years.
I am trying to restore and upscale an old anime show from source DVDs. I've ripped the VOB files from the DVD with DVDDecrypter, separated by chapter. When I use FFmpegSource2, the input file is never indexed at the correct time, so my output file is always 2-3 minutes shorter than my input. The frames in the output file are sped up at seemingly random intervals. This happens regardless of how I call it in: FFIndex("C:\Users\User\Desktop\Vid\VTS_03_CHAPTER_02_1.VOB") AudioDub(FFVideoSource("C:\Users\User\Desktop\Vid\VTS_03_CHAPTER_02_1.VOB"), FFAudioSource("C:\Users\User\Desktop\Vid\VTS_03_CHAPTER_02_1.VOB")) or FFmpegSource2("C:\Users\User\Desktop\Vid\VTS_03_CHAPTER_02_1.VOB") or the like. DirectShowSource indexes the time correctly, but it is just way too slow for the script I have (0.09FPS vs 0.70FPS FFmegsource2 in MeGUI). Let me know any suggestions, thanks! |
26th December 2014, 21:49 | #2 | Link |
Big Bit Savings Now !
Join Date: Feb 2007
Location: close to the wall
Posts: 1,546
|
DGIndex
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain) "Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..." |
27th December 2014, 01:40 | #3 | Link |
Registered User
Join Date: Jan 2013
Posts: 12
|
Thanks for the quick reply, Emulgator!
Is it possible to make a .dgi from a VOB, or just a .d2v with DGIndex? So far the .d2v I've made and the use of MPEG2Source in my script gives me the same speed and results as Directshowsource. |
27th December 2014, 03:55 | #4 | Link | |
Registered User
Join Date: Mar 2011
Posts: 4,829
|
Quote:
Just use MeGUI's File/Open menu. It'll offer to index the vob file with DGIndex, you should add the indexing job to the queue, run it, then create your script when the script creator opens. MeGUI will extract the audio and load it into the audio section for re-encoding while it indexes, should you want to re-encode it. No need to do it via a script for vob files. If you still have speed problems, try a sample encode without any additional filters in the script. If the speed is fine, then it'd be due to the filtering/plugins you're using. If you're having problems you can't sort out, post your full script and maybe upload a sample somewhere. You should be able to split off a sample and save it as a ts file with tsmuxer. |
|
28th December 2014, 06:21 | #5 | Link |
Registered User
Join Date: Jan 2013
Posts: 12
|
Ok thanks for the clarification!
I've been using MeGUI's File Indexer to index my VOBs, and the output index file is set to .d2v and I can't seem to change it to .dgi. I have an amd graphics card so I can't use DGDecodeNV. Encoding utilizes full CPU without my script, but caps out around 20% with the script. So the issue is most definitely from a few of my plugins, specifically QTGMC and I believe LSFmod. No matter how I try to set up MT with QTGMC, I cannot push my CPU over 20% even when it is individually run. My script is a bit of an overkill but it does produce the results I like (SuperToon maintains dark lines well before dehaloing and I used multiple LSFmod calls under different settings to avoid distortion from awarpsharp2 used for line thinning). For reference, the source is a heavily haloed disc from Avatar the Last Airbender, I'll upload a clip tomorrow. I have one of the newer 4 core/8 threads, i7 4770k, and so far I'm capped out at 0.50FPS in MeGUI, the goal is 1FPS: MPEG2Source("C:\Iceberg.d2v") SetMemoryMax(1700) SetMTMode(1) AutoAdjust() QTGMC( Preset="Slower",FPSDivisor=2,Sharpness=0.2,SubPel=4,truemotion=true,chromamotion=true,globalmotion=true,ProgSADMask=18.0,edithreads=6) SuperToon(power=0.2, mode=2, Nthr=30, Nstr=0, Ncap=25, Lthr=0, Hthr=255, Lcap=255, cont=180, show=false) SuperToon(power=16, mode=3, Nthr=4, Nstr=0, Ncap=150, Lthr=0, Hthr=255, Lcap=255, cont=180, show=false) DeHalo_alpha_2BD(rx=2.0, ry=2.0, darkstr=1.0, brightstr=1.0, lowsens=15, highsens=20, ss=22.0) SetMTMode(2) GradFun2DBmod(thr=1.0, mode=2, str=0.8, strC=0, temp=50, adapt=0, mask=True, radius=4, range=4, show=False) UnsharpHQ(THRESHOLD=5, SHARPSTR=0.7, SMOOTH=0.4) SetMTMode(2) LSFmod(defaults="fast", smode=4,strength=140, kernel=19, preblur="OFF", secure=True, Szrp=16, Spwr=4, SdmpLo=4, SdmpHi=48, overshoot=1, undershoot=1, overshoot2=2, undershoot2=2, soothe=True, keep=50, edgemode=0, show=False,ss_x=4.00,ss_y=4.00) MSharpen(threshold=2, strength=74, mask=false, highq=true) SetMTMode(2) awarpsharp2(thresh=117,blur=12,type=1,depth=22,chroma=3) SetMTMode(2) LSFmod(defaults="fast", smethod=2,smode=3,strength=300, kernel=20, preblur="OFF", secure=True, Szrp=100, Spwr=6, SdmpLo=2, SdmpHi=10, overshoot=1, undershoot=1, overshoot2=2, undershoot2=2, soothe=True, keep=90, edgemode=0,edgemaskHQ=true,ss_x=4.00,ss_y=4.00) SetMTMode(2) GradFun2DBmod(thr=1.0, mode=2, str=0.8, strC=0, temp=50, adapt=0, mask=True, radius=4, range=4, show=False) SetMTMode(2) LSFmod(defaults="fast", smode=3,smethod=4,strength=293, kernel=19, preblur="OFF", secure=True, Szrp=255, Spwr=5, SdmpLo=24, SdmpHi=24, overshoot=1, undershoot=1, overshoot2=2, undershoot2=2, soothe=True, keep=80, edgemode=2,edgemaskHQ=true,ss_x=4.00,ss_y=4.00) SetMTMode(2) nnedi3_rpow2(rfactor=8,nsize=3,nns=4,cshift="Spline64Resize") SetMTMode(2) nnedi3_resize16(target_width=1440, target_height=1080, src_left=0, src_top=0, nsize=3,nns=4,kernel_d="spline", kernel_u="spline", f_d=1.0, f_u=1.0, taps=6, invks_d=True, invks_u=False, invkstaps=6, noring=True) |
30th December 2014, 21:39 | #6 | Link |
Registered User
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,407
|
Is that your script?
You only use SetMTMode() when changing the mode and most filters do not like mode 1: Code:
SetMemoryMax(1700) SetMTMode(3) MPEG2Source("C:\Iceberg.d2v") SetMTMode(2) AutoAdjust() QTGMC( Preset="Slower",FPSDivisor=2,Sharpness=0.2,SubPel=4,truemotion=true,chromamotion=true,globalmotion=true,ProgSADMask=18.0,edithreads=2) SuperToon(power=0.2, mode=2, Nthr=30, Nstr=0, Ncap=25, Lthr=0, Hthr=255, Lcap=255, cont=180, show=false) SuperToon(power=16, mode=3, Nthr=4, Nstr=0, Ncap=150, Lthr=0, Hthr=255, Lcap=255, cont=180, show=false) DeHalo_alpha_2BD(rx=2.0, ry=2.0, darkstr=1.0, brightstr=1.0, lowsens=15, highsens=20, ss=22.0) GradFun2DBmod(thr=1.0, mode=2, str=0.8, strC=0, temp=50, adapt=0, mask=True, radius=4, range=4, show=False) UnsharpHQ(THRESHOLD=5, SHARPSTR=0.7, SMOOTH=0.4) LSFmod(defaults="fast", smode=4, [snip]) MSharpen(threshold=2, strength=74, mask=false, highq=true) awarpsharp2(thresh=117,blur=12,type=1,depth=22,chroma=3) LSFmod(defaults="fast",smethod=2,smode=3, [snip]) GradFun2DBmod(thr=1.0, mode=2, str=0.8, strC=0, temp=50, adapt=0, mask=True, radius=4, range=4, show=False) LSFmod(defaults="fast", smode=3,smethod=4 [snip]) nnedi3_rpow2(rfactor=8,nsize=3,nns=4,cshift="Spline64Resize") nnedi3_resize16(target_width=1440, target_height=1080, [snip]) Maybe I will argue with one pair of lines specifically, why nnedi3_rpow2 and nnedi3_resize16? Why not simply use nnedi3_resize16? What is AutoAdjust() doing to help? (oops, another one) |
31st December 2014, 12:38 | #7 | Link |
Angel of Night
Join Date: Nov 2004
Location: Tangled in the silks
Posts: 9,559
|
Eight sharpeners in a row? Really? I can't come up with anything that could possibly benefit from that. Surrealism, maybe?
As a newbie, stick to 1s: One input, one deinterlace/ivtc, one smoother, one sharpener, one resizer, at most. Tweak the ones you have, first, and then add more one at a time, tweaking them, so you'll understand when and where to put them. Your way, it's just stacking turning the artifacts of each filter into more pronounced artifacts in the next. Purely from a CPU standpoint, QTGMC is probably nearly 50% of your processing time, and two instances of NNEDI3 probably the other other 50%. |
8th January 2015, 07:26 | #8 | Link |
Registered User
Join Date: Jan 2013
Posts: 12
|
Ah, ok thanks again guys! And yes, surrealism is a good way to describe the look that my script was giving me. Personally, I liked it on a couple of frames.... but once I encoded I realized how terribly it distorted other frames. I hope you guys at least got a good kick out of seeing such a crazy script!
So I took away the LSFMod's, the first NNEDI3 and awarpsharp and my speed jumped up to 0.90 FPS! I also added MCAA to get rid of some (but not all) of the shimmering I got after QTGMC. Unfortunately, I can't set up any source functions to MT because I cannot get the script to stay under 2GB RAM usage so avs4x264mod doesn't crash, no matter what I set my memory limit to. So I'm pleased considering the heavily haloed source. I'll post a before and after example clip once I tweak a few more things. I'll also post and example of some residual outlines that appear in some succeeding frames (don't know exactly what to call it, possibly chroma bleeding). Here is my current script: SetMemoryMax(1900) MPEG2Source("C:\Desktop\Vid\VTS_03_CHAPTER_02_1.d2v") SetMTMode(2) QTGMC(Preset="Slower",FPSDivisor=2,Sharpness=0.2,SubPel=4,truemotion=true,chromamotion=true,globalmotion=true,ProgSADMask=18.0) MCAA(dark=0.0, thin=0, sharp= 10, smooth=1, stabilize=true, aafilter="taa", predown=false, predownMC=false, aarepair=24) DeHalo_alpha_2BD(rx=2.0, ry=2.0, darkstr=1.0, brightstr=1.0, lowsens=2, highsens=2, ss=10.0) MSharpen(threshold=5, strength=28, mask=false, highq=true) UnsharpHQ(THRESHOLD=8, SHARPSTR=0.1, SMOOTH=2.0) HQDeringmod(mrad=4, msmooth=4,mthr=200, minp=4, dither=6) nnedi3_resize16(target_width=1440, target_height=1080, src_left=0, src_top=0, nsize=0,nns=4,kernel_d="spline", kernel_u="spline", f_d=1.0, f_u=1.0, taps=6, invks_d=True, invks_u=true, invkstaps=6, noring=True) |
8th January 2015, 16:19 | #9 | Link | ||
Registered User
Join Date: Mar 2011
Posts: 4,829
|
Quote:
If I'm only running one encode, I make a copy or two of the script and split the encoding job by adding Trim() to the end of each, so each script encodes part of the video, then I run them simultaneously. The encoded video can be appended later with MKVMergeGUI, as long as "stitchable" is checked in the x264 encoder configuration. My main issue with MT Avisynth was not being able to get it 100% stable. A couple of times I left the encoding job running and returned to find the PC had spent most of the time displaying an error message instead of encoding, so I went back to single threaded mode. Quote:
I ask because someone at VideoHelp was asking about slow encoding speed recently and avs4x264mod seemed like it might be the culprit. The way I understand it MeGUI uses avs4x264mod when Avisynth is 32 bit and the x264 encoder is 64 bit. I don't know if there's a way to bypass it when running a 64 bit OS. I'm using XP myself so avs4x264mod is never used. flacattack, Out of curiosity, what sort of speed to you get if you only use QTGMC from your previous script and nothing else? Last edited by hello_hello; 8th January 2015 at 16:23. |
||
8th January 2015, 21:40 | #10 | Link | |
Registered User
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,407
|
Quote:
There are other options though, I use a newer version of avs2yuv so I am not sure how avs4x264mod works but maybe avs4x264mod doesn't call distributor() itself so it would need to be added to the end of a MT script. |
|
9th January 2015, 02:18 | #11 | Link |
Registered User
Join Date: Jan 2013
Posts: 12
|
I stopped using awarpsharp, but I'm still getting a lot of "motion blobbing" in my final video, especially around lines. Would NNEDI3 or Msharpen be the culprit?
hello_hello: The encoding speed jumps up to about 7.5-8FPS just using QTGMC. Which is slow, but a heck of a lot better than the current 0.81FPS I'm getting. Speed decreases by a good amount if I stack anything after in the script, no matter what it is. Asmodian: I'm using 32-bit Avisynth on my 64-bit system, but MeGUI defaults to using the 32-bit version of avs4x264mod. That may be part of the issue. Is there any way to make MeGUI default to 64-bit x264 encoding in this case? |
9th January 2015, 02:47 | #12 | Link |
Registered User
Join Date: Jan 2013
Posts: 12
|
Here is an example of how I managed to exaggerate some of the blobbing/blurring on finer areas. Still some aliasing present too.
http://www32.zippyshare.com/v/7752381/file.html Here is the source: http://www27.zippyshare.com/v/1142365/file.html Last edited by flacattack; 9th January 2015 at 03:07. |
9th January 2015, 04:26 | #13 | Link |
Angel of Night
Join Date: Nov 2004
Location: Tangled in the silks
Posts: 9,559
|
I think you're thrashing the cache, that would explain why speed drops considerably: You have no room left to store processed old frames, but another filter wants them anyway, so Avisynth has to rerun QTGMC to recreate them. The more temporal filters you have, the more cache thrashing you get, as the radius of frames that need to be touched expands. Since you're hitting the 2GB barrier anyway, just raising the cache size will only lead to out-of-memory crashes.
If you stay in 32-bit mode you only have one good choice: chain multiple avisynth scripts in series, so each gets its own 2GB slice of memory. If you can find 64-bit of everything you need (or switch to Vapoursynth), you can alleviate that problem, as long as you have enough memory. |
9th January 2015, 09:00 | #14 | Link | |||
Registered User
Join Date: Mar 2011
Posts: 4,829
|
Quote:
Quote:
After making a copy of the script and running the two together, the second is doing 0.78fps, so I'm totalling about 2fps with CPU usage roughly 90%. According to Task Manager, each x264 process is using about 1.3GB of RAM. Could multi-threading be slowing things down somehow? Quote:
I think there's only a 32 bit version of avs4x264mod, which then sends the video to the 64 bit encoder. |
|||
9th January 2015, 10:05 | #15 | Link | |||
Registered User
Join Date: Mar 2011
Posts: 4,829
|
Quote:
I never work with animation so I don't know what'd be the best way to IVTC it, although something simple like this seems to work: Quote:
http://avisynth.nl/index.php/Vinverse/2.5 I don't really know where you'd go from there to clean it up as animation's definitely not my thing, but I'm not sure QTGMC is the right tool for the job. Edit: Quote:
DGIndex creates d2v index files, and for DGDecodeNV it's dgi, so that seems pretty normal. Last edited by hello_hello; 9th January 2015 at 10:26. |
|||
10th January 2015, 03:05 | #16 | Link |
Registered User
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,407
|
avs4x264mod needs to be 32-bit to connect with 32-bit Avisynth, it can then pipe the video to 64-bit x264. If you were using 32-bit x264 you would not need to use avs4x264mod and could simply give x264 the script directly.
|
6th May 2015, 21:17 | #17 | Link | |
Registered User
Join Date: Mar 2011
Posts: 4,829
|
I'm back posting in this thread because suddenly I'm suffering from a similar speed problem to flacattack with some specific videos, and the problem appears to be 100% repeatable. The source video is h264. If I start encoding from the beginning, the encoding speed starts off normally (very slow), but never achieves "normal" speed, and gradually slows to a crawl while CPU usage also slowly drops. If I start the encoding a few minutes in, the speed stays consistent. It's not like the video itself is more complex at the beginning and causing encoding speed to slow though, because when I let the encode run from the beginning it never speeds up again.
This is from MeGUI's log file (x264 standard error stream) after encoding the first few minutes of video with the following script: Quote:
I assume the above would be the average encoding speed and the actual speed by the end of the encode would have been even slower. If I change Trim() in the script to encode from frame 3814 to the end, the log file tells me this: encoded 68188 frames, 3.46 fps, 3192.46 kb/s I'm at a complete loss as to why encoding from the beginning causes encoding speed to slow, but I've been able to repeat it quite reliably with these sources (I watch it gradually slowing courtesy of MeGUI's encoding status window), to the point I'm encoding the remainder by splitting each encode into three sections and appending the encoded video together..... A small first section to get the slow encoding out of the way, then splitting the remaining video into two sections I can encode simultaneously at normal speed. It's quite baffling. My testing so far seems to indicate it only happens when QTGMC and LSFMod are used in the same script. Individually, they don't cause a problem. Next I'll try changing the plugins they have in common (Masktools2 and RemoveGrain, I think) and test some more. If I discover anything new, I'll post back. I'm sure I've bumped into this problem previously, where the odd encode has seemed to run unusually slowly, and maybe once or twice where after restarting the encode it's run at normal speed, but it's not become an obvious problem until now, where it's 100% repeatable. I'm using XP and AVisynth+ (portable version courtesy of MeGUI). Edit: Problem solved, I hope! (It seems for me the problems in question only occurs when running QTGMC in progressive mode, so it's possibly not related to flacattack speed problem as I thought it might be) After running a bunch of encodes with various versions of MaskTools2, I again confirmed that this version is the fastest, but not the cause of my problem. That always seems to be the last thing I test, which in this case was the QTGMC script itself. Switching from the Dogway modded version of QTGMC (version 3.33d) back to version 3.33 seems to have solved the problem. I'll run some more encodes from scratch now while monitoring encoding speed and if the problem appears to be fixed I'll report the issue in Dogway's thread over at VideoHelp to see if he has any idea as to what's happening. (I don't know if there's an "official" link for version 3.33 as the QTGMC thread still lists the latest official version as 3.32, but there's a link for 3.33 buried in the Doom9 QTGMC thread somewhere..... in case anyone is wondering) Edit 2: Much better. After switching QTGMC from 3.33d to 3.33, encoding speed for the script posted above went from this: encoded 3814 frames, 0.97 fps, 3548.04 kb/s To this: encoded 3814 frames, 4.00 fps, 3562.11 kb/s Interesting the bitrate changed a tad, but it's still a mystery as to why the speed only takes a hit when encoding from the beginning of the video...... Last edited by hello_hello; 7th May 2015 at 21:47. |
|
Tags |
dvd, ffmpegsource2, mpeg2, time, timing |
|
|