Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 13th August 2008, 20:14   #1  |  Link
Perscitus
Registered User
 
Join Date: Jan 2007
Posts: 6
Getting terrible results encoding anime.

It has been almost 2 years since I was encoding video and I'm trying to make a backup of an anime. I have spent a good amount of time scouring the forums for info and I'm getting awful results. So I'm looking for any suggestions of what I'm doing wrong. My goal is to create a high quality x264 encode of an anime with vfr and I'm trying to fit it to a 175MB(or 233MB) file size.

What I'm using: MeGui 0.3.0.1020, AviSynth 2.57, x264 (928 Jarod's patched build), DGIndex 1.5.0.0, nnedi 1.3.0.0, TIVTC 1.0.5.0, TDeint 1.1.0.0, TFM 1.0.4, MKVMerge 2.2.0

......1st Pass:
AviSynth
Code:
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,output="c:\DVD\Working\test01-tfm.txt")
tdecimate(mode=4,output="c:\DVD\Working\test01-tdec.txt")


crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
Lanczos4Resize(640,352)
x264
Code:
program --pass 1 --bitrate 905 --stats "C:\DVD\Working\test01.stats" --bframes 5 --b-pyramid --direct auto --filter 1,1 --subme 1 --partitions none --me dia --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "input" --psy-rd 0
......2nd Pass:
Avisynth
Code:
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,input="C:\DVD\Working\test01-tfm.txt")
tdecimate(mode=5,hybrid=2,vfrDec=0,input="C:\DVD\Working\test01-tdec.txt",tfmIn="C:\DVD\Working\test01-tfm.txt",mkvOut="C:\DVD\Working\test01-timecodes.txt",batch=true)

crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
Lanczos4Resize(640,352)
x264
Code:
program --pass 2 --bitrate 905 --stats "C:\DVD\Working\test01.stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid --b-rdo --bime --direct auto --filter 1,1 --subme 6 --trellis 2 --partitions none --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --psy-rd 0
My Result: download here
My Source: I'm not sure if it violates rules to link the VOB of the source, I will link it(or not) as soon as I find this out.
__________________
One must be a fox in order to recognize traps, and a lion to frighten off wolves.
- Niccolò Machiavelli
Perscitus is offline   Reply With Quote
Old 13th August 2008, 20:40   #2  |  Link
DarkZell666
aka XaS
 
DarkZell666's Avatar
 
Join Date: Jun 2005
Location: France
Posts: 1,122
Any particular reason you used --partitions none ?
Using that certainly doesn't help, particularly on the second pass. Use the default setting instead.

Also, why use vfr for output ? That's really looking for extra trouble since static frames barely take up any space at all (x264 marks the static parts of the picture as "skip" and doesn't even encode them). You can easily fit a clean 25min anime episode unresized (720x576 anamorphic, or 720x480 anamorphic) on 175MB and still get acceptable quality (though what's "acceptable" is very subjective).

When I say "clean", I mean correctly denoised. Try adding hqdn3d(2,2,2) just before the resizing step to your script (you'll need the hqdn3d plugin), since DVD's all have noise, even anime ones (and the video you posted is full of noise). If that isn't enough, try hqdn3d() on default settings or fft3dfilter() on default settings instead (much slower).

(I'll let you google the pages for those 2 filters, they're easy to find ).

Also, what you posted isn't as awful as I first imagined. Also, why disable psy-rd ? Was it worse with it enabled ?
__________________

Q9300 OC @ 3.2ghz / Asus P5E3 / 4GB PC10600 / Geforce 8600 GTS

Last edited by DarkZell666; 13th August 2008 at 20:44.
DarkZell666 is offline   Reply With Quote
Old 13th August 2008, 21:11   #3  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,092
You know what? Try again with RipBot264. Enable Denoiser (ANIME profile) and everything should be ok.
Atak_Snajpera is offline   Reply With Quote
Old 13th August 2008, 21:31   #4  |  Link
cogman
The Crazy Idahoan
 
Join Date: Jun 2007
Location: Idaho
Posts: 233
Also, it looks like your settings for first pass and second pass are almost too different. For example. Im pretty sure that Psy-rdo should be enabled on both passes.

Upping the bframes to 16 should help somewhat as well (since this is anime). an really, why aren't you using all partitions?

You can try the "Dup" plugin for avi-synth as well, it should offer some savings to you.
cogman is offline   Reply With Quote
Old 13th August 2008, 23:34   #5  |  Link
Adub
Fighting spam with a fish
 
Adub's Avatar
 
Join Date: Sep 2005
Posts: 2,685
Nah, he is disabling Psy-rdo on both passes. Some tests have shown that it doesn't help with anime as it does with live action material.
__________________
FAQs:Bond's AVC/H.264 FAQ
Site:Adubvideo
Adub is offline   Reply With Quote
Old 14th August 2008, 06:27   #6  |  Link
Perscitus
Registered User
 
Join Date: Jan 2007
Posts: 6
I took some of the suggestions and made a few changes, definite improvement. I don't know how I managed to miss "--partitions none", that shouldn't have happened. Anyway, here are the updated settings and sample.

......1st Pass:
AviSynth
Code:
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\nnedi.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\hqdn3d.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,output="c:\DVD\Working\test01-tfm.txt")
tdecimate(mode=4,output="c:\DVD\Working\test01-tdec.txt")


crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
hqdn3d(2,2,2)
Lanczos4Resize(640,352)
x264
Code:
program --pass 1 --bitrate 905 --stats "C:\DVD\Working\test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "input"

......2nd Pass:
AviSynth
Code:
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\nnedi.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\hqdn3d.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,input="C:\DVD\Working\test01-tfm.txt")
tdecimate(mode=5,hybrid=2,vfrDec=0,input="C:\DVD\Working\test01-tdec.txt",tfmIn="C:\DVD\Working\test01-tfm.txt",mkvOut="C:\DVD\Working\test01-timecodes.txt",batch=true)

crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
hqdn3d(2,2,2)
Lanczos4Resize(640,352)
x264
Code:
program --pass 2 --bitrate 905 --stats "C:\DVD\Working\test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input"
Result: download here
The most noticeable issue I still have is apparent in the solid color areas, primarily the dark ones. Am I getting too picky or is there a filter that would fix that?
__________________
One must be a fox in order to recognize traps, and a lion to frighten off wolves.
- Niccolò Machiavelli
Perscitus is offline   Reply With Quote
Old 14th August 2008, 12:31   #7  |  Link
cogman
The Crazy Idahoan
 
Join Date: Jun 2007
Location: Idaho
Posts: 233
I may be wrong here. But the problem may stem from the AQ settings. Try relaxing the strength a bit.
cogman is offline   Reply With Quote
Old 14th August 2008, 12:57   #8  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
it may be.
try disabling AQ reducing the strenght to 0. see if the situation gets better and if it does then try to find the sweetspot rising the AQ strenght until you get the desired results.

Last edited by Sharktooth; 14th August 2008 at 15:42.
Sharktooth is offline   Reply With Quote
Old 14th August 2008, 15:27   #9  |  Link
desta
Registered User
 
Join Date: Jan 2006
Posts: 220
Quote:
Originally Posted by cogman View Post
Also, it looks like your settings for first pass and second pass are almost too different. For example. Im pretty sure that Psy-rdo should be enabled on both passes.
I was led to believe that psy-rdo wasn't required for the first pass.
desta is offline   Reply With Quote
Old 14th August 2008, 15:43   #10  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
infact it's not.
Sharktooth is offline   Reply With Quote
Old 15th August 2008, 05:45   #11  |  Link
Shinigami-Sama
Solaris: burnt by the Sun
 
Shinigami-Sama's Avatar
 
Join Date: Oct 2004
Location: /etc/default/moo
Posts: 1,923
Quote:
Originally Posted by Sharktooth View Post
infact it's not.
+1
useless on first pass
__________________
Quote:
Originally Posted by benjust View Post
interlacing and telecining should have been but a memory long ago.. unfortunately still just another bizarre weapon in the industries war on image quality.
Shinigami-Sama is offline   Reply With Quote
Old 15th August 2008, 05:59   #12  |  Link
Comatose
Registered User
 
Join Date: Dec 2007
Posts: 639
So does it do anything or does it get disabled?
Comatose is offline   Reply With Quote
Old 15th August 2008, 06:02   #13  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by Comatose View Post
So does it do anything or does it get disabled?
Its on if you use subme=6/etc... it is potentially useful, but only if you have time to waste.
Dark Shikari is offline   Reply With Quote
Old 15th August 2008, 06:07   #14  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,413
You know that with VFR, the first (mode 4) and second (mode 5) pass of the avisynth have to be run first, then the first and second encoding pass? To streamline it, you can run the first x264 pass over both avisynth passes, overwriting the first's stats, then run the second x264 pass. If you run the first pass on the first avisynth and the second on the other, you will be destroying rate control, scene-cut detection, and probably getting wildly undersized files as well.

Make sure that bitrate is also based on the 2nd pass avisynth's length, not the original's. They'll be different since you didn't change the FPS.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 15th August 2008, 11:55   #15  |  Link
Comatose
Registered User
 
Join Date: Dec 2007
Posts: 639
Quote:
Originally Posted by Dark Shikari View Post
Its on if you use subme=6/etc... it is potentially useful, but only if you have time to waste.
Time to do some testing, then!
Comatose is offline   Reply With Quote
Old 15th August 2008, 15:09   #16  |  Link
Perscitus
Registered User
 
Join Date: Jan 2007
Posts: 6
Quote:
Originally Posted by foxyshadis View Post
You know that with VFR, the first (mode 4) and second (mode 5) pass of the avisynth have to be run first, then the first and second encoding pass? To streamline it, you can run the first x264 pass over both avisynth passes, overwriting the first's stats, then run the second x264 pass. If you run the first pass on the first avisynth and the second on the other, you will be destroying rate control, scene-cut detection, and probably getting wildly undersized files as well.

Make sure that bitrate is also based on the 2nd pass avisynth's length, not the original's. They'll be different since you didn't change the FPS.
So should I be making an analysis pass, followed by a lossless encode with only IVTC filter applied. Then use that as my source for the 1st and 2nd pass with other AviSynth filters applied and compression(x264)? If I were to change the mode=5 to mode=2 for the second pass AviSynth script, I know it wouldn't be vfr(I didn't realize vfr wasn't a big help for x264) anymore, but in that case, I would be OK running just the 1st and 2nd passes, right?

BTW, I turned off AQ and it seemed to help it out a little bit and made it less noticeable.
__________________
One must be a fox in order to recognize traps, and a lion to frighten off wolves.
- Niccolò Machiavelli

Last edited by Perscitus; 15th August 2008 at 15:13.
Perscitus is offline   Reply With Quote
Old 15th August 2008, 16:44   #17  |  Link
ChronoCross
Does it really matter?
 
ChronoCross's Avatar
 
Join Date: Jun 2004
Location: Chicago, IL
Posts: 1,542
Quote:
Originally Posted by Perscitus View Post
So should I be making an analysis pass, followed by a lossless encode with only IVTC filter applied. Then use that as my source for the 1st and 2nd pass with other AviSynth filters applied and compression(x264)? If I were to change the mode=5 to mode=2 for the second pass AviSynth script, I know it wouldn't be vfr(I didn't realize vfr wasn't a big help for x264) anymore, but in that case, I would be OK running just the 1st and 2nd passes, right?

BTW, I turned off AQ and it seemed to help it out a little bit and made it less noticeable.
1) Take your 1st pass script and edit out everything except TFM and tdecimate. run an analysis pass with the absolute fastest settings OR load it into vdub and play the video all the way through. This will create the timecodes files.
2) Take your 2nd pass script and run 2 passes of x264 like you normally would as an encode.
ChronoCross is offline   Reply With Quote
Reply

Tags
anime, quality, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.