Can I make these options more efficient?

[Digital Corpus]

I'm running a personal server for recording and encoding HD TV. Depending on network, the source can be kind of bad where I get major blocking in fade to/from blacks and cross fades. I've set this up to be a fully autonomous project with predictable results in speed and quality. I have a few target devices and other misc resolutions and bitrates I encode for. I would use x264's presets but I cannot when using mencoder. Since even with 98% reliability of signal quality with OTA broadcasts I still get the occasional split second drop, mencoder does a better job and slewing A/V sync back to what it should be. As such, I set everything manually and through mencoder. I've done my best to convert the syntax but I might have made a mistake somewhere.

I have 4 basic presets in two groups. Basically "SD" and "HD" types of resolutions. Bitrate varies between resolutions but in a close to linear ratio of # macroblocks per frame. 1080p is the exception due to storage constraints.

Low-Res preset #1
--subme 8 --no-mbtree --no-fast-pskip --dct-decimate --keyint 240 or 300 --keyint-min 12 or 15 --mixed-refs --deblock 0:0 --bframes 0 --b-adapt 0 --no-cabac --qpmin 1 --qpmax 51 --qpstep 25 --ratetol 25 --qcomp 0.50 --direct auto --partitions i4x4,p4x4,p8x8 --no-8x8dct --psy-rd 0.4,0.2 --trellis 1

Low-Res preset #2
--subme 8 --no-mbtree --no-fast-pskip --dct-decimate --keyint=$max-key --keyint-min=$min-key --mixed-refs --deblock 1:1 --bframes 0 --b-adapt 0 --no-cabac --qpmin 1 --qpmax 51 --qpstep 25 --ratetol 35 --qcomp 0.75 --direct auto --partitions i4x4,p4x4,p8x8 --no-8x8dct --psy-rd 0.4,0.2 --trellis 1

Hi-Res preset #1
--no-mbtree --fast-pskip --no-dct-decimate --keyint=$max-key --keyint-min=$min-key --mixed-refs --deblock 2:2 --bframes 1 --b-adapt 0 --b-pyramid strict --cabac --qpmin 1 --qpmax 51 --qpstep 25 --ratetol 35 --qcomp 0.75 --direct auto --partitions i8x8,p8x8,b8x8 --8x8dct --psy-rd 0.4,0 --trellis 0

Hi-Res preset #2
--no-mbtree --fast-pskip --no-dct-decimate --keyint=$max-key --keyint-min=$min-key --mixed-refs --deblock 3:3 --bframes 2 --b-adapt 0 --b-pyramid strict --cabac --qpmin 1 --qpmax 51 --qpstep 15 --ratetol 15 --qcomp 0.75 --direct auto --partitions i8x8,p8x8,b8x8 --8x8dct --psy-rd 0.4,0 --trellis 0

Resolution specific options (n = native = no super-sampling/scaling)
160x90 --bitrate 80 --ref 6 --merange 4 --me umh --level 20 --threads 3 # Low Res #1
228x128 --bitrate 160 --ref 6 --merange 8 --me umh --level 20 --threads 3 # Low Res #1
320x180 --bitrate 320 --ref 6 --merange 8 --me umh --level 20 --threads 4 # Low Res #2
480x270 --bitrate 690 --ref 6 --merange 16 --me umh --level 21 --threads 4 # Low Res #2
640x360 --bitrate 1260 --ref 6 --merange 16 --me umh --level 30 --threads 5 # Low Res #2
854x480 --bitrate 2160 --ref 6 --merange 16 --me hex --level 31 --subme 9 --threads 5 # Hi Res #1
1024x576 --bitrate 3150 --ref 6 --merange 16 --me hex --level 31 --subme 7 --threads 5 # Hi Res #1
1280x720 --bitrate 4950 --ref 5 --merange 16 --me dia --level 31 --subme 6 --threads 5 # Hi Res #2 Down sampled from inverse telecine 1080i content
1280x720 --bitrate 4950 --ref 5 --merange 16 --me dia --level 31 --subme 6 --threads 5 # Hi Res #2
1920x1080 --bitrate 9000 --ref 4 --merange 16 --me dia --level 41 --subme 7 --threads 4 # Hi Res #2 Inverse telecine 1080i content
(I specifically checked each merange value in multiples of 4. Increaseing them further resulted in no increase of quality based on psnr or ssim.)

Naturally the lower resolutions naturally cannot be threaded as well as the higher resolutions, or so I've learned. I would still like to learn what I can/should tweak in the first two in order to increase threading efficiency if possible and increase their quality a bit if I can.

I added a fixed number of b-frames in my higher res encodings since they actually sped up my encodings while their increasing quality. Now about speed, I have found it to be considerably faster to encode and inverse telecine a 1080i @ 30 fps stream to 720p @ 24 fps than it is to encode a 720p @ 60 fps stream to 1024x576 @ 30fps despite having more macroblocks per second. I would much like to solve this conundrum if possible. However, I can encode at real time or faster speeds when I drop resolutions "one notch" like the above examples and I do not want to loose this speed. and I actually would like to gain more speed in this area if possible without a drop in quality.

I've done my best to understand what each option does to the encoding and these settings are my 3rd revision from when I started this project more than a year ago. By no means am I an expert and in some respects I'm a complete n00b. I would like to ask if you guys don't mind helping me refine these settings to make encoding more efficient for compression and speed.

One last note, I use ABR and not CRF since I much prefer to have some control over final file size output. A 5-10% deviation is acceptable for my needs. Unless CFR provides avery significant difference in quality/speed, I don't see it as a viable alternative.

Thank you all greatly in advance

[nm]

Originally Posted by Digital Corpus

One last note, I use ABR and not CRF since I much prefer to have some control over final file size output. A 5-10% deviation is acceptable for my needs. Unless CFR provides avery significant difference in quality/speed, I don't see it as a viable alternative.

Depends on what you consider significant, but generally it does. Why do you need to have control over the size of individual encodes? How about VBV-capped CRF if it's a bandwidth issue?

[Digital Corpus]

That is a thought worth considering. Now I have a lot of tweaks for rate control so I'm not sure how those would be needed or not if I used CRF. Do you think there would be likely drops in quality with the use of VBV? I'm still trying to understand it so I can possibly use it for encoding high[er] quality streaming video.

[nm]

Originally Posted by Digital Corpus

That is a thought worth considering. Now I have a lot of tweaks for rate control so I'm not sure how those would be needed or not if I used CRF.

I don't think they are needed with ABR either. Setting ratetol higher than default may be a good idea in your case, but I wouldn't touch those other settings.

Also, you seem to be tweaking lots of things back and forth between the presets and the resolution-specific options. Personally I'd just copy x264's medium or slow preset and only change things that are required by the playback devices.

Do you think there would be likely drops in quality with the use of VBV?

That depends on many things. If you can set it high enough (perhaps 15 Mbps with a 10 second buffer for 1080p), it would probably be fine considering your OTA sources.

[Dark Shikari]

Originally Posted by Digital Corpus

I added a fixed number of b-frames in my higher res encodings since they actually sped up my encodings while their increasing quality.

Oh dear...

Originally Posted by Digital Corpus

--no-mbtree --fast-pskip --no-dct-decimate --keyint=$max-key --keyint-min=$min-key --mixed-refs --deblock 2:2 --bframes 1 --b-adapt 0 --b-pyramid strict --cabac --qpmin 1 --qpmax 51 --qpstep 25 --ratetol 35 --qcomp 0.75 --direct auto --partitions i8x8,p8x8,b8x8 --8x8dct --psy-rd 0.4,0 --trellis 0

Everything bolded here made me want to stab myself with a rusty spoon. If you don't understand why each of those options makes me want to acquire some corroded utensils, you shouldn't be using them.

Originally Posted by Digital Corpus

I would like to ask if you guys don't mind helping me refine these settings to make encoding more efficient for compression and speed.

I think you need to throw everything out and settle on one or two simple commandlines with about 1/3 as many options each, because your current options are completely nonsensical. In fact, you could get quite good results simply by dropping every single option listed there.

It is posts like this that make me want to pull a Theora and remove almost all user-facing options completely because if you give someone a button, they will press it. And even after it shoots them in the leg, they'll go limping around for more buttons to press!

[Digital Corpus]

Originally Posted by Dark Shikari

Oh dear...Everything bolded here made me want to stab myself with a rusty spoon. If you don't understand why each of those options makes me want to acquire some corroded utensils, you shouldn't be using them.I think you need to throw everything out and settle on one or two simple commandlines with about 1/3 as many options each, because your current options are completely nonsensical. In fact, you could get quite good results simply by dropping every single option listed there.

I recently found the x264 wiki. I never knew it existed so I've been operating off of documentation in mplayer and google. I was hoping for a more mature response that actually was useful in helping me understand the error of my ways. I gave you a breif overview as to why I have all the options listed as I cannot use the presets via mplayer afaik.

Not using MB-tree was a decision I had to make since it increase encoding time by about 25% when I first started using it. I have a copy of x264 from 11-24-09. I've tried manually compiling mplayer and x264 separately instead of through portage, but mplayer isn't reading the x264 install. Until that is fixed, I cannot easily tell how much the recent fixes have sped up mbtree so I have it disabled.

As for mixed-refs, I didn't see enough of a reason to unset it. I have it listed so I know what is default so I know it is redundant.

The combination of bframe settings was picked due to seeing an increase in psnr and ssim while still maintaining faster than real time encoding speed. I would use my own eye to judge this, but they become glazed over when watching the same clip more than about 5 times. If you want to knock me for this logic, explain why.

Now for the various qp settings, I set those ages ago and see through the wiki that the options I have are fairly stupid. This kind of detail should be listed under the logical "--fullhelp" since it's kinda fairly useful. If poor documentation makes for poor decisions, then don't get mad at the user for uninformed decisions.

Partition selections and 8x8dct are chosen for speed reasons and negligible differences in qualitly. The most understandable explination as to how these related to each other I found in mplayer's man page:

       partitions=<list>
              Enable some optional macroblock types (default: p8x8,b8x8,i8x8,i4x4).
                 p8x8 Enable types p16x8, p8x16, p8x8.
                 p4x4 Enable types p8x4, p4x8, p4x4.  p4x4 is recommended only with subq >= 5, and only at low res-
                      olutions.
                 b8x8 Enable types b16x8, b8x16, b8x8.
                 i8x8 Enable type i8x8.  i8x8 has no effect unless 8x8dct is enabled.
                 i4x4 Enable type i4x4.
                 all  Enable all of the above types.
                 none Disable all of the above types.
              Regardless of this option, macroblock types p16x16, b16x16, and i16x16 are always enabled.
              The idea is to find the type and size that best describe a certain area of the picture.  For example,
              a global pan is better represented by 16x16 blocks, while small moving objects are better represented
              by smaller blocks.

I am not using i4x4, p4x4, or b4xb for reasons explained in the above excerpt, and for that the resolutions I have setup to work with these settings, a 4x4 pixel area is roughly equivalent to the noise in the source materials I'm working with. Not using them noted a very small penalty in quality and a lot more speed.

Originally Posted by Dark Shikari

It is posts like this that make me want to pull a Theora and remove almost all user-facing options completely because if you give someone a button, they will press it. And even after it shoots them in the leg, they'll go limping around for more buttons to press!

If x264 had a bit more documentation in it's install, I bet post like this would happen less often. The info directly from x264 is cryptic at best. Even mplayer's documentation and explanation is better despite it being outdated. Not all of us know or understand how everything works like you do. I already explained why I have the options set as I do, please read what I have posted. If I had already found a fast resource economical way to keep A/V sync integrity over less than optimal sources from mplayer and passing a YUV stream directly to x264, I wouldn't be pressing all the buttons. When managing over 3TiB of data of video files in my spare while having a life, it's easier to just have everything listed and set in the script instead of find out out one days everything just slowed down; e.g. the mbtree addition caught me off guard and reduced my capacity to encode video by a very large margin. I already asked for a little respect when posting this and indirectly said that something is bound to be "stupid" rubbing it in my face without saying why is highly immature.

[Dark Shikari]

Originally Posted by Digital Corpus

If x264 had a bit more documentation in it's install, I bet post like this would happen less often.

It is not x264's job to teach you how video encoding works. That's the job of a textbook.

If you don't want to learn how the internals of x264 work, use the presets. x264 --fullhelp lists them, so you can easily copy them over to mencoder.

Originally Posted by Digital Corpus

The info directly from x264 is cryptic at best. Even mplayer's documentation and explanation is better despite it being outdated. Not all of us know or understand how everything works like you do.

And here you completely miss the point. The documentation is "cryptic" because it is not intended for you. It is intended for video professionals who know what the options mean. Ordinary users who don't understand the internals should not be messing with dozens of obscure options. The basic options that an ordinary user should need are listed in the --help: everything beyond that shouldn't be touched unless you know what you're doing.

Originally Posted by Digital Corpus

I was hoping for a more mature response that actually was useful in helping me understand the error of my ways.

What else is there to explain? You arbitrarily poked buttons that you weren't supposed to poke, and then complained when I told you to stop poking them.

[Digital Corpus]

Originally Posted by Dark Shikari

It is not x264's job to teach you how video encoding works. That's the job of a textbook.

If you don't want to learn how the internals of x264 work, use the presets. x264 --fullhelp lists them, so you can easily copy them over to mencoder.And here you completely miss the point. The documentation is "cryptic" because it is not intended for you. It is intended for video professionals who know what the options mean. Ordinary users who don't understand the internals should not be messing with dozens of obscure options.What else is there to explain? You arbitrarily poked buttons that you weren't supposed to poke, and then complained when I told you to stop poking them.

Please don't choose your time mocking me instead. That is simply irrational and counter-intuitive. I'm not asking for a full explanation, but even a copy of the wiki into --fullhelp so it is then "full" seems logical. Everyone has to start somewhere. Search works, but only if you think of the exact way that someone else does to pull up the info you need. If you don't want to provide *some* help, why post an answer?

I thought my first post was concise enough to explain why I had to choose the route I did so I would not be mocked. I was wrong. I didn't want to write an essay explaining the countless hours I've spent staring at dozens of video encodings, looking at metrics, and comparing encoding times and the decisions I made based on all of that. Half of what I read on these boards states that the quality is subjective. If that is true, then why mock me for finding something that works for me?

[Dark Shikari]

Originally Posted by Digital Corpus

Please don't choose your time mocking me instead.

{enormous amounts of tl;dr omitted}

Wow. I try to help, I give you instructions on how to get better results, I explain the answer to many of your questions, and now I get accused of mocking you.

I would advise you next time to not attack the people who are trying to help you.

Good luck finding someone to spend hours for free trying to meet your unreasonable expectations.

[Blue_MiSfit]

LOL!!!!

Oh man..

He's right, you know - all drama set aside

Just use something very simple, like the defaults!!!! If it's too slow, speed it up with --preset fast or something. If you can tolerate slower speeds, then use --preset slow, slower, or veryslow.

Add --tune film or --tune animation if you want.

And of course, do specify the rate control mode you're interested in. If you must have control over file size, 2 pass is the way to do it. If you can afford some unpredictibility, then CRF with some VBV isn't a terrible idea.

~MiSfit

[Digital Corpus]

Originally Posted by Blue_MiSfit

LOL!!!!

Oh man..

He's right, you know - all drama set aside

Just use something very simple, like the defaults!!!! If it's too slow, speed it up with --preset fast or something. If you can tolerate slower speeds, then use --preset slow, slower, or veryslow.

Add --tune film or --tune animation if you want.

And of course, do specify the rate control mode you're interested in. If you must have control over file size, 2 pass is the way to do it. If you can afford some unpredictibility, then CRF with some VBV isn't a terrible idea.

~MiSfit

I understand that, but I was hoping for a response a bit more professional such as yours. I know it's a long post, but I stated why I haven't used the presets. I have kept this simple, but unfortunately I need reliability since I'm going through about 15 hours a week of video in several different resolutions. Letting mplayer/mencoder handle possibly broken transport streams is simpler than demuxing the video with projectX, finding where the audio gaps are if just the audio corrupted and then doing the same with video, and then reassembling everything together and passing it through x264 with presets to use. Granted mencoder will have to slew the audio or video to bring them back in sync, but it is simpler since I don't have to deal with that. As I "have" to use mplayer, the presets were not an option. Though at the same time, mbtree isn't an option that exists in mencoder but I can still pass it to the x264 code. That was a surprise and I have to admit I haven't tested if the preset will work the same way since it wasn't broken, I didn't want to break it to fix it with my not so much free time.

[Dark Shikari]

Originally Posted by Digital Corpus

I understand that, but I was hoping for a response a bit more professional such as yours. I know it's a long post, but I stated why I haven't used the presets. I have kept this simple, but unfortunately I need reliability since I'm going through about 15 hours a week of video in several different resolutions. Letting mplayer/mencoder handle possibly broken transport streams is simpler than demuxing the video with projectX, finding where the audio gaps are if just the audio corrupted and then doing the same with video, and then reassembling everything together and passing it through x264 with presets to use.

You have yet to explain what is stopping you from using the presets in mencoder. It's a simple matter of reading the --fullhelp and using your copy and paste buttons.

Originally Posted by Digital Corpus

Granted mencoder will have to slew the audio or video to bring them back in sync, but it is simpler since I don't have to deal with that. As I "have" to use mplayer, the presets were not an option. Though at the same time, mbtree isn't an option that exists in mencoder but I can still pass it to the x264 code.

What do you mean, "option that exists in mencoder"? mencoder passes all options blindly to x264. It does not have its own option system. Any option that is in libx264 is in mencoder.

[Digital Corpus]

Originally Posted by Dark Shikari

You have yet to explain what is stopping you from using the presets in mencoder. It's a simple matter of reading the --fullhelp and using your copy and paste buttons.What do you mean, "option that exists in mencoder"? mencoder passes all options blindly to x264. It does not have its own option system. Any option that is in libx264 is in mencoder.

I found that out by fluke, and even though it does do that, it doesn't use the same syntax.

What i have pasted above, looks like this in mencoder:

nombtree:fast_pskip:nodct_decimate:keyint=$max_key:keyint_min=$min_key:mixed_refs:deblock=2,2:bframes=1:nob_adapt:b_pyramid=strict:cabac:qp_min=1:qp_max=51:qp_step=25:ratetol=35:qcomp=0.75:direct_pred=auto:partitions=i8x8,p8x8,b8x8:8x8dct:psy-rd=0.4,0:trellis=0

That is hardly a blind pass.

[Digital Corpus]

Originally Posted by Blue_MiSfit

And of course, do specify the rate control mode you're interested in. If you must have control over file size, 2 pass is the way to do it. If you can afford some unpredictibility, then CRF with some VBV isn't a terrible idea.

~MiSfit

I'll probably give this a shot. Will the size output be constrained so that it will not go beyond a certain size due to the VBV algorithms? I have no problem with going smaller and saving bits. I'm mostly worried about over allocation and going beyond what I can reasonably expect. Would I notice any speed difference between CRF and ABR?

[Blue_MiSfit]

Let's get this sorted. It doesn't need to be complicated or dramatic.

Originally Posted by Digital Corpus

I'll probably give this a shot. Will the size output be constrained so that it will not go beyond a certain size due to the VBV algorithms? I have no problem with going smaller and saving bits. I'm mostly worried about over allocation and going beyond what I can reasonably expect. Would I notice any speed difference between CRF and ABR?

VBV is more for enforcing peak bitrate limitations for hardware or streaming compatibility. in your case, it may not be necessary.

ABR and CRF are similar in terms of speed, though CRF will give vastly superior quality at the cost of unpredictable file sizes.

Now, as I understand it, you want to use mplayer to decode your sources so you can take advantage of its error fixing magic, and pipe the result into x264's CLI interface. That should work perfectly fine, and will enable you to "use the presets", as we've all been screaming at you to do

Do throw everything you know about settings out the window, and make another evaluation of quality:speed using only the presets Much changes, and the presets are very well chosen.

~MiSfit

[Digital Corpus]

@Asmodian
Thank you. I was hoping for something a bit more like that. I can understand the built up frustration that takes place by incessant postings like this, but that doesn't excuse the lack of civility.

I have deblocking turned up dues to the nature of MPEG-2 video from my local networks. What is broadcast in my area always has some minor blocking issues and depending on which network I'm recording from, it can actually be pretty bad. That seems to be my optimal range from the tests I've done and it doesn't affect quality more than the visible macroblocks already present.

I turned bframes off due to platform issues. I would like to maintain iPod/iPhone compatibility. I have my server setup with http access...

I'm trying to keep up. Thank you for a this for that take on various setitngs. I was hoping for information such as that to work from.


@Blue_MiSfit
Generally true, but I see VBV as a two fold option. As kinda hinted at, I have this setup for friends/coworkers that don't have their own DVR and would like to grab shows to watch. I'm mostly targeting the Xbox 360 and the iPhone/iPod Classic. Since this kind of requires web access, I thought I might as well make things streaming compatible if I could. That and if I can guarantee that a CRF encode won't surprise me with a large file size, then VBV will serve a purpose there. Unless you have a better suggestion.

If you can provide some help with me getting mplayer to pipe to x264, I'm all for it. My current attempts have failed, but there has to be a way.

Each revision I basically did that. If I have to do it again, I will. I have about 36 hours to play with this until I have to go back to work

[nm]

Originally Posted by Digital Corpus

I have deblocking turned up dues to the nature of MPEG-2 video from my local networks. What is broadcast in my area always has some minor blocking issues and depending on which network I'm recording from, it can actually be pretty bad. That seems to be my optimal range from the tests I've done and it doesn't affect quality more than the visible macroblocks already present.

Have you tried deblocking the source with MEncoder's filters? That should give better results.

Generally true, but I see VBV as a two fold option. As kinda hinted at, I have this setup for friends/coworkers that don't have their own DVR and would like to grab shows to watch. I'm mostly targeting the Xbox 360 and the iPhone/iPod Classic. Since this kind of requires web access, I thought I might as well make things streaming compatible if I could. That and if I can guarantee that a CRF encode won't surprise me with a large file size, then VBV will serve a purpose there. Unless you have a better suggestion.

You also need VBV for device compatibility, even when using ABR or 2-pass rate control.

If you can provide some help with me getting mplayer to pipe to x264, I'm all for it. My current attempts have failed, but there has to be a way.

There are many ways, but using yuv4mpeg and a FIFO is pretty easy on Linux:

mkfifo pipe.y4m
x264 --preset fast --crf 22 -o output.264 pipe.y4m &
mplayer -really-quiet -vo yuv4mpeg:file=pipe.y4m input.ts

Audio needs to be encoded separately and then muxed with the video to an appropriate container.


Or just copy the preset settings over to MEncoder as Dark Shikari suggested. Remember that the defaults (no settings) are the same as the medium preset, also in MEncoder, so you only need to add settings listed by x264 --fullhelp for each preset.

For example, to encode with the fast preset, put rc_lookahead=30:ref=2:subme=6 to x264encopts. The following command does the same thing as the piping procedure did above:

mencoder -ovc x264 -x264encopts crf=22:rc_lookahead=30:ref=2:subme=6 -nosound -of rawvideo -o output.264 input.ts

[Caroliano]

Originally Posted by Digital Corpus

@Asmodian: I have deblocking turned up dues to the nature of MPEG-2 video from my local networks. What is broadcast in my area always has some minor blocking issues and depending on which network I'm recording from, it can actually be pretty bad. That seems to be my optimal range from the tests I've done and it doesn't affect quality more than the visible macroblocks already present.

That is not the correct use for the in-loop deblocking filter of H.264. This deblocking filter is applied after the frame is encoded, and before the frame is used as reference, and is intended to fight only the new blocking created by the encoder.

The best you can do is to use and deblocking/denoising filter before the encoding, in order to feed a cleaner video to x264. And then lower a little your deblock, depending on your taste for bluriness vs noise.

[`Orum]

At the risk of possibly making things worse, I suggest you check out MeWiki. Just don't use MeGUI for encoding, but some of the info on the wiki is helpful.

[Digital Corpus]

That is the wiki I came across earlier today. Thank you though. I am changing my qp settings to something sane as a result. I have a background in mathematics though through multivariable Calculus and I'm a professional photographer. Already stated I'm by no means perfect, but the only way to make a good decision if I think I want to tweak something to see how it comes out, the more I need to know. cplxblur is the only other option I've considered changing.