Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
16th July 2019, 17:38 | #1 | Link |
Registered User
Join Date: Nov 2016
Posts: 11
|
Highest quality to file size ratio
I didn't see anything about this in the guides or FAQ and googling also hasn't turned up the specific information I'm looking for, so I'm asking here: How can I use H.265 to achieve the highest quality to file size ratio?
A year or two ago I found this post by Ma in the main HEVC thread. Based on that I've been using these Handbrake settings. 1080p MKV format, 1920x1080, no filters, 10-bit H.265, constant framerate same as source, very slow preset, no encoder tune, auto-profile, 2-pass, 1500 bitrate, Extra Options: rc-lookahead=120:bframes=12:ref=6:subme=7 If it is a dark video then: rc-lookahead=120:bframes=12:ref=6:subme=7:aq-mode=3 720p (only listing what's different here and for subsequent resolutions) 1280x720, 760 bitrate 480p 854x480, 360 bitrate 360p 640x360, 300 bitrate These are a starting point, and I've had to adjust the bitrate or try CRF instead, depending on the source files. But I've noticed that others have outdone me when it comes to the quality to file size ratio. It seems there should be a better set of options to start with, or I'm missing something in terms of what settings I should know to try to tweak in different scenarios. I've seen these types of threads before and contributed to them, and they usually have answers like "just find what's right for you", but I would like to at least hear what baseline settings others use and what are the prime candidates for settings to tweak for those that value quality to file size as I do. Last edited by Ischemia24; 17th July 2019 at 13:18. |
16th July 2019, 19:54 | #2 | Link |
Registered User
Join Date: Dec 2008
Posts: 589
|
My advice would be to seriously analyze how much time disk space is really worth to you.
We're at the point where you can buy a 8 TB (~ 7250 GiB in real disk space) for $150 ... that's 0.02$ per GB. At 3 mbps, you're looking at ~ 22 MB per minute or 1.4 GB per hour ... If you're gonna spend maybe 2-3 hours per one hour of encoding you should at least use a higher bitrate, maybe something like 8-10mbps VBR for 1080p and 4-8 mbps VBR for 720p ... if you end up with around 4-6 GB for a 2h movie, you're still looking at less than 0.1$ in disk space cost. // also hmm.. you joined nov 2016 but today's your first post? // ps... also checked the other forum post you mentioned.. hope you realized he used a cartoon for tests... real motion video, bluray with grain etc will behave differently.. make sure you understand why he uses some settings there. |
16th July 2019, 22:48 | #3 | Link | |||
Registered User
Join Date: Nov 2016
Posts: 11
|
Quote:
Quote:
Quote:
I think the Big Buck Bunny movie sample he used is something that you would tune for Film if using H.264 rather than Animation. But I admit I don't know why he went with the particular settings he chose. It was just presented as "true placebo" mode for H.265 and I seized on it. |
|||
17th July 2019, 00:08 | #4 | Link |
Registered User
Join Date: May 2009
Posts: 331
|
Time to change your media server to something larger. Sorry, but that is the reality. Disk space is cheap, and it will actually cost you more in power to convert your movies, than to just store them. IE the amount it will cost in power to use your computer to "save space" by encoding, is more expensive than storing the original on a hard drive and just buying new drives when you run out of space. Especially if you care about quality in your encodes.
|
17th July 2019, 00:12 | #5 | Link |
Moderator
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,770
|
Turning the nebulous concept of "quality" into a linear value that a ratio can be calculated from is a massive unsolved problem in computer vision and human visual perception. Any scale you could come up with would be specific to a given scenario.
|
17th July 2019, 08:10 | #6 | Link |
Registered User
Join Date: Nov 2016
Posts: 11
|
I don't feel like the substance of my question is being addressed. I'm not looking to re-encode my entire library. Just a few things, and most new titles I add. I get that quality is difficult to quantify in the context of video encoding, but we do have methods for it, however imperfect.
Any input on encoding settings? |
17th July 2019, 09:12 | #7 | Link | |
Lost my old account :(
Join Date: Jul 2017
Posts: 325
|
Quote:
And for the "Highest quality to file size ratio" part, this is exactly what crf is good for, just start with a baseline crf of 18 or something and increase it until you hit your sweetspot between visual fidelity and size (cause only you can decide at what point you are pleased with quality), at that point you will have found the "best" "quality to fie size ratio". This will also effect your encoding settings, are you looking at maxing out compression with low bitrates and will accept a lot of quality degradation, or should it be close to visually transparent? Cause for example, --no-sao and --deblock -1:-1 is great for the later case, but no that great for the first one. Last edited by excellentswordfight; 17th July 2019 at 12:09. |
|
17th July 2019, 12:35 | #8 | Link | |
Registered User
Join Date: Nov 2016
Posts: 11
|
Quote:
I appreciate the input regarding no-sao and deblock -1:-1, thank you. I can't see the encoding settings PSArips use, but -- actually, I can see the settings on the Dollhouse episodes. Here's one: Format : HEVC Format/Info : High Efficiency Video Coding Format profile : Main 10@L4@High Codec ID : V_MPEGH/ISO/HEVC Duration : 50 min 17 s Bit rate : 1 200 kb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 23.976 (24000/1001) FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 10 bits Bits/(Pixel*Frame) : 0.024 Stream size : 432 MiB (81%) Writing library : x265 2.8+19-bcdc610cf5f0:[Windows][MSVC 1914][64 bit] 10bit Encoding settings : cpuid=1111039 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=72337 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=8 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=20 / lookahead-slices=6 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=1 / dynamic-rd=4.00 / ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=3 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / weightp / weightb / analyze-src-pics / deblock=0:0 / sao / sao-non-deblock / rd=3 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=0.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=abr / bitrate=1200 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=2 / cplxblur=20.0 / qblur=0.5 / vbv-maxrate=12000 / vbv-bufsize=24000 / vbv-init=0.9 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=10 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei Default : Yes Forced : No I don't know what to make of this because they've backed off some of the settings I specify, and I don't know what else they may have specified and what values are defaults. High-tier=1 seems significant, but I'm not seeing anything in the documentation about what benefits that could provide. I'm guessing it means poorer compatibility with decoders, but I'm not concerned about that. |
|
17th July 2019, 13:08 | #9 | Link |
Herr
Join Date: Apr 2009
Location: North Europe
Posts: 556
|
VMAF is a better video-quality metric than SSIM & PSNR,
https://streaminglearningcenter.com/...af-master.html |
17th July 2019, 13:08 | #10 | Link | |
Lost my old account :(
Join Date: Jul 2017
Posts: 325
|
Quote:
Also, the encode that you have sampled is by no means any special, it looks like something like this was used: --preset medium --profile main10 --bframes 8 --aq-mode 3 --sao-non-deblock --limit-sao --bitrate 1200 --vbv-maxrate 12000 --vbv-bufsize 24000. And afaik, main and high teir only effect allowed datarates @ given level, so it doesnt matter that much here, I would think that it is automatically set cause of --vbv-bufsize 24000. Last edited by excellentswordfight; 17th July 2019 at 13:40. |
|
17th July 2019, 14:26 | #11 | Link | |
Registered User
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
|
Quote:
Second, you can see all the settings they used. What's stopping you from trying their settings and experimenting? |
|
17th July 2019, 18:58 | #12 | Link | |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,733
|
Quote:
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
|
18th July 2019, 01:09 | #13 | Link | |||
Registered User
Join Date: Nov 2016
Posts: 11
|
Quote:
Quote:
Ugh, on further inspection, I do see they kind of "cheated". Filtered to be able to compress better -- I had no idea this was a thing. I see now in some release notes, "The SSIM measures the accuracy of the outputted encode verse the source. It does not reflect on the sharpness or awesomeness of the image, only how close it is to the source material. An "S##" is a score where filters where [sic] not used. "FS##" is an encode where we used filters before the encoder, usually for nasty amounts of grain, this can throw off the score usually in a positive way and we like to be clear and honest [...]". Huh. Looks like the encode settings I copied into this thread were from an episode that had been pre-filtered. Quote:
I have no idea how this pre-filtering for better compression works, but I'd rather just stick to encoding settings, if I can get Handbrake to utilize the CPU effectively. If pre-filtering has real merit, then I imagine it'll eventually become a part of mainstream encoding software. Oh, this is more than one. Psy-rd and Psy-rdoq. That makes experimentation more difficult. I would ask though, do you prefer to use one of these or both? If just one, which one? Or is there a case that merits one over the other, or both? |
|||
18th July 2019, 06:07 | #14 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,733
|
I would leave the psy options as default. They will cause a lower score in those artificial measurement methods, but that's their nature as they are meant to improve subjective quality when you watch the video.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
18th July 2019, 12:23 | #15 | Link | |
Registered User
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
|
Quote:
|
|
22nd July 2019, 19:26 | #17 | Link | |
Moderator
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,770
|
Quote:
And how to go from a number for every frame to some kind of title-level score isn't something well defined. A file that bounces from VMAF 50 to 90 is going to be more annoying to viewers than one that's a consistent 70, although both would have the same per-title score. The harmonic mean per GOP or as a 10 second rolling average might be good. I admit, I am not also encoding Dollhouse. I've just been using the same settings for a long time and I determined my subjective experience of the videos with the bitrates they had beyond what I could achieve. It's eye opening that the encode settings seem so pedestrian. Ugh, on further inspection, I do see they kind of "cheated". Filtered to be able to compress better -- I had no idea this was a thing. I see now in some release notes, "The SSIM measures the accuracy of the outputted encode verse the source. It does not reflect on the sharpness or awesomeness of the image, only how close it is to the source material. An "S##" is a score where filters where [sic] not used. "FS##" is an encode where we used filters before the encoder, usually for nasty amounts of grain, this can throw off the score usually in a positive way and we like to be clear and honest [...]". Huh. Looks like the encode settings I copied into this thread were from an episode that had been pre-filtered. Better in this case is subjective, and drawn on all of my past experiences with the encodes I've done. It seemed to me that more detail was retained at a lower bitrate than I've ever managed. The episode[s] looked superior to what I've been able to achieve at that bitrate. What can I say? I give up easily. I have no idea how this pre-filtering for better compression works, but I'd rather just stick to encoding settings, if I can get Handbrake to utilize the CPU effectively. If pre-filtering has real merit, then I imagine it'll eventually become a part of mainstream encoding software. Oh, this is more than one. Psy-rd and Psy-rdoq. That makes experimentation more difficult. I would ask though, do you prefer to use one of these or both? If just one, which one? Or is there a case that merits one over the other, or both? Last edited by benwaggoner; 22nd July 2019 at 19:26. Reason: Bad quote |
|
31st July 2019, 17:54 | #18 | Link |
Registered User
Join Date: Nov 2016
Posts: 11
|
So I have made a few changes to my baseline settings and I feel as though the quality retained at the restrained bitrates I'm using has really leveled up! Thank you Excellentswordfight for the suggestion of using no-sao and deblock -1:-1.
I came across a post claiming that H.264 changes to these settings when tuning for Film (which is what I need most of the time) deblock=-1:-1 psy-rd=1.0:0.15 So now my default settings look like this. rc-lookahead=120:bframes=12:ref=6:subme=7:deblock=-1:-1: psy-rd=1.0:0.15:no-sao I should do some additional testing to find out how I feel about the subjective quality of CRF at values that result in similar bitrates to what I am specifying. I recently had a problematic 480p encode that ended up looking great just changing to CRF 22 (the video bitrate went up ~40% though). I looked into VMAF and while I appreciate that it exists I decided I would rather subjectively evaluate my encodes. Are there any other settings I should consider applying to improve the quality to file size ratio? I understand that some settings could be adjusted for better encoding performance in different situations with no noticeable quality loss, and feel free to chime in about that if you'd like, but I would like to get the highest quality to size ratio possible within the scope of the H.265 settings. How would you approach dealing with a particularly grainy source? Any thoughts on the usage of the frame-threads setting? A higher number seems like it would increase CPU utilization, but my understanding is that it would go against my goals (and that if I want to be even more ridiculous, I should use frame-threads=1), so I decided I should leave it at the default. |
31st July 2019, 20:19 | #19 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,733
|
In a recent enough build, there's the new option --hme which is meant to improve motion estimation.
Personally I've also raised qcomp from 0.6 to 0.7 and set rd=6 and use --rd-refine.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
31st July 2019, 20:20 | #20 | Link |
Registered User
Join Date: Jun 2016
Posts: 116
|
First let me say, that I used to have a similar goal as you. I used to say the most quality per bit... I'd have encodes that took several days even on a 16c Xeon v4. Then I changed and figured a couple hundred megs more per file wasn't worth days of encoding.
But anyways... Increase your --bframes to 16 which is the max HEVC supports while saying in spec. You can also use --bframe-bias to increase the likely hood the encoder will use a b frame. I'd say a sane range would be 0-10 for bframe-bias. You might find --me sea or full will give a bit better compression and quality but with an increase in encoding time. If you should consider dropping --subme 7 to 5. You'll find --me sea is more likely to provide better quality and possibly a smaller file size. There is a relatively new setting for motion search --hme. It changes how MS is done. It is supposed to be better quality while being less computational intensive. I've found the settings below are about 7-9% slower while producing a file 1.5% smaller with about the quality as --me 3 --subme 5. --hme --hme-search 1,2,3 This was about 2.2% smaller but a 43% longer encoding time. I found it noticeably better quality - subjectively of course. --hme --hme-search 1,3,3 I would suggest trying lower values for --rc-lookahead 120, it is very high. I'm not sure your getting substantially better frame placement vs it set to 60. It'll improve your coding time with negligible difference in size. You could also set --tu-inter 4 and --tu-intra 4. Those will allow the encoder to look for better compression opportunities. FYI, I doubt handbrake will support the new HME settings. This was recently made available. You could try something like ripbot264 and download the latest x265 build from the HEVC thread on Doom9. Good luck. |
|
|