Help, x264 crf18 10bit, encoded file 5GB larger?

[xekon]

I have been encoding some bluray content, 1080p, Ive been using the same x264 settings for all content, most of the time my files are 50-80% smaller. but I have had a couple where the file size goes up 2-4 GB, I understand some content doesn't compress as well as others, I am just having a hard time understanding why it would take up more space than the original, especially with 10bit being so good at compressing.

I can see the bitrate goes up from 25.9 Mbps to 34.4 Mbps, I'm guessing its because its grainy old sources, but why larger than the original?

here is my x264 encode

x264_10 --preset veryslow --crf 18 --ref 5 --bframes 16 --no-fast-pskip --qcomp 0.8 --output "jobz.mkv" "E:\AviSynth\bd.avs"

bd.avs:

LoadPlugin("E:\AviSynth\plugins\ffms2.dll")
A = FFAudioSource("E:\AviSynth\Turtles_II.mkv")
V = FFVideoSource("E:\AviSynth\Turtles_II.mkv")
AudioDub(V, A)
crop(0,20,0,-20)

source mediainfo:

File size : 16.0 GiB 
Duration : 1h 28mn 
Overall bit rate : 25.9 Mbps 
Video 
ID : 1 
Format : VC-1 
Format profile : Advanced@L3 
Codec ID : V_MS/VFW/FOURCC / WVC1 
Codec ID/Hint : Microsoft 
Duration : 1h 28mn 
Bit rate : 24.7 Mbps 
Width : 1 920 pixels 
Height : 1 080 pixels 
Display aspect ratio : 16:9 
Frame rate mode : Constant 
Frame rate : 23.976 fps 
Color space : YUV 
Chroma subsampling : 4:2:0 
Bit depth : 8 bits 
Scan type : Progressive 
Compression mode : Lossy 
Bits/(Pixel*Frame) : 0.497 
Stream size : 15.3 GiB (96%) 
Language : English 
Default : No 
Forced : No

output mediainfo:

File size : 21.3 GiB 
Duration : 1h 28mn 
Overall bit rate : 34.4 Mbps 
Video 
ID : 1 
Format : AVC 
Format/Info : Advanced Video Codec 
Format profile : High 10@L5.0 
Format settings, CABAC : Yes 
Format settings, ReFrames : 5 frames 
Codec ID : V_MPEG4/ISO/AVC 
Duration : 1h 28mn 
Width : 1 920 pixels 
Height : 1 040 pixels 
Display aspect ratio : 1.85:1 
Frame rate mode : Constant 
Frame rate : 23.976 fps 
Color space : YUV 
Chroma subsampling : 4:2:0 
Bit depth : 10 bits 
Scan type : Progressive 
Writing library : x264 core 129 r2245 bc13772 
Encoding settings : cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.80 / qpmin=0 / qpmax=81 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00 
Language : English 
Default : Yes 
Forced : No

[Asmodian]

Understand that a bluray is already heavily compressed. Your source uncompressed would be ~366.7 GB and this is the information x264 starts with. The first step after getting the file off the disc is to decompress it.

[xekon]

Yes, I understood that. but you can never get More quality out of something than what you started with right?

I am trying to figure out if I am wasting file space somehow with my settings? Does my additional bitrate help quality in any way over the original bluray?

[Guest]

Originally Posted by xekon

Yes, I understood that. but you can never get More quality out of something than what you started with right?

Nobody said you have more quality. You only have more bitrate. There will be the decoded content and encoding artifacts from the original encode in the uncompressed intermediate. In your output encode you are spending more bits to better represent the content and these artifacts. But you haven't improved the quality of the original content because you cannot recover any information that is not present in the original.

[xekon]

thanks for the responses guys, okay so I am guessing that with older grainy sources you get more artifacts which leads to a larger than source file size at crf 18? I am trying to keep my collection transparent, I did some testing a year ago with a few different titles, playing it back, looking at differences etc, and at crf 18 I could not notice a difference between original and encoded.

Like I said most of the time encoding results in a smaller file size, but sometimes it don't. 5GB increase in file size does seem excessive to me though. I am wondering if I should set some sort of limit on bitrate for the few movies that want to exceed the original file size.

What would you do or suggest?

[Asmodian]

I would strongly suggest keeping the original video in these cases, a larger lower quality file doesn't sound good to me. Do you need to re-encode for some reason?

[Sparktank]

Originally Posted by xekon

I am wondering if I should set some sort of limit on bitrate for the few movies that want to exceed the original file size.

What would you do or suggest?

It really depends on the source. Every movie will be different (as exhibited through this scenario).

For de facto encodes that exceed the original file size where you would want to re-encode with different settings, you can try try create (maximum of) 5 minute video samples and do test encodes with different settings on that to (very loosely) gauge an idea of what you're looking at.

SelectRangeEvery() is handy select very short clips (linearly) through the video to create a sample.
I add ".Trim(0,7192)" (NSTC-based fps) to trim the SRE clip to 5 minutes (in case it exceeds 5 minutes and I don't want to test a 9 minute clip or so.

SelectRangeEvery(every=1500, length=71, offset=0, audio=true).Trim(0,7192)

(You use video with audio, you'll have to add it after AudioDub)

Encode that with CRF=18 then do another encode using something CRF=20 or even try 2-pass bitrates to set your maximum file size. You could also try using some of the tune settings: try it with --tune film or even --tune grain. (diong all of them wouldn't hurt to see which one gives a (relatively) small size without degrading picture quality as much. I'm finding 19/20 to do reasonably well without much destruction to the video).

If the film is really grainy, you can try the --fgo (film grain optimization) settings in x264.
Though, you'd better research that more to see how it works. Can't say I've ever tried it myself.

You're not really going to be able to tell which movies are giong to exceed the source file size.
If you watch the encode status, the projected stats (eta, size, etc) are all just very, very, very vague interpretations for what has been processed. It fluctuates constantly. It only gets accurate the longer you run it (ie, going until the end of file, you get thee most accurate results).
You can do a 30 minute test encode (using that "SelectEveryRange().Trim(0,43156)" ... 43156 frames for 30 minutes, based on NTSC values) with the different settings to see what comes out better (provided you keep your data for the full encode with CRF18 and the 30 minute encode of CRF18 with exact same settings as the full encode; so you can compare the CRF18@30min with CRF20@30min/2pass@30/CRF20+tune=film@30min/etc/etc/etc... a long winded project to keep your file size down).

It's not an exact science since the video itself provides too many variables.
Which requires a controlled testing environment.

For science!

[Sparktank]

Originally Posted by Asmodian

I would strongly suggest keeping the original video in these cases, a larger lower quality file doesn't sound good to me. Do you need to re-encode for some reason?

That's my second suggestion, if you don't want to do the mega project.

It's likely that you won't be playing these on anything but the PC since most (if not all) stand-alone media players don't support 10bit, and most(if not all) DLNA servers don't support 10bit either.

You won't get any more transparent than the source and it's already a smaller file size (compared to your x264 encode).

[F J Walter]

Originally Posted by xekon

5GB increase in file size does seem excessive to me though. I am wondering if I should set some sort of limit on bitrate for the few movies that want to exceed the original file size.

What would you do or suggest?

If you find that your re-encode needs to approach or exceed the original in terms of file size for you to be satisfied with the quality, it is better to just keep the original.

When compressing blu rays with x264, you are going from h264 to h264 - ie you are transcoding to the *same* codec. All that is going to happen is you lose quality. The only reason you would want to transcode from a codec to the same codec is to save bitrate, so if you're not saving any bitrate with your chosen quality settings, then don't transcode at all. Remember, the original is 100% faithful to the original.

The standard thing to suggest is to pick a lowish CRF (resulting in high bitrate) then keep incrementing the CRF until you can no longer live with the amount of quality loss. The CRF you used just before that is the best CRF to use. Remember that sitting closer to the screen than you intend to upon later viewing, or freeze-framing, will lead to you selecting a higher quality than you really need - although it's always a good idea to have a little slack.

If you find that the encodes are still coming out almost as big, or bigger than, the original, then just keep the original without transcoding. Re-mux if you have to.

[F J Walter]

why larger than the original?

Every transcoding step introduces some small error/distortion in the video. Transcode an already compressed video and you're transcoding not from the original but from the original + some distortion.

When you re-encode an already-compressed video with high enough quality, you'll start spending additional bandwidth just to faithfully re-encode the error/distortion (eg blocking, banding, etc) from the previous encode.

[detmek]

Denoise or degrain might help. Or QTGMC in progresive mode to stabilaze grain, depending on source.