x265 HEVC Encoder

[MeteorRain]

So do I, although I rarely post in this thread.

[seandarcy]

I've been ripping a bunch of 480p DVDs with x264. A lot of them are early technicolor.

I've been wondering whether 10 bit x265 would better preserve the color. (I don't have anything that would play 10 bit x264). But if I do that, what's the color range, still TV (limited) or full ? And if TV, what is the limited color range for 10 bit ? 16-235 goes to xxx - yyy ?

And the colorprim stays smpte170m, correct ?

Anybody tried this ?

sean

[SeeMoreDigital]

Given the low cost of HDD's these days, why not back-up the original (8-bit) MPEG-2 stream from the DVD?!

No encoding required... Easy.


Cheers

[LigH]

The 8 or 10 bit of the AVC or HEVC encoder "engine" is not related to the bit depth of the image, but to the resolution of encoder internal quantization parameters after a mathematical transformation. If you never learned about the Fourier transformation of discrete samples to a frequency spectrum, you may not understand much what that means... and despite a limited range, the Discrete Cosine Transform as well as the HEVC Integer Transform are still closely related to Fourier sequences (which are originally infinite).

MPEG-2 video may technically have 8..11 bit internal DCT precision, but in DVD Video media most commonly 9 bit, rarely 10 bit in high quality productions. Recoding will lose quality already by the requantization of the material. I can only agree to SeeModeDigital: The best way to preserve the quality in your DVD material is not to convert it anymore. Just remultiplex the main movie PGC to an MKV (makeMKV) for easy handling on a PC or smart TV.

[seandarcy]

OK, I'll remux to mkv. But what about the soft telecine ? This is a 24fps film soft telecined to 29.97. Can I undo the soft telecine without reencoding ?

sean

[videoh]

You can undo the soft telecine with DGPulldown.

[seandarcy]

Unless I'm misreading this, DGPulldown telecines a stream. It doesn't undo the telecine.

[videoh]

From the user manual:

"Note that if the custom rate conversion is selected, and if the source rate is specified as equal to the destination rate, then all pulldown is removed and the stream is flagged as having a rate equal to the specified destination rate."

[FranceBB]

Originally Posted by seandarcy

I've been wondering whether 10 bit x265 would better preserve the color. (I don't have anything that would play 10 bit x264).

x265 uses high bit depth internally and 10bit files are compatible with the majority of players; in other words, even if you feed it with an 8bit stream and you choose Main10, it will use high bit depth internally and encode it to 10bit.

Originally Posted by seandarcy

But if I do that, what's the color range, still TV (limited) or full ?

You can encode it in both Full Range or Limited Range, depending on your target.
Please note that the standard is Limited Tv Range.
Besides, if the content was originally in Tv Range, you shouldn't change it.

Originally Posted by seandarcy

And if TV, what is the limited color range for 10 bit ? 16-235 goes to xxx - yyy ?

The old 8bit Tv Range was 0.0-0.7V, in other words, 16-235.
For 10bit, Tv Range is 64-940.
x265 has a flag to specify "limited range" which is --range limited (just like x264), and it also has two additional parameters --min-luma 64 --max-luma 940 to apply clipping if you have to be 100% sure that values don't exceed Tv Range (useful for TV Stations).

Originally Posted by seandarcy

Can I undo the soft telecine without reencoding ?

If it's really soft telecine (just a flag specified while muxing the original MPEG-2 file), yes, you can, otherwise if it's really telecined with repeated fields, you gotta do reverse telecine and re-encode.

Originally Posted by LigH

MPEG-2 video may technically have 8..11 bit internal DCT precision, but in DVD Video media most commonly 9 bit, rarely 10 bit in high quality productions.

I used to make official DVDs time ago and you are correct, but I've seen some atrocities done in the past, including a playback 1:1 from a Sony VTR playing a BetaCAM "master", or playback using an Omneon SD playback port connected via SDI, or "recently" an Omneon FULL HD playback port routed via SDI to an hardware downscaler which performed FastBilinear and outputted the result to the Master DVD copy, when all this could have been done better and faster via software... but hey, stone-age procedures and stubborn co-workers...
Luckily, these days are over and nowadays everything it's handled via software.

[BLKMGK]

Some advice on chunked encoding for a newb please!

I've been working out how to use the chunk commands to encode pieces of a file with an eye towards using some form of cluster to speed single file encoding. One thing that I'm observing is slow encode speed but it may be because of the VM I'm using despite 12 cores being made available. When chunk is used and the specified frames are deep into the file the encoder is able to skip to those frames yes, it need not start at the beginning and search for them? I'm not currently specifying key frames and am allowing chunk to find those on its own, I also realize it will encode some frames before and after the keyframes it chooses to start on but I'd like to be sure that in the end I've saved time! I'm encoding a test now with a csv file specified for statistics and it's got nothing in it after 5 mins, I was hoping to see some information as it went along in hopes of determining progress

In some tests I've done things like specify color space, key intervals, and other settings, am I better off allowing the encoder to do most of that itself? My intuition says yes lol.

I have attempted to use a cropping filter in ffmpeg feeding X265, would I be better off doing this in X265? I've had trouble figuring that out so far so an example would rock - especially if it uses the same syntax as ffmpeg. Would cropping in ffmpeg prevent a video from seeking to the chunk frame I desire or be slower than using X265? I'm just trying to ensure I'm doing this as efficiently as possible, honestly must ffmpeg be used at all if I'm not adding filters? I've not been able to use X265 standalone to accomplish what I want yet.

I've also noticed that when I encode a chunk like this that VLC won't open the file with a MKV extension but if I use .265 (which I've associated with VLC) it opens, it's as if assumptions are made from the extension that prove invalid. When I've concatenated a number of chunks together and muxed in audio the result has been fine though.

Here are two examples of commandline I'm working with right now:

ffmpeg -hide_banner -i INPUT.mkv -f yuv4mpegpipe - | x265 - --no-open-gop --chunk-start 166675 --chunk-end 169500 --colorprim bt709 --transfer bt709 --colormatrix bt709 --crf=20 --fps 24000/1001 --min-keyint 24 --keyint 240 --sar 1:1 --preset slow --ctu 16 --y4m --pools + -o chunky-nocrop.265

ffmpeg -hide_banner -i INPUT.mkv -filter:v "crop=1920:800:0:140" -f yuv4mpegpipe - | x265 - --no-open-gop --chunk-start 166675 --chunk-end 169500 --crf=20 --fps 24000/1001 --preset slow --ctu 16 --y4m -o chunky1.mkv

Any advice would be welcomed, I'm doing this in Linux FWIW.

[FranceBB]

Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC.

For the records, I use Windows at work, but I have Fedora at home.

[BLKMGK]

Originally Posted by FranceBB

Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC.

For the records, I use Windows at work, but I have Fedora at home.

Right now it's a Linux VM on ESX for testing, I'm able to run the same command in Windows however as it's crossplatform (my intent). What seems odd to me is the destination file remains empty for an extended length of time but in the end has data. I've shortened up the gap between the chunks to 2875 frames to speed things and am dropping encoding quality to try and speed further but it still takes awhile.

My concern is that I'm somehow forcing it to read through and do processing on the entire file before it reaches it's target range and does real work - that would make breaking jobs up pointless!

End goal: break a large HD encoding task into 60 or more "jobs" and throw those jobs to encoding containers on various machines around the house that have idle cycles. Combine the resulting sections of video, mux in audio, have a complete video. Kubernetes or something like that to manage the containers. Have to have a proper working "job" before I can start the clustering part though hence my testing.

I've done this already, not using chunk, by indexing a video, making jobs that started and ended on keyframes, and then combining etc. but "chunk" seems made for this and allows me to skip the indexing, mathing of start/end points, and sounds like it uses the encoder better.

Hoping that makes more sense!

P.S. Yes, I intend to share whatever I come up with. Every other project I've found that does anything similar hasn't been updated in ages other than RipBot which works great on Windows but I've got more Linux resources and desire cross-platform as a result.

[asarian]

Originally Posted by FranceBB

Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC.

For the records, I use Windows at work, but I have Fedora at home.

I have done many (x264) encodings in a VM (ESXi). Surprisingly, it actually isn't all that much slower, as, while the CPU's are virtualized, of course, they're not like software-emulated or something, and, for the most part, are accessed almost directly. The speed drop was about 2-3% vs. the real deal. Mostly VM I/O was actually a bottleneck, for me.

The beauty of VM encoding, is that you can suspend the entire VM, mid-encoding! And it will continue just fine where you left of, when you start it up again. I stopped doing it, though, as I love using GPU-assisted encoding, and figured, if I have to buy a second video-card, just to make it passthru to ESXi, then I'm better off just using my main card directly, outside Vmware.

[BLKMGK]

Agree, I don't see ESX slowing things too much. I am presently shying from GPU encoding mostly because I'm under the impression it's not as high a quality and only one Windows machine I've got could do it. Learning abnout clustering is also an incentive I'll admit.

That said - I have two jobs running right now in CMD Windows in WIN10. One seems to be saying it's run through 94K frame, the other 115K - they've been running over an hour each. The chunk start and end were only 2800 frames apart! One is moving at 23FPS, the other 56FPS, neither datafile has any damn data in it yet.

Am I doing something wrong? Surely chunk isn't meant to read from SOF and should jump to where it's been told to begin encoding right? I really don't want to use --seek and --frames to do this

Edit: Rereading the little I can find on chunk. Surely they don't mean ALL frames preceeding do they?! I had read this as a function to help multiple machines work on a single file but it's surely not that if ALL preceeding frames are encoded and discarded as you get into a file

option:`--chunk-start and --chunk-end` Frames preceding first frame of chunk in display order will be encoded, however, they will be discarded in the bitstream. Frames following last frame of the chunk in display order will be used in taking lookahead decisions, but, they will not be encoded. This feature can be enabled only in closed GOP structures. Default disabled.

[BLKMGK]

Just answered part of my own question "chunk" reads and encodes from start of file all the way to it's given starting point to create it's "chunk". It then reads to the given end point and completes. I've seen this mentioned as something that could be used for clustering jobs but not with this behavior it isn't! Is this intended? Why does it not advance to it's given start frame, look backwards to a previous keyframe, and start encoding there? It could discard the frames prior to it's given start but have given the encoder frames to prime it. If this is intended behavior I don't understand the usefulness.

Edit: Just tried an end point for the chunk that wasn't right against EOF. When it hits its endpoint it keeps encoding but discards the work.

[asarian]

Originally Posted by BLKMGK

Agree, I don't see ESX slowing things too much. I am presently shying from GPU encoding mostly because I'm under the impression it's not as high a quality and only one Windows machine I've got could do it.

Coding HEVC directly with (hardware-accelerated) NVENC, for instance, is an order of magnitude faster (and sometimes several orders of magnitude even) than what we can achieve with x265 itself, but you have no control over the process, and the quality is definitely lower than that of the latter.

I was more talking about OpenCL in x264; and especially used for the KNLMeansCL denoiser (QTGMC), where it speeds things up significantly. Even though I recently heard even OpenCL yields a theoretical lower quality.

That said - I have two jobs running right now in CMD Windows in WIN10. One seems to be saying it's run through 94K frame, the other 115K - they've been running over an hour each. The chunk start and end were only 2800 frames apart! One is moving at 23FPS, the other 56FPS, neither datafile has any damn data in it yet.

As for the latter, typically you have to refresh your explorer window (a few times) to see how much is actually output already, or it will stay at 0 bytes.

As for chunks, I never used them before, so I can't really say something useful about that.

[Ma]

If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc

which expands to:
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc

Finally you can put these 3 parts together:
mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc
-----------------
example for Windows

[BLKMGK]

Originally Posted by asarian

Coding HEVC directly with (hardware-accelerated) NVENC, for instance, is an order of magnitude faster (and sometimes several orders of magnitude even) than what we can achieve with x265 itself, but you have no control over the process, and the quality is definitely lower than that of the latter.

I was more talking about OpenCL in x264; and especially used for the KNLMeansCL denoiser (QTGMC), where it speeds things up significantly. Even though I recently heard even OpenCL yields a theoretical lower quality.



As for the latter, typically you have to refresh your explorer window (a few times) to see how much is actually output already, or it will stay at 0 bytes.

As for chunks, I never used them before, so I can't really say something useful about that.

Ah, makes sense to use them for programmatic speedups and for filters I agree. I may try using my GPU a few times just to see if the results are acceptable to me but so far haven't been able to get it working the few times I've tried

This definitely wasn't a refresh issue in explorer, Chunk really doesn't appear to save any data until it hits its start frame and continues encoding after it hits its end frame. Ugh!

Originally Posted by Ma

If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc

which expands to:
ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc

Finally you can put these 3 parts together:
mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc
-----------------
example for Windows

I've done this except I started and ended on keyframes with the understanding that not doing so could have poor joins. The issue with that is it requires first pulling an index and then skipping through it to build the jobs. The first test attempted I built the latter part by hand (PITA) and after encoding I couldn't detect the seams at all. Seemed a good path forward!

I then noticed chunk in the release notes. Chunk, from its description and some discussion I'd seen here, seemed tailor made for clutering but my tests sure don't seem to prove it out. I'll try your example and see if I can detect the seams, perhaps I've been too paranoid?

Thank you for the example of how to build the jobs!

Edit: Tested this method ignoring keyframes and thought I detected a hitch in the video. Sure enough VLC shows two dropped frames where the hitch occurs and I see it elsewhere too. I'll keep testing and see if the keyframes are the issue. I'll pull an index and create some jobs. I need an automated way to build jobs from the index but not at every break lol

[Ma]

I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after):

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc

mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc

[BLKMGK]

Originally Posted by Ma

I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after):

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc

ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc

mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc

That looks like a very good idea to get the encoder to take into account frames before and after the specified range in order for the encoder to be more efficient! It's not breaking on keyframes, am I wrong about their importance?

I tried your example on my sample file, I'm not spotting glitches or dropped frames in the statistics and I managed to extend it out some.

ffmpeg -i INPUT.MKV -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 3100 --frames 1200 --chunk-start 101 --chunk-end 1100 --csv stats.csv c3.hevc

If I'm wrong about the keyframes that will make life much easier. I'll try doing an entire film once I've incorporated this into my bash script, mux in sound, and check the synch.

Thank you! I really appreciate you providing examples too as it makes understanding what you're doing much simpler.