Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
18th December 2018, 16:16 | #6522 | Link |
Registered User
Join Date: Oct 2007
Posts: 26
|
10 bit color for 480p ?
I've been ripping a bunch of 480p DVDs with x264. A lot of them are early technicolor.
I've been wondering whether 10 bit x265 would better preserve the color. (I don't have anything that would play 10 bit x264). But if I do that, what's the color range, still TV (limited) or full ? And if TV, what is the limited color range for 10 bit ? 16-235 goes to xxx - yyy ? And the colorprim stays smpte170m, correct ? Anybody tried this ? sean |
18th December 2018, 16:28 | #6523 | Link |
Life's clearer in 4K UHD
Join Date: Jun 2003
Location: Notts, UK
Posts: 12,227
|
Given the low cost of HDD's these days, why not back-up the original (8-bit) MPEG-2 stream from the DVD?!
No encoding required... Easy. Cheers
__________________
| I've been testing hardware media playback devices and software A/V encoders and decoders since 2001 | My Network Layout & A/V Gear |
|
18th December 2018, 16:43 | #6524 | Link |
German doom9/Gleitz SuMo
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,784
|
The 8 or 10 bit of the AVC or HEVC encoder "engine" is not related to the bit depth of the image, but to the resolution of encoder internal quantization parameters after a mathematical transformation. If you never learned about the Fourier transformation of discrete samples to a frequency spectrum, you may not understand much what that means... and despite a limited range, the Discrete Cosine Transform as well as the HEVC Integer Transform are still closely related to Fourier sequences (which are originally infinite).
MPEG-2 video may technically have 8..11 bit internal DCT precision, but in DVD Video media most commonly 9 bit, rarely 10 bit in high quality productions. Recoding will lose quality already by the requantization of the material. I can only agree to SeeModeDigital: The best way to preserve the quality in your DVD material is not to convert it anymore. Just remultiplex the main movie PGC to an MKV (makeMKV) for easy handling on a PC or smart TV. |
18th December 2018, 20:18 | #6528 | Link |
Useful n00b
Join Date: Jul 2014
Posts: 1,667
|
From the user manual:
"Note that if the custom rate conversion is selected, and if the source rate is specified as equal to the destination rate, then all pulldown is removed and the stream is flagged as having a rate equal to the specified destination rate." |
19th December 2018, 04:32 | #6529 | Link | ||||
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
|
Quote:
Quote:
Please note that the standard is Limited Tv Range. Besides, if the content was originally in Tv Range, you shouldn't change it. Quote:
For 10bit, Tv Range is 64-940. x265 has a flag to specify "limited range" which is --range limited (just like x264), and it also has two additional parameters --min-luma 64 --max-luma 940 to apply clipping if you have to be 100% sure that values don't exceed Tv Range (useful for TV Stations). If it's really soft telecine (just a flag specified while muxing the original MPEG-2 file), yes, you can, otherwise if it's really telecined with repeated fields, you gotta do reverse telecine and re-encode. Quote:
Luckily, these days are over and nowadays everything it's handled via software. |
||||
22nd December 2018, 19:35 | #6530 | Link |
Registered User
Join Date: Feb 2008
Posts: 145
|
Some advice on chunked encoding for a newb please!
I've been working out how to use the chunk commands to encode pieces of a file with an eye towards using some form of cluster to speed single file encoding. One thing that I'm observing is slow encode speed but it may be because of the VM I'm using despite 12 cores being made available. When chunk is used and the specified frames are deep into the file the encoder is able to skip to those frames yes, it need not start at the beginning and search for them? I'm not currently specifying key frames and am allowing chunk to find those on its own, I also realize it will encode some frames before and after the keyframes it chooses to start on but I'd like to be sure that in the end I've saved time! I'm encoding a test now with a csv file specified for statistics and it's got nothing in it after 5 mins, I was hoping to see some information as it went along in hopes of determining progress In some tests I've done things like specify color space, key intervals, and other settings, am I better off allowing the encoder to do most of that itself? My intuition says yes lol. I have attempted to use a cropping filter in ffmpeg feeding X265, would I be better off doing this in X265? I've had trouble figuring that out so far so an example would rock - especially if it uses the same syntax as ffmpeg. Would cropping in ffmpeg prevent a video from seeking to the chunk frame I desire or be slower than using X265? I'm just trying to ensure I'm doing this as efficiently as possible, honestly must ffmpeg be used at all if I'm not adding filters? I've not been able to use X265 standalone to accomplish what I want yet. I've also noticed that when I encode a chunk like this that VLC won't open the file with a MKV extension but if I use .265 (which I've associated with VLC) it opens, it's as if assumptions are made from the extension that prove invalid. When I've concatenated a number of chunks together and muxed in audio the result has been fine though. Here are two examples of commandline I'm working with right now: ffmpeg -hide_banner -i INPUT.mkv -f yuv4mpegpipe - | x265 - --no-open-gop --chunk-start 166675 --chunk-end 169500 --colorprim bt709 --transfer bt709 --colormatrix bt709 --crf=20 --fps 24000/1001 --min-keyint 24 --keyint 240 --sar 1:1 --preset slow --ctu 16 --y4m --pools + -o chunky-nocrop.265 ffmpeg -hide_banner -i INPUT.mkv -filter:v "crop=1920:800:0:140" -f yuv4mpegpipe - | x265 - --no-open-gop --chunk-start 166675 --chunk-end 169500 --crf=20 --fps 24000/1001 --preset slow --ctu 16 --y4m -o chunky1.mkv Any advice would be welcomed, I'm doing this in Linux FWIW. |
22nd December 2018, 20:59 | #6531 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,905
|
Encoding in a Virtual Machine is gonna be slower than encoding files on your physical hardware; it doesn't matter if you allow many cores and you have. Since you are doing this on Linux in a VM, unless you need a frameserver (like Avisynth) to do any sort of post-processing, drop the Windows Virtual Machine and do it on Linux via ffmpeg and x265 (they can both be compiled using GCC, CMake and NASM).
As to the player, MPV works perfectly well on Linux, you just need to install the additional codecs and you are good to go; no need to use VLC. For the records, I use Windows at work, but I have Fedora at home. |
22nd December 2018, 21:33 | #6532 | Link | |
Registered User
Join Date: Feb 2008
Posts: 145
|
Quote:
My concern is that I'm somehow forcing it to read through and do processing on the entire file before it reaches it's target range and does real work - that would make breaking jobs up pointless! End goal: break a large HD encoding task into 60 or more "jobs" and throw those jobs to encoding containers on various machines around the house that have idle cycles. Combine the resulting sections of video, mux in audio, have a complete video. Kubernetes or something like that to manage the containers. Have to have a proper working "job" before I can start the clustering part though hence my testing. I've done this already, not using chunk, by indexing a video, making jobs that started and ended on keyframes, and then combining etc. but "chunk" seems made for this and allows me to skip the indexing, mathing of start/end points, and sounds like it uses the encoder better. Hoping that makes more sense! P.S. Yes, I intend to share whatever I come up with. Every other project I've found that does anything similar hasn't been updated in ages other than RipBot which works great on Windows but I've got more Linux resources and desire cross-platform as a result. |
|
22nd December 2018, 21:57 | #6533 | Link | |
Registered User
Join Date: May 2005
Posts: 1,462
|
Quote:
I have done many (x264) encodings in a VM (ESXi). Surprisingly, it actually isn't all that much slower, as, while the CPU's are virtualized, of course, they're not like software-emulated or something, and, for the most part, are accessed almost directly. The speed drop was about 2-3% vs. the real deal. Mostly VM I/O was actually a bottleneck, for me. The beauty of VM encoding, is that you can suspend the entire VM, mid-encoding! And it will continue just fine where you left of, when you start it up again. I stopped doing it, though, as I love using GPU-assisted encoding, and figured, if I have to buy a second video-card, just to make it passthru to ESXi, then I'm better off just using my main card directly, outside Vmware.
__________________
Gorgeous, delicious, deculture! |
|
22nd December 2018, 22:04 | #6534 | Link |
Registered User
Join Date: Feb 2008
Posts: 145
|
Agree, I don't see ESX slowing things too much. I am presently shying from GPU encoding mostly because I'm under the impression it's not as high a quality and only one Windows machine I've got could do it. Learning abnout clustering is also an incentive I'll admit.
That said - I have two jobs running right now in CMD Windows in WIN10. One seems to be saying it's run through 94K frame, the other 115K - they've been running over an hour each. The chunk start and end were only 2800 frames apart! One is moving at 23FPS, the other 56FPS, neither datafile has any damn data in it yet. Am I doing something wrong? Surely chunk isn't meant to read from SOF and should jump to where it's been told to begin encoding right? I really don't want to use --seek and --frames to do this Edit: Rereading the little I can find on chunk. Surely they don't mean ALL frames preceeding do they?! I had read this as a function to help multiple machines work on a single file but it's surely not that if ALL preceeding frames are encoded and discarded as you get into a file option:`--chunk-start and --chunk-end` Frames preceding first frame of chunk in display order will be encoded, however, they will be discarded in the bitstream. Frames following last frame of the chunk in display order will be used in taking lookahead decisions, but, they will not be encoded. This feature can be enabled only in closed GOP structures. Default disabled. Last edited by BLKMGK; 22nd December 2018 at 22:11. |
22nd December 2018, 22:30 | #6535 | Link |
Registered User
Join Date: Feb 2008
Posts: 145
|
Just answered part of my own question "chunk" reads and encodes from start of file all the way to it's given starting point to create it's "chunk". It then reads to the given end point and completes. I've seen this mentioned as something that could be used for clustering jobs but not with this behavior it isn't! Is this intended? Why does it not advance to it's given start frame, look backwards to a previous keyframe, and start encoding there? It could discard the frames prior to it's given start but have given the encoder frames to prime it. If this is intended behavior I don't understand the usefulness.
Edit: Just tried an end point for the chunk that wasn't right against EOF. When it hits its endpoint it keeps encoding but discards the work. Last edited by BLKMGK; 22nd December 2018 at 23:03. |
22nd December 2018, 22:41 | #6536 | Link | ||
Registered User
Join Date: May 2005
Posts: 1,462
|
Quote:
I was more talking about OpenCL in x264; and especially used for the KNLMeansCL denoiser (QTGMC), where it speeds things up significantly. Even though I recently heard even OpenCL yields a theoretical lower quality. Quote:
As for chunks, I never used them before, so I can't really say something useful about that.
__________________
Gorgeous, delicious, deculture! |
||
22nd December 2018, 23:37 | #6537 | Link |
Registered User
Join Date: Feb 2015
Posts: 326
|
If you want to encode without VBV buffer, you could use --seek and --frames options, for example:
for %i in (0 1000 2000) do ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek %i --frames 1000 c%i.hevc which expands to: ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 0 --frames 1000 c0.hevc ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 1000 --frames 1000 c1000.hevc ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --seek 2000 --frames 1000 c2000.hevc Finally you can put these 3 parts together: mkvmerge -o c-whole.mkv c0.hevc + c1000.hevc + c2000.hevc ----------------- example for Windows |
23rd December 2018, 00:10 | #6538 | Link | ||
Registered User
Join Date: Feb 2008
Posts: 145
|
Quote:
This definitely wasn't a refresh issue in explorer, Chunk really doesn't appear to save any data until it hits its start frame and continues encoding after it hits its end frame. Ugh! Quote:
I then noticed chunk in the release notes. Chunk, from its description and some discussion I'd seen here, seemed tailor made for clutering but my tests sure don't seem to prove it out. I'll try your example and see if I can detect the seams, perhaps I've been too paranoid? Thank you for the example of how to build the jobs! Edit: Tested this method ignoring keyframes and thought I detected a hitch in the video. Sure enough VLC shows two dropped frames where the hitch occurs and I see it elsewhere too. I'll keep testing and see if the keyframes are the issue. I'll pull an index and create some jobs. I need an automated way to build jobs from the index but not at every break lol Last edited by BLKMGK; 23rd December 2018 at 00:37. |
||
23rd December 2018, 01:29 | #6539 | Link |
Registered User
Join Date: Feb 2015
Posts: 326
|
I tried to add --chunk-start/--chunk-end to the example and it is like this:
Encoding in parts of 1000 frames (+100 before, +100 after): ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 0 --frames 1100 --chunk-start 1 --chunk-end 1000 c0.hevc ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 900 --frames 1200 --chunk-start 101 --chunk-end 1100 c1.hevc ffmpeg -i ../lighthouse_lossless.mp4 -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 1900 --frames 1200 --chunk-start 101 --chunk-end 1100 c2.hevc mkvmerge -o chunks.mkv c0.hevc + c1.hevc + c2.hevc |
23rd December 2018, 05:00 | #6540 | Link | |
Registered User
Join Date: Feb 2008
Posts: 145
|
Quote:
I tried your example on my sample file, I'm not spotting glitches or dropped frames in the statistics and I managed to extend it out some. ffmpeg -i INPUT.MKV -v warning -f yuv4mpegpipe - | x265 --y4m - --crf 20 --no-open-gop --seek 3100 --frames 1200 --chunk-start 101 --chunk-end 1100 --csv stats.csv c3.hevc If I'm wrong about the keyframes that will make life much easier. I'll try doing an entire film once I've incorporated this into my bash script, mux in sound, and check the synch. Thank you! I really appreciate you providing examples too as it makes understanding what you're doing much simpler. |
|
|
|