Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
7th March 2021, 10:28 | #1 | Link |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
I Have Published a HDR10 to HLG Converter
Version 1.0.1 is out. Get it HERE.
Hello everyone: The hour is late here in Idaho, but I have finally published something I've been working on for over a year now. It started when I decided to rip my Blu-ray library for personal use. This would allow me complete control over the copies and I could then freely put them on any of my devices. I ran into trouble when I went to consider how to handle my 4K discs. These use entirely different gamma and color from what my PC and laptop use. Effectively, I would have to rip both the 4K and standard Blu-ray discs to keep both my current SDR devices happy along with my future HDR ones. After thinking about this prospect of ripping both discs for 3/4 of a second, I decided to look into making these 4K discs viewable on existing devices. Enter HLG. What followed was studying piles of ITU-R printouts, experimenting, more studying, more experimenting, running around in some circles, and finally a (tentatively) complete product. Anyway, enough of all this rambling. I need to get to bed, so here you go: You can download current packages here: https://github.com/wswartzendruber/hlg-tools/releases You can check out the top-level README for the project here: https://github.com/wswartzendruber/h...main/README.md. It contains the conversion guide. I should probably disclose that I have not yet gotten my hands on a good HLG display, only a Samsung that left a lot to be desired. In fact, the SDR downconversion via MPV looked leaps and bounds better than that display's interesting interpretation of things. I did briefly get to evaluate a transcode on a much more expensive Samsung set, and that looked much better. In both cases, the TV recognized the signal as HLG. -- Things are looking much better on the set that looked bad. All right, it's 2:30 AM. I'm off to bed! EDIT: Hold on just a minute! I forgot to credit FranceBB with helping me here and there along the way. EDIT: And kolak, and SeeMoreDigital, and frank... Last edited by wswartzendruber; 12th May 2022 at 03:04. Reason: Giving credit where it's due, and, a more future-proof releases link. |
7th March 2021, 14:43 | #2 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
It's been a long way, but wswartzendruber you finally made it to the end.
I'm glad you were finally able to publish everything, it took a lot of effort and research on your side but I'm sure the results are now accurate, so well done. I'm sure this will help many people who want to convert their disks and other sources to HLG correctly, so well done for this and for making everything open source, that's the right way. |
9th March 2021, 15:12 | #3 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
Do you use any low pass filtering when calculating your max values?
Any encode will introduce overshoots and your max points can be very high, but somehow unreal due to compression. Also maybe ffmpeg scaling should use nearest neighbour method instead of default bicubic. |
9th March 2021, 16:52 | #4 | Link |
Life's clearer in 4K UHD
Join Date: Jun 2003
Location: Notts, UK
Posts: 12,227
|
This is why this forum could do with a 'thumbs up' button
__________________
| I've been testing hardware media playback devices and software A/V encoders and decoders since 2001 | My Network Layout & A/V Gear |
|
9th March 2021, 17:53 | #5 | Link | |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
Quote:
|
|
9th March 2021, 18:22 | #6 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
Seen so different results for HDR analysis mainly because compression overshoots. It all works fine as long as you stay with eg. 16bit TIFF, but once you touch ProRes etc. (not even talking about h265) then 1000nits max becomes eg. 2000, which can be just a single pixels overshoot in the whole movie. Very fake value.
|
9th March 2021, 18:45 | #7 | Link | |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
Quote:
|
|
9th March 2021, 21:27 | #8 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
No idea about specifics. I think some specs may mention it.
I know some tools do use it, eg. Cortex when analysing HDR, but I think values are own MTI Film calculated. Last edited by kolak; 9th March 2021 at 21:46. |
10th March 2021, 01:21 | #9 | Link | |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
Quote:
Question: Is it a safe assumption that the brightest pixel is always white? EDIT: I'm asking because pqstat just told me that Aquaman clocks in with a MaxCLL of 3,190, yet the metadata puts it at 1,506. I've never seen that kind of disparity before. Last edited by wswartzendruber; 10th March 2021 at 01:24. |
|
10th March 2021, 17:17 | #10 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
This is exactly what can happen with compression overshoots. You get very misleading results.
MaxCLL quite often come form the top of the production chain, where people work with EXRs or TIFF, so those should be more accurate than ones calculated form h265 compressed files. I would still blindly not rely on MaxCLL. There should be free EBU etc. doc about calculating MaxCLL. There is this: http://avisynth.nl/index.php/MaxCLLFind Some paper: None of it mentions low pass filtering, but this also (quietly I think ) assumes that measurements are done on uncompressed assets. Last edited by kolak; 10th March 2021 at 17:36. |
10th March 2021, 19:24 | #11 | Link |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
From that little snippet alone it would appear that I have been calculating MaxCLL incorrectly anyway. Is there any way I can get my hands on ST.2086?
EDIT: Part of the reason I want to calculate MaxCLL manually is that Alita: Battle Angel changes after the 20th Century Fox logo. It goes from 233 nits to 737 nits. Last edited by wswartzendruber; 10th March 2021 at 19:33. |
10th March 2021, 21:28 | #12 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
2nd page is for MaxFALL (in case this is confusing).
this ZIP has random info regarding HDR: https://spaces.hightail.com/space/nEaXy Last edited by kolak; 10th March 2021 at 21:30. |
11th March 2021, 01:32 | #13 | Link |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
So MaxCLL really is assumed to be white, and furthermore, no single color channel for any pixel can exceed what it would be to produce white at that brightness...
If that's how this really works, I'm going to completely redo tonemapping in pq2hlg. I won't have everyone waste their time with running the whole thing through pqstat. I'll just have them feed MaxCLL into pq2hlg instead. -- EDIT -- Where on GOD'S GREEN EARTH is MaxCLL formally defined?! It's not a part of BT.2100, BT.2408, or BT.2390. It's also not a part of ST.2086. Is it a part of the HDR10 specification? Because I can't find that, either! All I can find is that it (HDR10) is controlled by the CTA. What SMPTE publication is that above? Or is MaxCLL a part of that document dump? Last edited by wswartzendruber; 11th March 2021 at 06:22. Reason: Haplessly searching for MaxCLL's formal definition. |
11th March 2021, 11:50 | #14 | Link |
Life's clearer in 4K UHD
Join Date: Jun 2003
Location: Notts, UK
Posts: 12,227
|
Apart from HDR10, that particular 4K UHD disc has also been encoded with/contains HDR10+ and (MEL) Dolby Vision...
__________________
| I've been testing hardware media playback devices and software A/V encoders and decoders since 2001 | My Network Layout & A/V Gear |
Last edited by SeeMoreDigital; 11th March 2021 at 21:51. |
11th March 2021, 15:11 | #15 | Link | |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
Quote:
|
|
11th March 2021, 21:45 | #16 | Link |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
I found where the two SMPTE snippets above come from:
https://www.murideo.com/uploads/5/2/...-ecosystem.pdf This includes the algorithm for calculating MaxCLL. It demonstrates that MaxCLL is not necessarily the Y value of the brightest pixel in the stream as many sites suggest. Instead, MaxCLL tells the display what magnitude each color channel can reach, but all three don't have to reach this level, only one does. In practice, though, if the brightest pixel is truly white, then all color channels will reach this level. EDIT: Working on hlg-tools-0.2.0 now... Version 0.1.0 is not even remotely correct. Last edited by wswartzendruber; 12th March 2021 at 06:59. |
12th March 2021, 11:08 | #17 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Quote:
@wswartzendruber... so what Kolak means is that in the final encode you're gonna have things that are not there in the original source in which the measurements should have been done. I actually have never done a measurement on a final compressed encode for consumers, but I tried and indeed things are different. Take a look at this waveform coming directly from a synthetic 16bit .tiff sequence: and this is what happened after the H.265 10bit planar compression: Now, leaving aside the fact that the scope is "wrong" in the sense that it's not HDR Aware etc and it's not good to check those things, what you see is essentially what happens when you compress something with a consumer-tier bitrate most of the time. I've run the analysis and indeed I've got two very different results from the .tiff sequence compared to the final encoded version, sadly, which is expected, but still... How could have I missed that... Only humble pie for me today... |
|
12th March 2021, 16:26 | #18 | Link | |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
Quote:
hlg-tools-0.2.0 will simply use whatever the movie has for MaxCLL, although I'm telling users in the latest README to skip around to make sure the indicated value doesn't change (looking at you, Alita). And now that I know what MaxCLL really is (which should have been easier to find), pq2hlg just uses that and the indicated reference white value to generate the LUT. The end result is that the stream won't be over-compressed (tone mapped) and the source also doesn't have to do a separate pass through pqstat. All in all, I'm quite happy with the remedy to this latest complication. As a side note, I've noticed that the ITU doesn't want anything to do with HDR10 or any of the other metadata schemes. They'll redefine PQ (and even change its reference white level), but they won't touch PQ metadata. In fact, the word "metadata" appears just twice in BT.2390:
The word does not appear in BT.2100 or BT.2408. |
|
12th March 2021, 18:05 | #19 | Link |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
Not sure if I understand whole conversion correctly but...
MaxCLL can be misleading anyway even if value is correct. MaxCLL is based on single frame. Whole movie can be relatively dark and have one scene with 1000nits peak. If we use this 1000nits as max value won't we end up with not optimal HLG result? Would it not be better to have that one scene nod ideal (clipping?) then rest of the movie dull? Is this the case with your tool? I assume maybe not as we are converting to HLG which is also HDR. This is the problem when converting to SDR and this is why all good tools work "locally" not globally. |
12th March 2021, 19:34 | #20 | Link |
hlg-tools Maintainer
Join Date: Feb 2008
Posts: 413
|
The two main requirements for a good PQ-to-HLG conversion are:
1. Reference white needs to be at 203 nits. 2. MaxCLL needs to be at 1,000 nits. So here's an overview of how pq2hlg does things: 1. What is the reference white level? Adjust the linear brightness to bring whatever that is to 203 nits. 2. What is MaxCLL now that that's happened? Adjust MaxCLL by the same ratio to reflect the reference white adjustment. 3. Tone map MaxCLL down to 1,000 nits. 4. Run the HLG iOOTF. I've watched a few movies that I've done with this converter on MPV. The SDR range (everything below reference white) has looked good for a while now. But the utility has never really tone mapped the HDR range correctly. I intend to have that corrected by Monday. You can find SDR screenshots at the bottom of the Git repo's README. These still use the incorrect method of tone mapping, but it's not so obvious when MPV is downconverting to SDR. Last edited by wswartzendruber; 12th March 2021 at 19:37. |
|
|