Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > DVD & BD Rebuilder

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th August 2020, 00:09   #29821  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Update:

I'm currently still working on CQM/CRF encode prediction improvements. In testing, I'm finding that maybe my current prediction tables for HEVC/2160p sources using X265 may need to be adjusted. Unfortunately the means encoding multiple sources at every possible CRF value so I can calculate the relationship of each value to all the rest. It's just taking forever to do that at 4-7fps.

I'm also tuning the actual encode routine to prevent some of the anomalies that have been reported.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 8th August 2020, 00:24   #29822  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
@meadrocks

Can you post the ALTERNATE preset you used for HEVC/MP4 output? I tested it on different sources and it worked for me.

I notice you are using a multipart source. Try it on a single part source and see if that's related to the problem. I'd do it but my system is bogged down doing prediction table encodes.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 8th August 2020, 00:40   #29823  |  Link
meadrocks
Registered User
 
Join Date: Dec 2006
Location: Long Beach, Ca USA
Posts: 99
Quote:
Originally Posted by jdobbs View Post
@meadrocks

Can you post the ALTERNATE preset you used for HEVC/MP4 output?
Sorry, where do I find that info?
Do you mean this?
[00041]
caption=MP4 Container, HEVC 1920x1080, Auto-AAC
vEncoder=1
vBitrate=2000
vKeyint=Auto
aBitrate=*
aType=1
vFormat=5
cType=5

Last edited by meadrocks; 8th August 2020 at 00:45.
meadrocks is offline   Reply With Quote
Old 8th August 2020, 08:23   #29824  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,997
Quote:
Originally Posted by jdobbs View Post
I ran numerous tests on numerous sources using that setting. Even though the encode slowed down significantly (making you think it does more), the SSIM results were worse than the next lower setting (quality, without multipass). All the existing settings position in the quality list were based on SSIM values first and encode speed second.

My results could be wrong... but it would have to be wrong on every source I tried.
I have some doubt whether NVIDIA's 2pass method contributes much to quality in the sense of best match with the original or highest SSIM. They write:
Quote:
As a result, with 2-pass rate control modes, NVENC can distribute the bits more optimally within the frame and can reach closer to the target bitrate, especially for CBR encoding. Note, however, that everything else being the same, performance of 2-pass rate control mode is lower than that of 1-pass rate control mode. The client application should choose an appropriate multi-pass rate control mode after evaluating various modes, as each of the modes has its own advantages and disadvantages.
In earlier tests I found that disabling -aq resulted in better SSIM than using it. The explanation is probably that -aq is to some extent a psychovisual optimization which conflicts with PSNR -- and possibly also with SSIM --optimization.
From NVIDIA on Spatial AQ:
Quote:
Spatial AQ mode adjusts the QP values based on spatial characteristics of the frame. Since the low complexity flat regions are visually more perceptible to quality differences than high complexity detailed regions, extra bits are allocated to flat regions of the frame at the cost of the regions having high spatial detail. Although spatial AQ improves the perceptible visual quality of the encoded video, the required bit redistribution results in PSNR drop in most of the cases. Therefore, during PSNR-based evaluation, this feature should be turned off.
Perhaps we see a similar effect with the 2pass modes?

Last edited by Sharc; 8th August 2020 at 08:33.
Sharc is offline   Reply With Quote
Old 8th August 2020, 11:18   #29825  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 205
Quote:
Originally Posted by jdobbs
I ran numerous tests on numerous sources using that setting. Even though the encode slowed down significantly (making you think it does more), the SSIM results were worse than the next lower setting (quality, without multipass). All the existing settings position in the quality list were based on SSIM values first and encode speed second.
Quote:
Originally Posted by Sharc
I have some doubt whether NVIDIA's 2pass method contributes much to quality in the sense of best match with the original or highest SSIM. They write:
Quote:
As a result, with 2-pass rate control modes, NVENC can distribute the bits more optimally within the frame and can reach closer to the target bitrate, especially for CBR encoding. Note, however, that everything else being the same, performance of 2-pass rate control mode is lower than that of 1-pass rate control mode. The client application should choose an appropriate multi-pass rate control mode after evaluating various modes, as each of the modes has its own advantages and disadvantages.
Well that is surprising ... and I though 2pass gives the ultimate quality ...
Do you see the same on the other quality settings, where 2pass is involved?

Quote:
Originally Posted by jdobbs
Just as an aside: Have you tested the HDR10+ capability I added to v0.61.10? My player doesn't support HDR10+ so I could only do limited testing (header and value checks, etc.).
Actually my player also doesn't support HDR10+ while my TV does
I'm still in the process of evaluating players ... (any suggestions?)
I was hoping to get the meta information in an alternate output.

[edit]
Tried an encode of a HDR10+ disk... all fine up to the point where the source resolution (or compression) changes:
Code:
---------------------
[08.08.20] BD Rebuilder v0.61.10
[09:59:02] Source:  HOBBS_AND_SHAW
  - Input BD size: 87,50 GB
  - Approximate total content: [04:37:04.228]
  - Target BD size: 47,36 GB
  - Windows Version: 6.2 [9200]
  - Quality: Ultra-High (Extremely Slow), CQM
  - Decoding/Frame serving: NVENCC
  - Audio Settings: AC3=0 DTS=0 HD=1 Kbs=640
[09:59:05] PHASE ONE, Encoding
 - [09:59:05] Processing: VID_00165 (1 of 63)
 - [09:59:05] Extracting A/V streams [VID_00165]
 - [09:59:10] Reencoding video [VID_00165]
   - Source Video: HEVC, 3840x2160
   - Rate/Length: 23,976fps, 576 frames
 - [09:59:10] Performing CQM Prediction...
   - Analyzing 18,40 21,55 22,00 [22,08]
 - [09:59:32] Encoding using constant quality mode.
 - [09:59:51] Video Encode complete
 - [09:59:51] Processing audio tracks
   - Track 4352 (eng): Keeping original audio
 - [09:59:51] Multiplexing M2TS
 - [09:59:56] Processing: VID_00294 (2 of 63)
 - [09:59:56] Extracting A/V streams [VID_00294]
 - [10:05:08] Reencoding video [VID_00294]
   - Source Video: HEVC, 3840x2160
   - Rate/Length: 23,976fps, 195.520 frames
 - [10:05:08] Performing CQM Prediction...
   - Analyzing 18,90 21,45 21,75 [21,75]
 - [10:10:56] Encoding using constant quality mode.
 - [11:49:51] Video Encode complete
 - [11:49:51] Processing audio tracks
   - Track 4352 (eng): Keeping original audio
   - Track 4353 (deu): Keeping original audio
 - [11:49:51] Multiplexing M2TS
 - [11:51:52] Processing: VID_00368 (3 of 63)
 - [11:51:52] Extracting A/V streams [VID_00368]
 - [11:52:10] Reencoding video [VID_00368]
   - Source Video: HEVC, 3840x2160
   - Rate/Length: 23,976fps, 14.728 frames
 - [11:52:10] Performing CQM Prediction...
   - Analyzing 20,15 23,20 24,60 24,90 [24,90]
 - [11:52:38] Encoding using constant quality mode.
 - [12:00:04] Video Encode complete
 - [12:00:04] Processing audio tracks
   - Track 4352 (eng): Keeping original audio
 - [12:00:04] Multiplexing M2TS
 - [12:00:09] Processing: VID_00369 (4 of 63)
 - [12:00:09] Extracting A/V streams [VID_00369]
 - [12:00:16] Reencoding video [VID_00369]
   - Source Video: MPEG-4 (AVC), 1920x1080
   - Rate/Length: 23,976fps, 5.241 frames
 - [12:00:16] Performing CQM Prediction...
   - Analyzing 16,35 [12:00:17] - Failed video encode, aborted
Also tried an alternate output that fails if autocrop is enabled. Without ... went fine, but the MKV output file flickers when played directly on my TV, like the brightness meta info is switched on and off every half a second ...

Last edited by cartman0208; 8th August 2020 at 18:08.
cartman0208 is offline   Reply With Quote
Old 8th August 2020, 11:55   #29826  |  Link
Ch3vr0n
Registered User
 
Join Date: Jan 2009
Posts: 1,368
I tried the new FRIM, decode worked, encode process was stuck at , FPS (yes just a comma, that's not a typo) on my 9900k, then just failed on the newer one. Encoding wouldn't even start (after enabling dual GPU in the bios and installing the proper drivers and rebooting). Could be because the newer 1.30 zip contains a whole lot more files than the default 1.27. I just copied over the same 4 files and renamed the new FRIMDecode32.exe etc to FRIMDecode.exe

Tried the default one. Only copied the audio track for some reason. 26GB down to under 3GB for the main title including HD audio, yeah that aint right

oh and that is with the 61.05 version (i'll wait for the newer one until it gets updated with a newer frim)

Last edited by Ch3vr0n; 8th August 2020 at 12:11.
Ch3vr0n is offline   Reply With Quote
Old 8th August 2020, 22:50   #29827  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by cartman0208 View Post
Also tried an alternate output that fails if autocrop is enabled. Without ... went fine, but the MKV output file flickers when played directly on my TV, like the brightness meta info is switched on and off every half a second ...
Interesting. That would make sense, since the metadata is inserted into i-frames (about every half second).
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 8th August 2020, 22:58   #29828  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 205
In the alternate output with intact video there's no such issues ... does that mean the metadata only fits to the original P, B and I-Frame sequence (or at least I-Frame interval)?
I guess, it's pretty hard to synchronize that in a reencode
cartman0208 is offline   Reply With Quote
Old 8th August 2020, 23:45   #29829  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by cartman0208 View Post
In the alternate output with intact video there's no such issues ... does that mean the metadata only fits to the original P, B and I-Frame sequence (or at least I-Frame interval)?
I guess, it's pretty hard to synchronize that in a reencode
It should be by scene, and a scene should always start with an i-frame. So my belief is that the sequence within that GOP (P & B frames) shouldn't matter.

In order to make sure the metadata matches the original i-frame positioning, BD-RB grabs the frame numbers of all i-frames in the original and uses them to force i-frames at the same position in the reencode (in the .CHP file).

That takes place for sure when creating a new BD structure. It's possible that I may be losing that when doing ALTERNATE output. It may be affected by the ability to create much longer GOP sequences in ALTERNATE files. For example, in a BD there is a limit as to how long a GOP can be (typically it's every 24 frames for a film source). But in ALTERNATE output you often (very often) see it expanded to 240 frames. That might play havoc with positioning of the metadata (just speculating)

The actual placement of the metadata in the newly encoded stream is done by X265 or NVENCC using a JSON file that was created from the original source.

This is all speculation until I look at the code and do some testing. I'll take a look and see what I can find. Unfortunately I'm no expert on HDR10+ and/or exactly how JSON files are applied. BD-RB uses 3rd party software to create the JSON and X265/NVENCC to apply it.

The downside is that the reason most people use GOP lengths of 240 is for encoding efficiency. If you have to keep the original i-frame positions, that goes out the window. Generally an encoder senses scene changes and inserts an i-frame when it changes -- that makes it even more complex. I believe the .CHP file overrides that, since it explicitly forces i-frames at certain points.

[Edit] Oh, and by the way, the --multipass option does seem to give better SSIM values in all the other modes. It's just in QUALITY mode that I saw that anomaly.

[Edit - again] I just scanned a JSON file, and there seems to be an entry in the file for every frame in the source. That would seem to imply that the scene change information would be kept correctly even if I didn't use the .CHP to force i-frames. Any experts out there want to chime in?
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 9th August 2020 at 00:41.
jdobbs is offline   Reply With Quote
Old 9th August 2020, 00:46   #29830  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 205
Quote:
Originally Posted by jdobbs View Post
That takes place for sure when creating a new BD structure. It's possible that I may be losing that when doing ALTERNATE output. It may be affected by the ability to create much longer GOP sequences in ALTERNATE files. For example, in a BD there is a limit as to how long a GOP can be (typically it's every 24 frames for a film source). But in ALTERNATE output you often (very often) see it expanded to 240 frames. That might play havoc with positioning of the metadata (just speculating)
So just removing vKeyint=Auto from the alternate.txt could solve the problem?
I didn't know what to set and left it on all presets to Auto, since it's usually a good choice
cartman0208 is offline   Reply With Quote
Old 9th August 2020, 01:26   #29831  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by cartman0208 View Post
So just removing vKeyint=Auto from the alternate.txt could solve the problem?
I didn't know what to set and left it on all presets to Auto, since it's usually a good choice
After my last edit comment on my post, I'm not so sure. Since the JSON file contains data on every frame, it implies that the encoder can put a keyframe anywhere it wants -- but it can force one at the point where the metadata changes (a new scene) in the JSON file and add an SEI to the i-frame [keyframe] implementing the new metadata.

So that kinda' puts the theory of the GOP length as the cause as less likely.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 9th August 2020, 10:22   #29832  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 205
Quote:
Originally Posted by jdobbs View Post
After my last edit comment on my post, I'm not so sure. Since the JSON file contains data on every frame, it implies that the encoder can put a keyframe anywhere it wants -- but it can force one at the point where the metadata changes (a new scene) in the JSON file and add an SEI to the i-frame [keyframe] implementing the new metadata.

So that kinda' puts the theory of the GOP length as the cause as less likely.
OK, as you said... I had a look at source and output and the Keyframes match .. so that should not be the issue ...
I'll do some more testing on other sources
If that is fruitless, maybe we could join in here

[edit]
I ran several encodes with NVEnc ... all the same flickering while playing.
Then I decided to make a SW encode and despite the fact that Mediainfo is almost identical, the SW encoded file with HDR10+ Metadata looked very good on my TV

Last edited by cartman0208; 9th August 2020 at 22:35.
cartman0208 is offline   Reply With Quote
Old 9th August 2020, 23:22   #29833  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,997
Quote:
Originally Posted by jdobbs View Post
[Edit] Oh, and by the way, the --multipass option does seem to give better SSIM values in all the other modes. It's just in QUALITY mode that I saw that anomaly.
I am not aware of a document which explains what the presets or modes actually do in terms of setting or tuning the encoding parameters, and whether presets overwrite possibly conflicting commandline options. Or what is at the end just double stitching.
Maybe I am just missing something.
Sharc is offline   Reply With Quote
Old 10th August 2020, 05:16   #29834  |  Link
meadrocks
Registered User
 
Join Date: Dec 2006
Location: Long Beach, Ca USA
Posts: 99
Howls Moving Castle

Quote:
Originally Posted by jdobbs View Post
@meadrocks

Can you post the ALTERNATE preset you used for HEVC/MP4 output? I tested it on different sources and it worked for me.

I notice you are using a multipart source. Try it on a single part source and see if that's related to the problem. I'd do it but my system is bogged down doing prediction table encodes.
Watchmen Directors Cut Bluray also has the same problem with the English subtitles, they look correct duration in the source, but come out double length in the mkv. It also fails making a mp4 file. Its a multi part source. I cant find any single part source disks that have this problem.
meadrocks is offline   Reply With Quote
Old 10th August 2020, 13:16   #29835  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by Sharc View Post
I am not aware of a document which explains what the presets or modes actually do in terms of setting or tuning the encoding parameters, and whether presets overwrite possibly conflicting commandline options. Or what is at the end just double stitching.
Maybe I am just missing something.
Neither am I. One of the reasons it took so long to implement NVENC was all the testing I had to do to determine things like which settings actually improved SSIM values (so I could decide which settings would represent the different BD-RB quality presets).
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 10th August 2020, 13:38   #29836  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by cartman0208 View Post
OK, as you said... I had a look at source and output and the Keyframes match .. so that should not be the issue ...
I'll do some more testing on other sources
If that is fruitless, maybe we could join in here

[edit]
I ran several encodes with NVEnc ... all the same flickering while playing.
Then I decided to make a SW encode and despite the fact that Mediainfo is almost identical, the SW encoded file with HDR10+ Metadata looked very good on my TV
By SW encode, do you mean using "--avsw" in the command line instead of "--avhw"?

Have you tried using --avhw but changing the use of the JSON file to "--dhdr10-info copy"?
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 10th August 2020, 13:41   #29837  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by meadrocks View Post
Watchmen Directors Cut Bluray also has the same problem with the English subtitles, they look correct duration in the source, but come out double length in the mkv. It also fails making a mp4 file. Its a multi part source. I cant find any single part source disks that have this problem.
Thanks. I'll look at the code and see how it might be impacted by multi-part sources.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 10th August 2020, 14:06   #29838  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by cartman0208 View Post
Also tried an alternate output that fails if autocrop is enabled. Without ... went fine, but the MKV output file flickers when played directly on my TV, like the brightness meta info is switched on and off every half a second ...
I forgot to mention on this one -- if you use AUTOCROP it would force use of an AVS, which in turn causes you to lose HDR. Of course it shouldn't fail, and I'll look at that.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 10th August 2020, 14:15   #29839  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 205
Quote:
Originally Posted by jdobbs View Post
By SW encode, do you mean using "--avsw" in the command line instead of "--avhw"?
No ... I switched the encoder from nvencc to x264 & x265

Quote:
Originally Posted by jdobbs View Post
Have you tried using --avhw but changing the use of the JSON file to "--dhdr10-info copy"?
Tried that, no change

Quote:
Originally Posted by jdobbs View Post
I forgot to mention on this one -- if you use AUTOCROP it would force use of an AVS, which in turn causes you to lose HDR. Of course it shouldn't fail, and I'll look at that.
I read in other forums, that crop or resize don't get along with HDR10+, so I'll stick with the original resolution for those cases.

The currently used autocrop version had other issues, as we found out earlier in the thread...

Last edited by cartman0208; 10th August 2020 at 14:27.
cartman0208 is offline   Reply With Quote
Old 10th August 2020, 16:21   #29840  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by cartman0208 View Post
I read in other forums, that crop or resize don't get along with HDR10+, so I'll stick with the original resolution for those cases.

The currently used autocrop version had other issues, as we found out earlier in the thread...
Yeah, that makes sense. The HDR10+ metadata adjusts for things like the average brightness of the picture, etc., and then when you crop the black portion suddenly it changes the picture dramatically and it's unlikely that the calculated values used in HDR10+ apply anymore.

So... the JSON works with X265, but fails with NVENCC. And if you remove it (replaced by "copy") it still doesn't work... that means it isn't the JSON (or the use of it). It sure sounds like the problem is within NVENCC (or at least the way BD-RB is using it).

Anybody else out there done any testing of HDR10+? Your experiences?
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 10th August 2020 at 16:24.
jdobbs is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:24.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.