Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > DVD & BD Rebuilder

Reply
 
Thread Tools Search this Thread Display Modes
Old 1st July 2020, 21:54   #29561  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Posts: 1,023
Maybe a driver related issue, memory usage is constant at about 582 MB on my GTX1080ti
Driver is 445.87
gonca is offline   Reply With Quote
Old 1st July 2020, 22:29   #29562  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 132
I don't see a lot of memory growth, even on the 32bit version ... starts at 750MB and finishes with 850MB used by nvencc.exe

What I also noticed ... no matter what preset or rate factor ... the processing speed stays the same for a given resolution ... 4K is always around 75fps for me

Graphics driver was updated yesterday (451.48), but I'm not on Win10 May 2020 update yet, if that helps.

Quote:
PS C:\WINDOWS\system32> U:\Temp\nvenc\NVEncC.exe --cqp 25 --codec h265 --preset quality --profile main10 --level 5.1 --output-depth 10 --max-bitrate 30000 --aq --aq-temporal --output-buf 128 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)" --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --max-cll "1000,0" --chromaloc 2 --output "U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc" --input "U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.h265" --repeat-headers
Code:
--------------------------------------------------------------------------------
U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc
--------------------------------------------------------------------------------
NVEncC (x86) 5.08 (r1585) by rigaya, Jul  1 2020 14:48:28 (VC 1900/Win/avx2)
OS Version     Windows 10 x64 (18363)
CPU            Intel Core i7-6800K @ 3.40GHz [TB: 3.80GHz] (6C/12T)
GPU            #0: GeForce GTX 1660 Ti (1536 cores, 1770 MHz)[2147483.64]
NVENC / CUDA   NVENC API 9.1, CUDA 11.0, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.265/HEVC, 3840x2160, 24000/1001 fps
Vpp Filters    copyDtoD
Output Info    H.265/HEVC main10 @ Level 5.1
               3840x2160p 1:1 23.976fps (24000/1001fps)
Encoder Preset quality
Rate Control   CQP  I:25  P:25  B:25
Lookahead      off
GOP length     240 frames
B frames       3 frames [ref mode: disabled]
Ref frames     3 frames, MultiRef L0:auto L1:auto
AQ             on
CU max / min   auto / auto
VUI            matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
MasteringDisp  G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(999.999975 0.000100)
MaxCLL/MaxFALL 1000/0
Others         mv:auto repeat-headers

encoded 175440 frames, 76.88 fps, 7280.96 kbps, 6351.12 MB
encode time 0:38:02, CPU: 9.8%, GPU: 7.4%, VE: 99.2%, VD: 20.6%
frame type IDR    731
frame type I      731,  total size    70.98 MB
frame type P    43860,  total size  2313.22 MB
frame type B   130849,  total size  3966.92 MB
Edit: 1080p runs at 175fps stable with above settings, no big memory increase either...

Last edited by cartman0208; 1st July 2020 at 22:42.
cartman0208 is offline   Reply With Quote
Old 1st July 2020, 22:59   #29563  |  Link
Mike-uk
Registered User
 
Join Date: Jun 2018
Location: Dorset
Posts: 136
Quote:
Originally Posted by cartman0208 View Post
I don't see a lot of memory growth, even on the 32bit version ... starts at 750MB and finishes with 850MB used by nvencc.exe

What I also noticed ... no matter what preset or rate factor ... the processing speed stays the same for a given resolution ... 4K is always around 75fps for me

Graphics driver was updated yesterday (451.48), but I'm not on Win10 May 2020 update yet, if that helps.


[CODE]
--------------------------------------------------------------------------------
U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc
--------------------------------------------------------------------------------
where is your --avhw to enable fully hardware encode ?
Mike-uk is offline   Reply With Quote
Old 1st July 2020, 23:24   #29564  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 132
Forgot about it ...
Maybe it was auto detected?

from the help text...
Code:
--avhw          use libavformat + hw decode for input
Does it matter what is used for decoding?
Just started another run ... not much of a difference in speed
cartman0208 is offline   Reply With Quote
Old 1st July 2020, 23:29   #29565  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Quote:
Originally Posted by gonca View Post
Maybe a driver related issue, memory usage is constant at about 582 MB on my GTX1080ti
Driver is 445.87
I'm using Driver 451.48, dated June 24th, 2020 on a GTX1660
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 1st July 2020, 23:44   #29566  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Quote:
Originally Posted by cartman0208 View Post
I don't see a lot of memory growth, even on the 32bit version ... starts at 750MB and finishes with 850MB used by nvencc.exe

What I also noticed ... no matter what preset or rate factor ... the processing speed stays the same for a given resolution ... 4K is always around 75fps for me

Graphics driver was updated yesterday (451.48), but I'm not on Win10 May 2020 update yet, if that helps.


Code:
--------------------------------------------------------------------------------
U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc
--------------------------------------------------------------------------------
NVEncC (x86) 5.08 (r1585) by rigaya, Jul  1 2020 14:48:28 (VC 1900/Win/avx2)
OS Version     Windows 10 x64 (18363)
CPU            Intel Core i7-6800K @ 3.40GHz [TB: 3.80GHz] (6C/12T)
GPU            #0: GeForce GTX 1660 Ti (1536 cores, 1770 MHz)[2147483.64]
NVENC / CUDA   NVENC API 9.1, CUDA 11.0, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.265/HEVC, 3840x2160, 24000/1001 fps
Vpp Filters    copyDtoD
Output Info    H.265/HEVC main10 @ Level 5.1
               3840x2160p 1:1 23.976fps (24000/1001fps)
Encoder Preset quality
Rate Control   CQP  I:25  P:25  B:25
Lookahead      off
GOP length     240 frames
B frames       3 frames [ref mode: disabled]
Ref frames     3 frames, MultiRef L0:auto L1:auto
AQ             on
CU max / min   auto / auto
VUI            matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
MasteringDisp  G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(999.999975 0.000100)
MaxCLL/MaxFALL 1000/0
Others         mv:auto repeat-headers

encoded 175440 frames, 76.88 fps, 7280.96 kbps, 6351.12 MB
encode time 0:38:02, CPU: 9.8%, GPU: 7.4%, VE: 99.2%, VD: 20.6%
frame type IDR    731
frame type I      731,  total size    70.98 MB
frame type P    43860,  total size  2313.22 MB
frame type B   130849,  total size  3966.92 MB
Edit: 1080p runs at 175fps stable with above settings, no big memory increase either...
Part of that might be because you are using GOPs of 240 frames. That's 10 times larger than what it would need to be for a UHD-BD encode. The --repeat-headers would not be activated as often and would probably not hit the routine that causes the memory leak as often. Try adding --gop-len 24 and see what happens.

[Edit] Yeah. I just set the GOP length to 240 frames and it is barely growing at all... but, of course, that encode would fail to meet the UHD-BD standard. I would assume that it would also mean that as you attempt to fast forward, each picture step would jump 10 seconds instead of 1. The table used for these actions only logs IDR positions.

Quote:
Originally Posted by gonca View Post
Finished a remux and it seems to work FF, REW and CHP skips
I just went back and looked at your post again as well... and you are also using the default 240 frame GOP. That's probably why you're not seeing the issue as well. You are only having to insert the SPS/VPS/PPS once every 10 seconds instead of once per second. The headers are only inserted on the IDR frame that begins a GOP.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 2nd July 2020 at 00:12.
jdobbs is offline   Reply With Quote
Old 1st July 2020, 23:49   #29567  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Here's a look at memory usage at the start of an encode and then after 35,000 frames. Considering that a 2 hour movie would be 172,627 frames, you can see how this can become an issue.
Attached Images
  
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 1st July 2020 at 23:58.
jdobbs is offline   Reply With Quote
Old 2nd July 2020, 00:51   #29568  |  Link
Mike-uk
Registered User
 
Join Date: Jun 2018
Location: Dorset
Posts: 136
am i right in seeing nvencc uses SDK version 9.1, i think there is a version 10 now ??, unfortunatly that would mean we need to wait for nvencc to compile sdk 10 versions

mayby ask him if he intends to compile a SDK 10 verstion ? mayby that will solve the memory leak

What's new in Video Codec SDK 10
New H.264 NVENC presets for improved flexibility and granularity
Finer control of rate control modes
Use-case Setting (also referred to as Tuning information)
NVDEC Support for GA100

Last edited by Mike-uk; 2nd July 2020 at 00:56.
Mike-uk is offline   Reply With Quote
Old 2nd July 2020, 01:20   #29569  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
I think the bottom line for now is that I am going to have to insert the headers myself from within BD-RB, which will increase the overall job time by about 10 or 15 minutes. It's a shame, but based on the last response I saw, I have doubts that the issue is going to be addressed. Apparently no one except me is even attempting to create a UHD-BD that is standards compliant (at least with NVENCC), or the --repeat-headers issue would have been brought up long ago.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 2nd July 2020, 01:40   #29570  |  Link
gonca
Registered User
 
Join Date: Jul 2012
Posts: 1,023
I should have checked the cmd line a little better
Still, 45 min encode and 15 min clean up for 4K movie isn't bad
By the way, username eddie haskell
Leave it to Beaver fan?
gonca is offline   Reply With Quote
Old 2nd July 2020, 02:17   #29571  |  Link
Mike-uk
Registered User
 
Join Date: Jun 2018
Location: Dorset
Posts: 136
how do you view if repeat headers worked ?? i tried another encoder that uses nvencc and added --repeat-headers and it started off allocating 1,2 gig of memory and stayed around there but no idea if headers are repeated
Mike-uk is offline   Reply With Quote
Old 2nd July 2020, 02:17   #29572  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Quote:
Originally Posted by gonca View Post
I should have checked the cmd line a little better
Still, 45 min encode and 15 min clean up for 4K movie isn't bad
By the way, username eddie haskell
Leave it to Beaver fan?
I watched it after school when I was a kid. Eddie Haskell was quite a character -- I've been using that name for one thing or another for years. If you look hard enough you might even find the name "Haskell Industries" buried somewhere in the BD Rebuilder code...
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 2nd July 2020, 02:22   #29573  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Quote:
Originally Posted by Mike-uk View Post
how do you view if repeat headers worked ?? i tried another encoder that uses nvencc and added --repeat-headers and it started off allocating 1,2 gig of memory and stayed around there but no idea if headers are repeated
It's not easy without the right tools. I have an HEVC browser.

If you're good with a hex editor, you can find the delimiters between headers, slices etc by searching for the start code delimiter (00 00 00 01) -- it is prohibited from appearing anywhere else in the elementary stream. But even if you find the delimiters you'll need to reference the H265 standard (T-REC-H.265-201802-I) to find the format of all the segments that might follow a delimeter (see page 68). The standard is available for free download as a PDF.

The best way to see if the headers are there is to use TSMUXER to create a UHD-BD structure and play it back on a UHD compatible BD player. If FF/REW works... it's probably safe to assume the headers are repeated.

Just out of curiosity... what is the other encoder? It would likely be random luck if it would use the same command line parameter. I think NVENCC uses it because I suggested it (so it would match the parameter in X265).
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 2nd July 2020 at 02:30.
jdobbs is offline   Reply With Quote
Old 2nd July 2020, 07:36   #29574  |  Link
GreasyGreedo
Registered User
 
Join Date: Nov 2019
Posts: 5
Hello,

I tried searching to see if this problem has been reported otherwise, but didn't see it.

When I author a UHD disc with multiple movie files (quick reauthored import), the subtitles are lost from the final disc. I have not had this problem with HD discs. Is there some difference that prevents the subtitles from being present on a UHD disc of this type?
GreasyGreedo is offline   Reply With Quote
Old 2nd July 2020, 09:09   #29575  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 132
Quote:
Originally Posted by jdobbs View Post
[Edit] Yeah. I just set the GOP length to 240 frames and it is barely growing at all... but, of course, that encode would fail to meet the UHD-BD standard. I would assume that it would also mean that as you attempt to fast forward, each picture step would jump 10 seconds instead of 1. The table used for these actions only logs IDR positions.
Ok, so that's why I saw blocky artifacts for several seconds when jumping to random positions in the SW Player...
So I started a new run with gop-len 24 and the memory increase is actually higher ... started out at around 600 and went up to 1500MB after 180000 frames.
Maybe with a 3hr movie file the 2GB mark would have cracked ... maybe...

Then I tried your commandline and an 1080p video ... stable at around 300MB, no increase at all
The only difference ... I didn't use the keyfile-parameter
Quote:
.\NVEncC.exe --avhw -i U:\Temp\Spectre_temp\Spectre.h264 --codec h264 --preset quality --bluray --qp-min 0 --vbr 22429 --sar 1:1 --aud --pic-struct --vbv-bufsize 30000 --max-bitrate 35000 --gop-len 24 -o U:\Temp\Spectre_temp\Spectre.264 --repeat-headers
Code:
--------------------------------------------------------------------------------
U:\Temp\Spectre_temp\Spectre.264
--------------------------------------------------------------------------------
NVEncC (x86) 5.08 (r1585) by rigaya, Jul  1 2020 14:48:28 (VC 1900/Win/avx2)
OS Version     Windows 10 x64 (18363)
CPU            Intel Core i7-6800K @ 3.40GHz [TB: 3.80GHz] (6C/12T)
GPU            #0: GeForce GTX 1660 Ti (1536 cores, 1770 MHz)[2147483.64]
NVENC / CUDA   NVENC API 9.1, CUDA 11.0, schedule mode: auto
Input Buffers  CUDA, 20 frames
Input Info     avcuvid: H.264/AVC, 1920x1080, 24000/1001 fps
Vpp Filters    copyDtoD
Output Info    H.264/AVC high @ Level auto
               1920x1080p 1:1 23.976fps (24000/1001fps)
Encoder Preset quality
Rate Control   VBR
Bitrate        22429 kbps (Max: 35000 kbps)
Target Quality auto
Initial QP     I:20  P:23  B:25
QP range       I:0-51  P:0-51  B:0-51
VBV buf size   30000 kbit
Lookahead      off
GOP length     24 frames
B frames       3 frames [ref mode: disabled]
Ref frames     3 frames, MultiRef L0:auto L1:auto
AQ             off
Others         mv:auto cabac deblock adapt-transform:auto bdirect:auto aud pic-struct repeat-headers

encoded 213072 frames, 412.48 fps, 21838.53 kbps, 23135.71 MB
encode time 0:08:36, CPU: 9.5%, GPU: 7.1%, VE: 96.2%, VD: 78.7%
frame type IDR   8878
frame type I     8878,  total size   1702.42 MB
frame type P    53268,  total size   8475.27 MB
frame type B   150926,  total size  12958.02 MB
Apparently I cannot reproduce the error ... only a memory increase
cartman0208 is offline   Reply With Quote
Old 2nd July 2020, 13:54   #29576  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
@cartman0208

I removed the keyfile parameter and the memory leak persists for me. Not sure how my test differs from yours. As for the encode failure... I think the problem would also depend on the source you are using. So it will happen, it's just a matter of time. You actually need the keyfile to make sure there is an IDR frame at each chapter point.

But... I have good news. If you change the output format from an elementary stream (.hevc) to a container (e.g. .m2ts)... the problem seems to go away. That provides a good workaround that won't require BD-RB to have to rewrite the entire HEVC stream! I should be able to use the .m2ts as an input file for the TSMUXER muxing. It'll require some code changes -- but I don't think it should be anything too extreme.

My thanks to jkiku101 for experimenting with encodes and finding this potential workaround!
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 2nd July 2020 at 16:12.
jdobbs is offline   Reply With Quote
Old 2nd July 2020, 14:00   #29577  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Quote:
Originally Posted by GreasyGreedo View Post
Hello,

I tried searching to see if this problem has been reported otherwise, but didn't see it.

When I author a UHD disc with multiple movie files (quick reauthored import), the subtitles are lost from the final disc. I have not had this problem with HD discs. Is there some difference that prevents the subtitles from being present on a UHD disc of this type?
I can't think of a difference. But I'll look and see.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 2nd July 2020, 18:13   #29578  |  Link
cartman0208
Registered User
 
Join Date: Jun 2010
Location: Germany
Posts: 132
Quote:
Originally Posted by jdobbs View Post
@cartman0208

I removed the keyfile parameter and the memory leak persists for me. Not sure how my test differs from yours. As for the encode failure... I think the problem would also depend on the source you are using. So it will happen, it's just a matter of time. You actually need the keyfile to make sure there is an IDR frame at each chapter point.

But... I have good news. If you change the output format from an elementary stream (.hevc) to a container (e.g. .m2ts)... the problem seems to go away. That provides a good workaround that won't require BD-RB to have to rewrite the entire HEVC stream! I should be able to use the .m2ts as an input file for the TSMUXER muxing. It'll require some code changes -- but I don't think it should be anything too extreme.

My thanks to jkiku101 for experimenting with encodes and finding this potential workaround!
Ok, then maybe it's really my 1909 Win10 Build for which I did not receive an update yet ... and 2004 is messed up with memory leak ...

Doesn't matter ... good to have a solution for this ... now to find the best settings for awesome quality
cartman0208 is offline   Reply With Quote
Old 3rd July 2020, 15:08   #29579  |  Link
Mike-uk
Registered User
 
Join Date: Jun 2018
Location: Dorset
Posts: 136
Quote:
Originally Posted by cartman0208 View Post

Doesn't matter ... good to have a solution for this ... now to find the best settings for awesome quality
yes being hardware encoding and to contend with x265/264 we need to get the best possible settings dialed in, but will be nice to have the option of considerably faster encodes but slightly less quality with HW, or still the SW/cpu at 16Hrs for absolute best :/
Mike-uk is offline   Reply With Quote
Old 4th July 2020, 01:32   #29580  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,665
Quote:
Originally Posted by Mike-uk View Post
yes being hardware encoding and to contend with x265/264 we need to get the best possible settings dialed in, but will be nice to have the option of considerably faster encodes but slightly less quality with HW, or still the SW/cpu at 16Hrs for absolute best :/
Yep. It's all about choice.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:02.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.