BD Rebuilder Beta - Bug Reports Only

[Mike-uk]

Originally Posted by MrVideo

I think you mean acceleration. Also, when posting long lists like that, please CODE wrap them.

yes acceleration, code wrap done

[jdobbs]

Originally Posted by cartman0208

Can you post that commandline?

"D:\BD_Rebuilder\tools\nvenc\nvencc.exe" --avhw -i "N:\BD\SOURCE\BDMV\STREAM\00500.m2ts" --codec h264 --preset quality --bluray --qp-min 0 --vbr 22429 --keyfile "D:\WORKING4\WORKFILES\VID_00500.CHP" --sar 1:1 --aud --pic-struct --vbv-bufsize 30000 --max-bitrate 35000 --gop-len 24 -o "D:\WORKING4\WORKFILES\VID_00500.AVS.264"

Originally Posted by Mike-uk

hmm im only get 66 FPS regardless how many time i restart, just a note 400 fps sounds too quick, gpu encoding doesnt give same quality as cpu, so id be happy for gpu to be spitting out the best quality it can do "--preset quality" even if that means less fps, im sure gpu at max quality will be a lot quicker than cpu, will be nice to have the option of HW excelearion, will you be including 264 hw exceleration also ?

I'm guessing you are talking about a 2160p source. The one in which I was getting the goofy numbers was a h264 stream at 1080p.

[Mike-uk]

Originally Posted by jdobbs

"
I'm guessing you are talking about a 2160p source. The one in which I was getting the goofy numbers was a h264 stream at 1080p.

yes was 2160p

[jdobbs]

Just a head's up. I tested the new version of NVENCC yesterday, and the new option to include headers results an encoding failure with a "Cannot allocate memory" message. Removing the new option from the command line makes the encoding error goes away. But, then, of course you're back to having the FF/REW/CHAP issues. I posted an issue report on GITHUB. It appears that the new option creates a memory leak (as can be observed by watching the process under the Windows 10 task manager). On the 32 bit version of NCENCC it hits the memory limitation pretty quickly (you have to use the 32 bit version to remain compatible with AVISYNTH 2.60 -- which must be used under certain circumstances).

So the options right now are to wait and see if it gets another quick fix, or to go ahead and implement the code that adds the headers within BD-RB. I'd prefer the first option, as that's a cleaner way to go. So I'm going to wait for a bit and see what happens.

[Mike-uk]

Id wait a while and see if it gets fixed, i can also conform its eating memory, at frame 69522 it consumed 1.6 gig of memory and then errored, im running 16 gig but according to task manager i was only at 50% mem ussage ?

[gonca]

I think the wait is over
5.08 is out

[Mike-uk]

Originally Posted by gonca

I think the wait is over
5.08 is out

unfortunatly not not good news

Please try with x64 build of NVEnc 5.08, it should not require nppc64_10.dll now.

It might be a memory leak but it's difficult to fix, as "--repeat-headers" only sets a parameter provided by NVENC SDK, there is little I can do here.

[gonca]

Yes, but issue 246 seemed to develop with 5.07 has been resolved
Running a quick test, to see if it finishes

[gonca]

Update 1
80000 frames out of 188855
RAM usage is steady 582.5 MB
x265 is using 6.7 GB

Update 2
110000 frames

Completed fine

[2020-07-01][12:02:46] Simple x264 Launcher (Build #1208), built 2020-06-25
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Job started at 2020-07-01, 12:02:46.
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Source file : I:\PACIFIC RIM.vpy
[2020-07-01][12:02:46] Output file : T:\\PACIFIC RIM.hevc
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] --- SYSTEMINFO ---
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Binary Path : C:\Users\LUIS\Desktop\x264_launcher.2020-06-25
[2020-07-01][12:02:46] Avisynth    : Yes
[2020-07-01][12:02:46] VapourSynth : Yes
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] --- SETTINGS ---
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Encoder : NVEncC, 64-Bit (x64), HEVC
[2020-07-01][12:02:46] Source  : VapourSynth (vpy)
[2020-07-01][12:02:46] RC Mode : CQP
[2020-07-01][12:02:46] Preset  : performance
[2020-07-01][12:02:46] Tuning  : <None>
[2020-07-01][12:02:46] Profile : main10
[2020-07-01][12:02:46] Custom  :
 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" 
--max-cll "2465,823" --frames 188855 --chromaloc 2 --colormatrix bt2020nc --colorprim bt2020 
--transfer smpte2084 --repeat-headers

[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] --- CHECK VERSION ---
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Detect video encoder version:
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Creating process:
[2020-07-01][12:02:46] C:\Users\LUIS\Desktop\x264_launcher.2020-06-25\toolset\x64\nvencc\nvencc64.exe --version
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] NVEncC (x64) 5.08 (r1585) by rigaya, Jul 1 2020 15:17:08 (VC 1916/Win/avx2)
[2020-07-01][12:02:46] [NVENC API v9.1, CUDA 10.1]
[2020-07-01][12:02:46] reader: raw, y4m, avi, avs, vpy, avsw, avhw [H.264/AVC, H.265/HEVC, MPEG2, VP8, VP9, VC-1, MPEG1, MPEG4]
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] WARNING: Your version of NVEncC is newer than the latest tested version, take care!
[2020-07-01][12:02:46] This application works best with NVEncC version 5.6. Newer versions may work or not.
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Detect video source version:
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Creating process:
[2020-07-01][12:02:46] "C:\Program Files\VapourSynth\core\vspipe.exe" --version
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] VapourSynth Video Processing Library
[2020-07-01][12:02:46] Copyright (c) 2012-2020 Fredrik Mellbin
[2020-07-01][12:02:46] Core R50
[2020-07-01][12:02:46] API R3.6
[2020-07-01][12:02:46] Options: -
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] > NVEncC version: 5.80 [rev #1585]
[2020-07-01][12:02:46] > VapourSynth version: r50 (API r3)
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] --- GET SOURCE INFO ---
[2020-07-01][12:02:46] 
[2020-07-01][12:02:46] Creating process:
[2020-07-01][12:02:46] "C:\Program Files\VapourSynth\core\vspipe.exe" --info "I:\\pacific rim.vpy" -
[2020-07-01][12:02:46] 
[2020-07-01][12:02:47] Width: 3840
[2020-07-01][12:02:47] Height: 2160
[2020-07-01][12:02:47] Frames: 188855
[2020-07-01][12:02:47] FPS: 24000/1001 (23.976 fps)
[2020-07-01][12:02:47] Format Name: YUV420P10
[2020-07-01][12:02:47] Color Family: YUV
[2020-07-01][12:02:47] Alpha: No
[2020-07-01][12:02:47] Sample Type: Integer
[2020-07-01][12:02:47] Bits: 10
[2020-07-01][12:02:47] SubSampling W: 1
[2020-07-01][12:02:47] SubSampling H: 1
[2020-07-01][12:02:47] 
[2020-07-01][12:02:47] Resolution: 3840 x 2160
[2020-07-01][12:02:47] Frame Rate: 24000/1001
[2020-07-01][12:02:47] No. Frames: 188855
[2020-07-01][12:02:47] 
[2020-07-01][12:02:47] --- ENCODING VIDEO ---
[2020-07-01][12:02:47] 
[2020-07-01][12:02:47] Creating input process:
[2020-07-01][12:02:47] "C:\Program Files\VapourSynth\core\vspipe.exe" --y4m "I:\\pacific rim.vpy" -
[2020-07-01][12:02:47] 
[2020-07-01][12:02:47] WARNING: Custom parameter "--frames" will be ignored in Pipe'd mode!
[2020-07-01][12:02:47] 
[2020-07-01][12:02:47] Creating encoder process:
[2020-07-01][12:02:47] C:\Users\LUIS\Desktop\x264_launcher.2020-06-25\toolset\x64\nvencc\nvencc64.exe 
--codec hevc --preset performance --cqp 30 --profile main10 
--master-display G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50) 
--max-cll 2465,823 
--chromaloc 2 --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --repeat-headers 
--output "T:\\PACIFIC RIM.hevc" --y4m --input -

[2020-07-01][12:02:47] 
[2020-07-01][12:02:50] --------------------------------------------------------------------------------
[2020-07-01][12:02:50] T:\\PACIFIC RIM.hevc
[2020-07-01][12:02:50] --------------------------------------------------------------------------------
[2020-07-01][12:02:50] Max B frames are 0 frames.
[2020-07-01][12:02:50] NVEncC (x64) 5.08 (r1585) by rigaya, Jul 1 2020 15:17:08 (VC 1916/Win/avx2)
[2020-07-01][12:02:50] OS Version Windows 10 x64 (14393)
[2020-07-01][12:02:50] CPU Intel Core i7-6900K @ 3.20GHz [TB: 3.50GHz] (8C/16T)
[2020-07-01][12:02:50] GPU #0: GeForce GTX 1080 Ti (3584 cores, 1670 MHz)[PCIe3x16][445.87]
[2020-07-01][12:02:50] NVENC / CUDA NVENC API 9.1, CUDA 11.0, schedule mode: auto
[2020-07-01][12:02:50] Input Buffers CUDA, 13 frames
[2020-07-01][12:02:50] Input Info y4m(yv12(10bit))->p010 [AVX2], 3840x2160, 24000/1001 fps
[2020-07-01][12:02:50] Vpp Filters copyHtoD
[2020-07-01][12:02:50] Output Info H.265/HEVC main10 @ Level auto
[2020-07-01][12:02:50] 3840x2160p 1:1 23.976fps (24000/1001fps)
[2020-07-01][12:02:50] Encoder Preset performance
[2020-07-01][12:02:50] Rate Control CQP I:30 P:30 B:30
[2020-07-01][12:02:50] Lookahead off
[2020-07-01][12:02:50] GOP length 240 frames
[2020-07-01][12:02:50] B frames 0 frames [ref mode: disabled]
[2020-07-01][12:02:50] Ref frames 3 frames
[2020-07-01][12:02:50] AQ off
[2020-07-01][12:02:50] CU max / min auto / auto
[2020-07-01][12:02:50] VUI matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
[2020-07-01][12:02:50] MasteringDisp G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
[2020-07-01][12:02:50] WP(0.312700 0.329000) L(4000.000000 0.005000)
[2020-07-01][12:02:50] MaxCLL/MaxFALL 2465/823
[2020-07-01][12:02:50] Others mv:auto repeat-headers
[2020-07-01][12:48:45] encoded 188855 frames, 68.51 fps, 4745.73 kbps, 4456.20 MB
[2020-07-01][12:48:45] encode time 0:45:56, CPU: 8.4, GPU: 35.4, VE: 32.4, VD: 39.3, GPUClock: 1936MHz, VEClock: 1645MHz
[2020-07-01][12:48:45] frame type IDR 787
[2020-07-01][12:48:45] frame type I 787, total size 40.78 MB
[2020-07-01][12:48:45] frame type P 188068, total size 4415.41 MB
[2020-07-01][12:48:45] vpyp [info]: Output 188855 frames in 2757.37 seconds (68.49 fps)
[2020-07-01][12:48:45] Final file size is 4.35 GB bytes.
[2020-07-01][12:48:45] 
[2020-07-01][12:48:45] --- COMPLETED ---
[2020-07-01][12:48:45] 
[2020-07-01][12:48:45] Job finished at 2020-07-01, 12:48:45. Process took 45 minutes, 59 seconds.

[gonca]

Finished a remux and it seems to work
FF, REW and CHP skips

[jdobbs]

Originally Posted by gonca

I think the wait is over
5.08 is out

The issue is still there. It will probably have to be fixed by NVIDIA.

It's very obvious. Try this:

1. Run NVENCC with the --repeat-headers parameter
2. Open up Windows Task Manager and click on "Details"
3. Watch the memory usage for NVENCC.EXE.

The application's memory usage will continually rise (pretty quickly) until it eventually runs out and the app crashes. It's just more obvious on the 32 bit version because of 32 memory limitations.

[jdobbs]

Originally Posted by gonca

Finished a remux and it seems to work
FF, REW and CHP skips

Even if you use the 64 bit version, the memory usage keeps growing (at least on my system). But the application is less likely to crash since 64 bit apps have much more memory available. I can use the --repeat-headers on the 64 bit version with a 2 1/2 hour movie and it will complete. But... the application is using about 600 Megabytes when it starts then slowly increases to point at which it is using over 2 GigaBytes when it finishes.

Who knows? It may act differently on your card and/or drivers.

[gonca]

Maybe a driver related issue, memory usage is constant at about 582 MB on my GTX1080ti
Driver is 445.87

[cartman0208]

I don't see a lot of memory growth, even on the 32bit version ... starts at 750MB and finishes with 850MB used by nvencc.exe

What I also noticed ... no matter what preset or rate factor ... the processing speed stays the same for a given resolution ... 4K is always around 75fps for me

Graphics driver was updated yesterday (451.48), but I'm not on Win10 May 2020 update yet, if that helps.

PS C:\WINDOWS\system32> U:\Temp\nvenc\NVEncC.exe --cqp 25 --codec h265 --preset quality --profile main10 --level 5.1 --output-depth 10 --max-bitrate 30000 --aq --aq-temporal --output-buf 128 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)" --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --max-cll "1000,0" --chromaloc 2 --output "U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc" --input "U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.h265" --repeat-headers

--------------------------------------------------------------------------------
U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc
--------------------------------------------------------------------------------
NVEncC (x86) 5.08 (r1585) by rigaya, Jul  1 2020 14:48:28 (VC 1900/Win/avx2)
OS Version     Windows 10 x64 (18363)
CPU            Intel Core i7-6800K @ 3.40GHz [TB: 3.80GHz] (6C/12T)
GPU            #0: GeForce GTX 1660 Ti (1536 cores, 1770 MHz)[2147483.64]
NVENC / CUDA   NVENC API 9.1, CUDA 11.0, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.265/HEVC, 3840x2160, 24000/1001 fps
Vpp Filters    copyDtoD
Output Info    H.265/HEVC main10 @ Level 5.1
               3840x2160p 1:1 23.976fps (24000/1001fps)
Encoder Preset quality
Rate Control   CQP  I:25  P:25  B:25
Lookahead      off
GOP length     240 frames
B frames       3 frames [ref mode: disabled]
Ref frames     3 frames, MultiRef L0:auto L1:auto
AQ             on
CU max / min   auto / auto
VUI            matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
MasteringDisp  G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(999.999975 0.000100)
MaxCLL/MaxFALL 1000/0
Others         mv:auto repeat-headers

encoded 175440 frames, 76.88 fps, 7280.96 kbps, 6351.12 MB
encode time 0:38:02, CPU: 9.8%, GPU: 7.4%, VE: 99.2%, VD: 20.6%
frame type IDR    731
frame type I      731,  total size    70.98 MB
frame type P    43860,  total size  2313.22 MB
frame type B   130849,  total size  3966.92 MB

Edit: 1080p runs at 175fps stable with above settings, no big memory increase either...

[Mike-uk]

Originally Posted by cartman0208

I don't see a lot of memory growth, even on the 32bit version ... starts at 750MB and finishes with 850MB used by nvencc.exe

What I also noticed ... no matter what preset or rate factor ... the processing speed stays the same for a given resolution ... 4K is always around 75fps for me

Graphics driver was updated yesterday (451.48), but I'm not on Win10 May 2020 update yet, if that helps.


[CODE]
--------------------------------------------------------------------------------
U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc
--------------------------------------------------------------------------------

where is your --avhw to enable fully hardware encode ?

[cartman0208]

Forgot about it ...
Maybe it was auto detected?

from the help text...

--avhw          use libavformat + hw decode for input

Does it matter what is used for decoding?
Just started another run ... not much of a difference in speed

[jdobbs]

Originally Posted by gonca

Maybe a driver related issue, memory usage is constant at about 582 MB on my GTX1080ti
Driver is 445.87

I'm using Driver 451.48, dated June 24th, 2020 on a GTX1660

[jdobbs]

Originally Posted by cartman0208

I don't see a lot of memory growth, even on the 32bit version ... starts at 750MB and finishes with 850MB used by nvencc.exe

What I also noticed ... no matter what preset or rate factor ... the processing speed stays the same for a given resolution ... 4K is always around 75fps for me

Graphics driver was updated yesterday (451.48), but I'm not on Win10 May 2020 update yet, if that helps.

--------------------------------------------------------------------------------
U:\Temp\ALITA_BATTLE_ANGEL_temp\ALITA_BATTLE_ANGEL.hevc
--------------------------------------------------------------------------------
NVEncC (x86) 5.08 (r1585) by rigaya, Jul  1 2020 14:48:28 (VC 1900/Win/avx2)
OS Version     Windows 10 x64 (18363)
CPU            Intel Core i7-6800K @ 3.40GHz [TB: 3.80GHz] (6C/12T)
GPU            #0: GeForce GTX 1660 Ti (1536 cores, 1770 MHz)[2147483.64]
NVENC / CUDA   NVENC API 9.1, CUDA 11.0, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.265/HEVC, 3840x2160, 24000/1001 fps
Vpp Filters    copyDtoD
Output Info    H.265/HEVC main10 @ Level 5.1
               3840x2160p 1:1 23.976fps (24000/1001fps)
Encoder Preset quality
Rate Control   CQP  I:25  P:25  B:25
Lookahead      off
GOP length     240 frames
B frames       3 frames [ref mode: disabled]
Ref frames     3 frames, MultiRef L0:auto L1:auto
AQ             on
CU max / min   auto / auto
VUI            matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
MasteringDisp  G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(999.999975 0.000100)
MaxCLL/MaxFALL 1000/0
Others         mv:auto repeat-headers

encoded 175440 frames, 76.88 fps, 7280.96 kbps, 6351.12 MB
encode time 0:38:02, CPU: 9.8%, GPU: 7.4%, VE: 99.2%, VD: 20.6%
frame type IDR    731
frame type I      731,  total size    70.98 MB
frame type P    43860,  total size  2313.22 MB
frame type B   130849,  total size  3966.92 MB

Edit: 1080p runs at 175fps stable with above settings, no big memory increase either...

Part of that might be because you are using GOPs of 240 frames. That's 10 times larger than what it would need to be for a UHD-BD encode. The --repeat-headers would not be activated as often and would probably not hit the routine that causes the memory leak as often. Try adding --gop-len 24 and see what happens.

[Edit] Yeah. I just set the GOP length to 240 frames and it is barely growing at all... but, of course, that encode would fail to meet the UHD-BD standard. I would assume that it would also mean that as you attempt to fast forward, each picture step would jump 10 seconds instead of 1. The table used for these actions only logs IDR positions.

Originally Posted by gonca

Finished a remux and it seems to work FF, REW and CHP skips

I just went back and looked at your post again as well... and you are also using the default 240 frame GOP. That's probably why you're not seeing the issue as well. You are only having to insert the SPS/VPS/PPS once every 10 seconds instead of once per second. The headers are only inserted on the IDR frame that begins a GOP.

[jdobbs]

Here's a look at memory usage at the start of an encode and then after 35,000 frames. Considering that a 2 hour movie would be 172,627 frames, you can see how this can become an issue.

[Mike-uk]

am i right in seeing nvencc uses SDK version 9.1, i think there is a version 10 now ??, unfortunatly that would mean we need to wait for nvencc to compile sdk 10 versions

mayby ask him if he intends to compile a SDK 10 verstion ? mayby that will solve the memory leak

What's new in Video Codec SDK 10
New H.264 NVENC presets for improved flexibility and granularity
Finer control of rate control modes
Use-case Setting (also referred to as Tuning information)
NVDEC Support for GA100