BD Rebuilder Beta - Bug Reports Only

[jdobbs]

Originally Posted by CraigWally

I'm loving the speed gains on this latest version,
However
Is the "Quicker encode for extras" still working right

I've just tried to The Great Escape BluRay 1080,
and the extras have come out larger than the original with the main movie being compressed more than necessary

I'm just doing some further test, I'll post settings/ini etc later if necessary

EDIT: Hmmm, somethings not right here
If I choose "movie and menus / auto blank extras" - it chooses all the extras, rather than blanking them - seems to be just this disc?!

When "quick for extras" is checked, no sizing checks are done on the extras. They are just encoded at a fixed CRF rate and the feature is adjusted based on their total end sizing. The whole point is to get them done quickly because they don't (typically) add up to much anyway. The normal CRF (or CQM) used varies based on the target -- on a BD-25 CRF is 20. Normally that would make them smaller than the originals -- but if the originals were encoded at a very low bitrate it could, possibly, make them larger. If you think they are taking too much space, you can adjust the CRF/CQM value by adding QUICK_CRF=nn (nn being the new target CRF) to the BDREBUILDER.INI file (see the explanation in HIDDENOPTS.TXT). If you choose a value larger than the default they will get smaller and show less quality.

[cartman0208]

Here's the log

[07.24.20] BD Rebuilder v0.61.09
[09:09:06] Source:  LUCYUHD
  - Input BD size: 58,50 GB
  - Approximate total content: [01:42:30.060]
  - Target BD size: 47,85 GB
  - Windows Version: 6.2 [9200]
  - Quality: Highest (Very Slow), CQM
  - Decoding/Frame serving: NVENCC
  - Audio Settings: AC3=0 DTS=0 HD=1 Kbs=640
[09:09:09] PHASE ONE, Encoding
 - [09:09:09] Processing: VID_00000 (1 of 3)
 - [09:09:09] Extracting A/V streams [VID_00000]
   - Extracting video streams [VID_00000]
   - Extracting audio/subtitle streams [VID_00000]
 - [09:15:57] Reencoding video [VID_00000]
   - Source Video: HEVC, 3840x2160
   - Rate/Length: 23,976fps, 128.721 frames
 - [09:15:57] Performing CQM Prediction...
   - Analyzing 1,00 [1,00]
 - [09:16:50] Encoding using constant quality mode.
 - [10:08:10] Video Encode complete
 - [10:08:10] Processing audio tracks
   - Track 4353 (eng): Keeping original audio
   - Track 4354 (deu): Keeping original audio
 - [10:08:11] Multiplexing M2TS
 - [10:10:00] Processing: VID_00003 (2 of 3)
 - [10:10:00] Extracting A/V streams [VID_00003]
 - [10:10:04] Reencoding video [VID_00003]
 - [10:10:04] Keeping original video (no reencode)
 - [10:10:04] Processing audio tracks
 - [10:10:04] Multiplexing M2TS
 - [10:10:08] Processing: VID_00004 (3 of 3)
 - [10:10:08] Extracting A/V streams [VID_00004]
 - [10:10:12] Reencoding video [VID_00004]
 - [10:10:12] Keeping original video (no reencode)
 - [10:10:12] Processing audio tracks
   - Track 4352 (eng): Keeping original audio
 - [10:10:12] Multiplexing M2TS
[10:10:16]PHASE ONE complete
[10:10:16]PHASE TWO - Rebuild Started
 - [10:10:16] Rebuilding BD file Structure
[10:10:22] - Encode and Rebuild complete
[10:10:22] Writing BD structure to ISO file
 - ImgBurn completed successfully
[10:13:00] JOB: LUCYUHD finished.
[/QUOTE]

and the .inf
[QUOTE][Status]
LABEL=LUCYUHD
VERSION=v0.61.09
SOURCE_SIZE=62817573023
SOURCE_VIDEO_SIZE=62142130176
TARGET_SIZE=51380224000
REDUCTION=.815948552284787
RESIZE_1080=0
RESIZE_1440=0
AUDIO_TO_KEEP=deu;eng;ger;
KEEP_HD_AUDIO=-1
SUBS_TO_KEEP=all
BACKUP_MODE=0
MOVIEONLY_TYPE=0
USE_LAVF=0
INSTANCES=1
DGDECNV=-1
DGDECIM=0
FRIMSOURCE=0
FFMS2=0
SSIF_MODE=0
UHD_V3_MODE=0
QUICK=0
ENCODE_STEP=0
COMPLETED=3
REBUILD_COMPLETE=1
[00000]
AUDIO=011
PGS=111111111111
VIDEO2=0
V2MBRATE=0
M2TS_TARGET=50587878624
NSTART=26999955
NEND=268593181
NSIZE=36097812480
FLINK=0
MLINK=0
[00003]
AUDIO=
PGS=
VIDEO2=0
V2MBRATE=0
M2TS_TARGET=10026
USE_ORIGINAL=1
NSTART=27000000
NEND=27011261
NSIZE=12288
FLINK=0
MLINK=0
[00004]
USE_ORIGINAL=1
AUDIO=1
PGS=
VIDEO2=0
V2MBRATE=0
M2TS_TARGET=116892502
NSTART=27000000
NEND=28081080
NSIZE=143271936
FLINK=0
MLINK=0

Another thing I noticed ...
currently CQM is set, but LASTCMD states "--vbr 0 --vbr-quality <cqm-prediction-result>" ... shouldn't that be "--cqp <cqm-prediction-result>" ?

[jdobbs]

Originally Posted by cartman0208

Another thing I noticed ...
currently CQM is set, but LASTCMD states "--vbr 0 --vbr-quality <cqm-prediction-result>" ... shouldn't that be "--cqp <cqm-prediction-result>" ?

No. cqp would set a constant quantizer value (or set of quantizer values for I, P, and B frames). Constant Quality Mode (CQM) would use "--vbr 0 --vbr-quality <float>"

I guess it's possible that shrinking the original by such a low factor might result in a value of 1.00, especially if the original was purposely oversized to force the need for a dual-layer disc -- but I would think that would be extremely rare (although I have seen it before).

What happens if you select a 25GB target size?

Did you get a chance to look at the size of the sample file?

[cartman0208]

Originally Posted by jdobbs

Did you get a chance to look at the size of the sample file?

I'm currently on another encode (where everything looks fine, several numbers on prediction, the sample file is around 750MB) but will retry the one in question after
I should mention, the one I posted, was way undersized ... around 35GB

[jdobbs]

Originally Posted by cartman0208

I'm currently on another encode (where everything looks fine, several numbers on prediction, the sample file is around 750MB) but will retry the one in question after

Thanks.

In your INF file it looks like the output M2TS size is actually smaller than the target size, even with a CQM value of 1.00. That makes me wonder if the original actually was purposefully oversized. I've even seen M2TS files that were padded with zeros to make them larger. The one here is a pretty short film (about 90 minutes). Weird.

I would have thought, though, that BD-RB would have noticed (after extracting the HEVC elementary stream) that no reencoding was needed and would have kept the original (unless "force_encode" was asserted). It is probably possible to pad an elementary stream too, I guess (you just wouldn't use zeros). Curious.

[cartman0208]

But would a CQM value of 1.00 not result in a HUGE output file, since its almost lossless?

[edit]
Mediainfo Output on the sample file

General
ID                                       : 1 (0x1)
Complete name                            : U:\BD-Output\WORKFILES\00000.AVS.SMPL.m2ts
Format                                   : MPEG-TS
File size                                : 604 MiB
Duration                                 : 1 s 1 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 956 Mb/s
FileExtension_Invalid                    : ts m2t m2s m4t m4s tmf ts tp trp ty

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@High
HDR format                               : SMPTE ST 2086, HDR10 compatible
Codec ID                                 : 36
Duration                                 : 2 s 2 ms
Bit rate                                 : 4 711 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 2)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 23.690
Stream size                              : 1.10 GiB
Writing library                          : ATEME Titan KFE 3.7.3 (4.7.3.1003)
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0050 cd/m2, max: 1000 cd/m2
Maximum Content Light Level              : 1000 cd/m2
Maximum Frame-Average Light Level        : 400 cd/m2

Menu
ID                                       : 4096 (0x1000)
Menu ID                                  : 1 (0x1)
Duration                                 : 1 s 1 ms
List                                     : 256 (0x100) (HEVC)
Service name                             : Service01
Service provider                         : FFmpeg
Service type                             : digital television

That's one big Bits/(Pixel*Frame) value

[jdobbs]

Originally Posted by cartman0208

But would a CQM value of 1.00 not result in a HUGE output file, since its almost lossless?

You would think so.

By the way, the sample file is nothing more than sets of frames pulled intact from the original stream. So what you are seeing there is the original information.

The only way to know is to look at the command line in LASTCMD.TXT (before the next encode starts) and see if it was actually used for the encode.

[cartman0208]

Originally Posted by jdobbs

You would think so.

The only way to know is to look at the command line in LASTCMD.TXT (before the next encode starts) and see if it was actually used for the encode.

... --max-cll 1000,400 --qp-min 0 --vbr 0 --vbr-quality 1 --aq-temporal --keyfile "U:\BD-OUTPUT\WORKFILES\VID_00000.CHP"...

[edit]
Well well ... on target-size BD25 I get:

- [15:29:49] Performing CQM Prediction...
- Analyzing 18,05 20,00 20,50 [20,50]

[Sharc]

Originally Posted by cartman0208

That's one big Bits/(Pixel*Frame) value

No wonder when you look at the bitrate:

Bit rate : 4 711 Mb/s

[jdobbs]

Originally Posted by Sharc

No wonder when you look at the bitrate:

Wow. That is an illegally high bitrate, even for a UHD/BD. I have to believe MEDIAINFO is reporting it incorrectly. It could be due to the way in which BD-RB creates the sample file by pulling sets of frames in small groups and throwing them together as an M2TS for sample reencoding. The PTS values might look a little crazy to it, as they start over again with each I frame at the start of a grouping. It won't affect the decoding/reencoding for sampling -- but might look goofy as a whole when trying to analyze it.

[cartman0208]

Sorry to be annoying, but I found another thing:
On Alternate Output the Cropping-Borders-Feature seems broken:

"U:\BD_Rebuilder\tools\nvenc\nvencc.exe" --avs -i "U:\BD-OUTPUT\WORKFILES\VID_00002.AVS" --codec hevc --preset performance --profile main10 --output-depth 10 --repeat-headers --chromaloc 2 --colorprim bt2020 --transfer smpte2084 --colormatrix bt2020nc --master-display G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1) --max-cll 435,289 --qp-min 0 --vbr 0 --vbr-quality 25 --aq-temporal --keyfile "U:\BD-OUTPUT\WORKFILES\VID_00002.CHP" --aud --pic-struct --vbv-bufsize 45000 --max-bitrate 48000 --gop-len 24 -o "U:\BD-OUTPUT\WORKFILES\VID_00002.AVS.hevc"
--------------------------------------------------------------------------------
U:\BD-OUTPUT\WORKFILES\VID_00002.AVS.hevc
--------------------------------------------------------------------------------
avs: invalid clip.
avs: DirectShowSource: RenderFile, the filter graph manager won't talk to me
avs: (U:\BD-OUTPUT\WORKFILES\VID_00002.AVS, line 2)
failed to initialize file reader(s).
Failed to open input file.

[00003]
caption=MKV, HEVC, 1080p, Auto-AC3
aBitrate=640
aType=0
cType=1
vEncoder=1
vFormat=5
vKeyint=Auto
vBitrate=0

#Created by BD Rebuilder - v0.61.09
DirectshowSource("U:\FULLDISC\UHD_ASTERIX\BDMV\STREAM\00002.m2ts", fps=23.976, framecount=123617, audio=false)
Spline16Resize(1920,1080)
LoadPlugin("U:\BD_Rebuilder\tools\autocrop.dll")
Autocrop(mode=0, hMultOf=4)
ConvertToYV12().AssumeFPS(24000,1001)

[edit]
I made a list of different scenarios, where autocrop is working or not:

Source: h264 1080p, Target: h264 1080p, Decoder: DShow, Encoder: x26x - working
Source: h264 1080p, Target: h264 1080p, Decoder: DShow, Encoder: NVenc - failed
Source: h264 1080p, Target: h264 1080p, Decoder: DGdecNV, Encoder: x26x - failed
Source: h264 1080p, Target: h264 1080p, Decoder: DGdecNV, Encoder: NVenc - failed
Source: h264 1080p, Target: hevc 1080p, Decoder: DShow, Encoder: x26x - hanging
Source: h264 1080p, Target: hevc 1080p, Decoder: DShow, Encoder: NVenc - working
Source: h264 1080p, Target: hevc 1080p, Decoder: DGdecNV, Encoder: x26x - hanging
Source: h264 1080p, Target: hevc 1080p, Decoder: DGdecNV, Encoder: NVenc - failed
Source: hevc 2160p, Target: h264 1080p, Decoder: DShow, Encoder: x26x - failed
Source: hevc 2160p, Target: h264 1080p, Decoder: DShow, Encoder: NVenc - failed
Source: hevc 2160p, Target: h264 1080p, Decoder: DGdecNV, Encoder: x26x - failed
Source: hevc 2160p, Target: h264 1080p, Decoder: DGdecNV, Encoder: NVenc - failed
Source: hevc 2160p, Target: hevc 1080p, Decoder: DShow, Encoder: x26x - working
Source: hevc 2160p, Target: hevc 1080p, Decoder: DShow, Encoder: NVenc - failed
Source: hevc 2160p, Target: hevc 1080p, Decoder: DGdecNV, Encoder: x26x - working
Source: hevc 2160p, Target: hevc 1080p, Decoder: DGdecNV, Encoder: NVenc - failed
Source: hevc 2160p, Target: h264 2160p, Decoder: DShow, Encoder: x26x - failed
Source: hevc 2160p, Target: h264 2160p, Decoder: DShow, Encoder: NVenc - failed
Source: hevc 2160p, Target: h264 2160p, Decoder: DGdecNV, Encoder: x26x - failed
Source: hevc 2160p, Target: h264 2160p, Decoder: DGdecNV, Encoder: NVenc - failed
Source: hevc 2160p, Target: hevc 2160p, Decoder: DShow, Encoder: x26x - working
Source: hevc 2160p, Target: hevc 2160p, Decoder: DShow, Encoder: NVenc - failed
Source: hevc 2160p, Target: hevc 2160p, Decoder: DGdecNV, Encoder: x26x - working
Source: hevc 2160p, Target: hevc 2160p, Decoder: DGdecNV, Encoder: NVenc - failed

[MrVideo]

@cartman0208: Please use code wrapping instead of quote wrapping for data lists. That way long lists will get scrollbars and use the correct font. Thanks.

[cartman0208]

Originally Posted by MrVideo

@cartman0208: Please use code wrapping instead of quote wrapping for data lists. That way long lists will get scrollbars and use the correct font. Thanks.

Yep, I know ... but as you can see, no horizontal scrollbars ... at least not in the post itself, but in the browser ... and that kind of messes up the thread, imho.

For the next long list I'll use CODE, but else I would rather use QUOTE, if that is OK?

[MrVideo]

Originally Posted by cartman0208

Yep, I know ... but as you can see, no horizontal scrollbars.

I get around the long lines by adding CRs to those long lines, making two, or more, shorter lines. They do not have to stay long.

[BuddTX]

Originally Posted by cartman0208

Sorry to be annoying, but I found another thing:
On Alternate Output the Cropping-Borders-Feature seems broken:[/CODE]

Thank you for finding this issue Cartman, as this has been happening to me also.

I have just been testing each "session" with "UHD_LIMIT_ENCODE=1" before I create the full length MKV, so that I dont waste 2-4 days processing a video that will fail.

Oh, btw, I am only doing software encoding at this time, if that matters.

On the videos that fail, I create the MKV file with "cropping borders" unchecked, but with all other videos, I like having this feature checked.

[jdobbs]

Unfortunately AUTOCROP is simply an AVISYNTH PlugIn that I added to the BD-RB package. BD-RB just adds it to the AVISYNTH file when the box is checked. If it has an issue... there's really not much I can do other than maybe removing it as an option.

If you run into this problem, pretty much all I can recommend is to just use AVISYNTH filters to do the cropping manually.

[BuddTX]

Originally Posted by jdobbs

Unfortunately AUTOCROP is simply an AVISYNTH PlugIn that I added to the BD-RB package. BD-RB just adds it to the AVISYNTH file when the box is checked. If it has an issue... there's really not much I can do other than maybe removing it as an option.

If you run into this problem, pretty much all I can recommend is to just use AVISYNTH filters to do the cropping manually.

Thanks JDobbs. I have not had this problem of a job failing with Autocrop selected, but the job successfully completed when unchecking the AutoCrop button until relatively recently.

I also have been playing with newer versions of Avisynth. (Currently using AviSynthPlus 3.6.1)

Wonder if this plugin would work "better" or "more reliably"?
RoboCrop

https://forum.doom9.org/showthread.p...t=autocrop.dll

Version 1.13 has a release date of 2-7-2020

However, I do not see any place to download this dll.

In post 95 of the above link, "hello_hello" wrote "RoboCropped.avsi" and posted a link.

from the link above:
Plugin's for both Avisynth v2.5 and v2.6.

RoboCrop is an automatic cropping solution to crop black borders from video clips, loosely based on (but using no code from)
Autocrop by Glenn Bussell.

For the most part, you can just call it with RoboCrop(clip) and not bother with any other arguements, it is intended to be pretty
much autonomous in its decisions. You might however want to alter eg WMod and HMod if your encoder has special requirements, and
perhaps Laced if your source is likely to be fully progressive.

Borders are detected by sampling at Samples frames, at scanlines (h/v) using AverageLuma (RGB is converted to Luma Y at either TV or
PC levels, See Matrix). This sampling is done on all 4 sides of the frame.
If a scanline Average luma is below or equal to Thresh, then is considered possible black border, above considered possible image,
if Baffle [default=4] adjacent scanlines above Thresh, then it IS image.


Thanks JDobbs!

[BuddTX]

Also, maybe I should try using your "officially recommended" avisynth version, or maybe go back to 2.5.7 and see if that "fixes" the problem, the next time I find a video that fails with the autocrop selected, but is sucessful with autocrop unchecked.

[cartman0208]

I've dug a little further...
Looking at the software-only results

Source: h264 1080p, Target: h264 1080p, Decoder: DShow, Encoder: x26x - working
Source: h264 1080p, Target: hevc 1080p, Decoder: DShow, Encoder: x26x - hanging
Source: hevc 2160p, Target: h264 1080p, Decoder: DShow, Encoder: x26x - failed
Source: hevc 2160p, Target: hevc 1080p, Decoder: DShow, Encoder: x26x - working
Source: hevc 2160p, Target: h264 2160p, Decoder: DShow, Encoder: x26x - failed
Source: hevc 2160p, Target: hevc 2160p, Decoder: DShow, Encoder: x26x - working

that's not an issue of Avisynth or the Autocrop plugin ... the conversions between x264 and x265 content (and vice versa) seem to cause trouble ... the above results are the same with or without Autocrop.

[jdobbs]

Originally Posted by cartman0208

I've dug a little further...
Looking at the software-only results

Source: h264 1080p, Target: h264 1080p, Decoder: DShow, Encoder: x26x - working
Source: h264 1080p, Target: hevc 1080p, Decoder: DShow, Encoder: x26x - hanging
Source: hevc 2160p, Target: h264 1080p, Decoder: DShow, Encoder: x26x - failed
Source: hevc 2160p, Target: hevc 1080p, Decoder: DShow, Encoder: x26x - working
Source: hevc 2160p, Target: h264 2160p, Decoder: DShow, Encoder: x26x - failed
Source: hevc 2160p, Target: hevc 2160p, Decoder: DShow, Encoder: x26x - working

that's not an issue of Avisynth or the Autocrop plugin ... the conversions between x264 and x265 content (and vice versa) seem to cause trouble ... the above results are the same with or without Autocrop.

I don't think I understand what you are saying. Can you provide an description of what you are doing that results in a hang of a failure? A log maybe?