BD Rebuilder Beta - Bug Reports Only

[Dreamweaver2000a]

I apologize if this is not the right place to post this info, but I'm sure it will be moved if necessary. The purpose of the post is to pass on info I have found on another site that has been of great help to an admitted BD rebuilder neophyte (me). I could not find this info on Doom9 by doing a search.....but now maybe the next person looking will be able to find it....Anyway, BD rebuilder kept reporting that "failed video encode, aborted" when trying to do a backup on the movie "Green Zone". Although I was able to backup other BD discs before and after these failures, no matter what I did, the encode failed on "Green Zone"....that is until I found this info....."Most likely it is caused by VC-1 being disabled in FFDSHOW. Go into ffdshow configuration and under VC-1, change from "disabled" to "libavcodec" After I reconfigured the FFDSHOW.....my problem was resolved! Hope this info will help others as it did me! Thanks jdobbs for BD rebuilder!

[jdobbs]

Originally Posted by Dreamweaver2000a

I apologize if this is not the right place to post this info, but I'm sure it will be moved if necessary. The purpose of the post is to pass on info I have found on another site that has been of great help to an admitted BD rebuilder neophyte (me). I could not find this info on Doom9 by doing a search.....but now maybe the next person looking will be able to find it....Anyway, BD rebuilder kept reporting that "failed video encode, aborted" when trying to do a backup on the movie "Green Zone". Although I was able to backup other BD discs before and after these failures, no matter what I did, the encode failed on "Green Zone"....that is until I found this info....."Most likely it is caused by VC-1 being disabled in FFDSHOW. Go into ffdshow configuration and under VC-1, change from "disabled" to "libavcodec" After I reconfigured the FFDSHOW.....my problem was resolved! Hope this info will help others as it did me! Thanks jdobbs for BD rebuilder!

As I mentioned in the first post of this thread -- I'd highly recommend you use "wmv9" rather than "libavcodec". There are some VC-1 streams that will not decode using libavcodec.

[Carter]

On a Intel i7 1366 (X5650) Six core, 12 Thread with hyperthreading X264 only uses around 25%. I don't have a clue why. Any idea ?

[jdobbs]

Originally Posted by Carter

On a Intel i7 1366 (X5650) Six core, 12 Thread with hyperthreading X264 only uses around 25%. I don't have a clue why. Any idea ?

Pass 1 or pass 2? Sometimes on very fast processors the frame serving portion of the process (AVISYNTH & FFDSHOW) can't keep up with the processor as it is actually reencoding.

You might try using LAVF for frame serving and see how that works out (under the SETUP menu).

[bobrap]

Having a few sizing problems and didn't know whether to start a new new thread or post here. I tried doing Sorcerer's Apprentice last night and ended with a final size of 29.1 GB. Tried attaching log and inf. Are those the files you need to see? Thanks for any help.

[Chuckwagon]

Originally Posted by Carter

On a Intel i7 1366 (X5650) Six core, 12 Thread with hyperthreading X264 only uses around 25%. I don't have a clue why. Any idea ?

-UPDATE- Ignore what I had posted here previously.

I've seen the same behavior on my X980. I just re-started an encode, and it looks as though x264 isn't even using the HT part of the cores, so only 6 of the 12 available "threads" are in use, and as a result Windows is miscalculating the total utilization. The threads that are being used bounce around 80-90%, which is pretty much fully utilized. So it seems the reported % utilized gets skewed because of the HT threads. Performance monitor doesn't properly assess the utilization. You can ignore it.

[Chuckwagon]

Originally Posted by bobrap

Having a few sizing problems and didn't know whether to start a hew new thread or post here. I tried doing Sorcerer's Apprentice last night and ended with a final size of 29.1 GB. Tried attaching log and inf. Are those the files you need to see? Thanks for any help.

Your attachments haven't been approved yet, so let me ask, are you using Auto Blank? Are you manually adding blanked streams back into your list?

It may be unrelated, but I have noticed that if you look at the input size reported by BD-RB when it starts an encode, you will see that it does not update as you add or remove streams from the list when you use the Auto Blank feature and then manually add/remove streams. For example, I have just tested this with a 45GB BD, that reports an input size of 25.75GB if I have the blank threshold set to 3600. (Only the main movie gets selected.) If I manually go add back in all of the streams that got blanked, BD-RB still reports an input size of 25.75GB. But if I change the threshold to 600, the input size goes to 35.64GB, and even if I then manually blank all but the main movie, the input size stays at 35.64GB.

So, I'm assuming the calculation of the input size, and therefore how much compression will be needed, is taking place when the Auto Blanking takes place, and is not being updated as streams are manually added or removed. This results in over-compression for builds where streams are manually removed, and under-compression where streams are manually added. Or at least that's my theory.

[Carter]

but shouldnt use x264 even HT ? the utilised threads dont use 80 as well. Even when misscalculated with HT it shouldnt be shown as 25% i guess. i tried x264 HD Benchmark 4.0 and this one utilises every single of the 12 threads 100%

[Chuckwagon]

Originally Posted by Carter

but shouldnt use x264 even HT ? the utilised threads dont use 80 as well. Even when misscalculated with HT it shouldnt be shown as 25% i guess. i tried x264 HD Benchmark 4.0 and this one utilises every single of the 12 threads 100%

Keep in mind that the benchmark you used has the CPU priority set to high for x264, and threads set to Auto, which may change it's behavior from how JDobbs calls it. My point was that the windows reports cannot be trusted. Your system is probably running as it should for how the software is using it.

The high priority CPU setting certainly insures that not much else is likely to get done by any other programs while x264 is running, so maybe he doesn't fire it off that way to allow for other things to be going on. I don't know what CPU priority JDobbs has chosen, or if he sets the threads in some way other than auto, but I'm sure he can tell you. Maybe he has a way to alter that behavior that you could tweak.

-UPDATE-
After a little more playing around with both the x264_Benchmark_HD_v4.0 commands and the x264 command for BD-RB found in the LASTCMD.TXT file I think I may have narrowed down why x264 behaves the way it does. I removed the "--preset ultrafast" portion of the command found in LASTCMD.TXT and ran the command manually. It resulted in all 12 threads being utilized and nearly 100% utilization. So, I'm guessing there is something in that preset which causes x264 to only use 6 threads instead of 12, and not to use them at full bore 100%.

Also, you can change the preset, or remove it, which will result in more threads being used, and in them having higher utilization, but it doesn't result in faster encodes. So it would seem the answer to the problem is "x264 doesn't use more threads nor a higher CPU utilization than it does because it doesn't need to." You aren't leaving any CPU power lying on the table, so to speak. It uses what it needs to.

[setarip_old]

@jdobbs
Both "How to Train Your Dragon" and "Constantine" utilize PIP, but use E-AC3 audio instead of DTS Express for the PIP audiostream and both BD-RB backup discs exhibit the same wretched behavior on my SONY S360 as do all DTS Express discs thusfar.

Additionally, opening the ORIGINAL disc's movie .M2TS in tsMuxer discloses that the E-AC3 audiostream only is listed as the very last item, after all PGS streams - while the BD-RB backup lists it in the conventional last of the audiostreams, and before all PGS streams. Also, tsMuxer will not allow me to change the relative position of the E-AC3 audiostream to last place after all PGS streams.

Since tsMuxer cannot process (and doesn't display) DTS Express audiostreams, I cannot determine whether they also are ORIGINALLY placed after all PGS streams. If they are, it would be my guess that this placement is critical to the S360s processing of both types, if not all types, of PIP-related audio.
Finally (I believe I mentioned this previously about DTS Express discs), the ORIGINAL disc lists the secondary video as "Profile High 3.2", whereas the BD-RB backup lists it as "Profile High 4.1" (same as it lists for primary video for both ORIGINAL and BD-RB backup).

[worknstiff]

@ bobrap RE: Having a few sizing problems and didn't know whether to start a hew new thread or post here. I tried doing Sorcerer's Apprentice last night and ended with a final size of 29.1 GB

All you have to do is run BD_RB in "Movie and Menus" mode and use 50,000 in the setup for custom size. After you select what you want to keep or blank let it run (only takes about 20 minutes or so since it keeps all the original video intact) untill it finishes creating a bluray with menu intact. Now feed this to BD_RB and set your custom size for a "FULL MODE BACKUP" to 24,000 or 24,250 if you want to push it, and you will get a really accurately sized finished encode. goodluck, worknstiff

[jdobbs]

Originally Posted by Chuckwagon

Keep in mind that the benchmark you used has the CPU priority set to high for x264, and threads set to Auto, which may change it's behavior from how JDobbs calls it. My point was that the windows reports cannot be trusted. Your system is probably running as it should for how the software is using it.

The high priority CPU setting certainly insures that not much else is likely to get done by any other programs while x264 is running, so maybe he doesn't fire it off that way to allow for other things to be going on. I don't know what CPU priority JDobbs has chosen, or if he sets the threads in some way other than auto, but I'm sure he can tell you. Maybe he has a way to alter that behavior that you could tweak.

-UPDATE-
After a little more playing around with both the x264_Benchmark_HD_v4.0 commands and the x264 command for BD-RB found in the LASTCMD.TXT file I think I may have narrowed down why x264 behaves the way it does. I removed the "--preset ultrafast" portion of the command found in LASTCMD.TXT and ran the command manually. It resulted in all 12 threads being utilized and nearly 100% utilization. So, I'm guessing there is something in that preset which causes x264 to only use 6 threads instead of 12, and not to use them at full bore 100%.

Also, you can change the preset, or remove it, which will result in more threads being used, and in them having higher utilization, but it doesn't result in faster encodes. So it would seem the answer to the problem is "x264 doesn't use more threads nor a higher CPU utilization than it does because it doesn't need to." You aren't leaving any CPU power lying on the table, so to speak. It uses what it needs to.

That just means that by removing that mode you've added additional work for the processors to do --but it won't likely finish any faster, and you probably won't notice a quality difference in the output. So even though you're using more processor, I'm not sure you're really accomplishing anything. As I said before, the limiting factor is the decoding and frame serving.

A better way to do what you're doing would be to turn off "Automatic Quality" settings and select "High Quality" as a permanent setting.

Did you try the LAVF setting I suggested?

[jdobbs]

Originally Posted by setarip_old

@jdobbs
Both "How to Train Your Dragon" and "Constantine" utilize PIP, but use E-AC3 audio instead of DTS Express for the PIP audiostream and both BD-RB backup discs exhibit the same wretched behavior on my SONY S360 as do all DTS Express discs thusfar.

Additionally, opening the ORIGINAL disc's movie .M2TS in tsMuxer discloses that the E-AC3 audiostream only is listed as the very last item, after all PGS streams - while the BD-RB backup lists it in the conventional last of the audiostreams, and before all PGS streams. Also, tsMuxer will not allow me to change the relative position of the E-AC3 audiostream to last place after all PGS streams.

Since tsMuxer cannot process (and doesn't display) DTS Express audiostreams, I cannot determine whether they also are ORIGINALLY placed after all PGS streams. If they are, it would be my guess that this placement is critical to the S360s processing of both types, if not all types, of PIP-related audio.
Finally (I believe I mentioned this previously about DTS Express discs), the ORIGINAL disc lists the secondary video as "Profile High 3.2", whereas the BD-RB backup lists it as "Profile High 4.1" (same as it lists for primary video for both ORIGINAL and BD-RB backup).

Moving the audio doesn't make a difference. I've tried it before.

I'll have to look at the profile issue -- BD-RB is supposed to use Level 3.2 for secondary video.

[Chuckwagon]

Originally Posted by jdobbs

That just means that by removing that mode you've added additional work for the processors to do --but it won't likely finish any faster, and you probably won't notice a quality difference in the output. So even though you're using more processor, I'm not sure you're really accomplishing anything. As I said before, the limiting factor is the decoding and frame serving.

A better way to do what you're doing would be to turn off "Automatic Quality" settings and select "High Quality" as a permanent setting.

Did you try the LAVF setting I suggested?

I agree, and that's what I was trying to say, though probably not clearly. There isn't really a problem, it just LOOKS like there might be if you are looking at the Performance Monitor because it seems you aren't using all the CPU power available. But x264 just doesn't NEED any more CPU power to accomplish what is being asked of it. For the given quality level, changing the preset WILL cause more horsepower to be needed, and you will see that in the Performance Monitor, but the end result is an encode that is essentially the same size but takes longer to create. So there is no problem with x264 not using all of the available CPU power.

What follows is the results of the test I ran. I tested this using both x264-64 with LAVF and x264-32 without LAVF, with and without --preset ultrafast. Using an input file of size 990,244 KB (one of the files from my last encode) and encoding it with the last command BD-RB issued on my last encode of a disc, only changing the use of LAVF or --preset ultrafast. The bitrate used was 15649 (which is what was last used on my most recent encode) Here is what resulted:

x264-64 with LAVF and with --preset ultrafast
Encoded 10809 frames, 58.54 fps, 15241.66 kb/s
Resulting file size - 838,785 KB

x264-64 with LAVF and no preset
Encoded 10809 frames, 22.88 fps, 15090.14 kb/s
Resulting file size - 830,447 KB

x264-32 without LAVF with --preset ultrafast
Encoded 10809 frames, 56.30 fps, 15241.82 kb/s
Resulting file size - 838,794 KB

x264-32 without LAVF and no preset
Encoded 10809 frames, 21.39 fps, 15090.21 kb/s
Resulting file size - 830,451 KB

With or without LAVF, the CPU utilization was about the same; around 25% or so and only 6 threads active when using --preset ultrafast, and all 12 threads active and maxed out when not using --preset ultrafast.

So it is possible to MAKE the x264 use more CPU power, but there isn't a NEED unless you want slower encodes for the given quality you have selected. There is no need to worry about what the Performance Monitor says about your CPU utilization while x264 is encoding, x264 is doing what it was asked and using what it needs to do so.

So there's no bug in BD-RB related to this perceived issue.

[jdobbs]

If it is disc bound (which is very likely), an SSD would probably really fly with your setup.

It might also be possible to modify BD Rebuilder to check on usage and do multiprocessing in addition to multithreading -- like encoding more than one video file at a time, or encoding audio concurrent with the video. More-and-more people are getting high-speed 6 and 8 core machines, and it would be nice to be able to take advantage of them.

[Ch3vr0n]

if you could pull that off and make it work on a simple quad core like my Q9550 that would be very impressive

[jdobbs]

Originally Posted by Ch3vr0n

if you could pull that off and make it work on a simple quad core like my Q9550 that would be very impressive

My quad core (Phenom X4) runs at around 96% in pass 2 and sometimes hits 100%. Pass 1 rarely gets much more than 60%.

[Chuckwagon]

Originally Posted by jdobbs

If it is disc bound (which is very likely), an SSD would probably really fly with your setup.

It might also be possible to modify BD Rebuilder to check on usage and do multiprocessing in addition to multithreading -- like encoding more than one video file at a time, or encoding audio concurrent with the video. More-and-more people are getting high-speed 6 and 8 core machines, and it would be nice to be able to take advantage of them.

I ran the tests from my SSD RAID 0 array, consisting of 3 Intel X25E 32GB drives. It's quite fast. But I haven't noticed it making much of a difference in encoding speed vs the mirrored 1GB standard HDDs I normally use. And I too think it would be great if you were able to multi-task rebuilds in some way, though I'm not unhappy with how it works now.

On to another issue dealing with a different problem, I just completed verifying that there does seem to be a problem with BD-RB setting the bitrate for encoding when streams are manually added or removed.

I allowed Auto Blank to remove all but the main movie from a rebuild, and BD-RB reports an input size of 25.75 GB and sets the bit rate to 19632, and the resulting ISO ends up being 22,922,240 KB. When I set the Threshold to 600 and then manually remove the rest of the streams so that only the main movie is left BD-RB reports an input size of 35.64 GB and the bitrate gets set to 12059, and the resulting ISO is 16,628,736 KB. The rebuilds are using the same streams each time, the material being encoded hasn't changed, only the way in which it was selected was changed.

So it seems that input size and bitrate are not being recalculated after manual selections are made, and as a result rebuilds are ending up either too small or too large depending on if streams were added or removed.

[Sharc]

Since a while (OS update? BD-RB update?) wavi.exe crashes at the end of its task instead of terminating properly. I can then terminate the application (wavi.exe) manually and BD-RB continues as expected, and the final result is o.k. Nothing is reported in BD-RB's logfile, so basically no problem with the exception that I have to intervene manually in order to let DB-RB to continue its task. I never had any issues with this until recently

[jdobbs]

Originally Posted by Chuckwagon

I ran the tests from my SSD RAID 0 array, consisting of 3 Intel X25E 32GB drives. It's quite fast. But I haven't noticed it making much of a difference in encoding speed vs the mirrored 1GB standard HDDs I normally use. And I too think it would be great if you were able to multi-task rebuilds in some way, though I'm not unhappy with how it works now.

On to another issue dealing with a different problem, I just completed verifying that there does seem to be a problem with BD-RB setting the bitrate for encoding when streams are manually added or removed.

I allowed Auto Blank to remove all but the main movie from a rebuild, and BD-RB reports an input size of 25.75 GB and sets the bit rate to 19632, and the resulting ISO ends up being 22,922,240 KB. When I set the Threshold to 600 and then manually remove the rest of the streams so that only the main movie is left BD-RB reports an input size of 35.64 GB and the bitrate gets set to 12059, and the resulting ISO is 16,628,736 KB. The rebuilds are using the same streams each time, the material being encoded hasn't changed, only the way in which it was selected was changed.

So it seems that input size and bitrate are not being recalculated after manual selections are made, and as a result rebuilds are ending up either too small or too large depending on if streams were added or removed.

When are you making the manual selections? If you do it after some encoding have been completed and you resume, I can see how it would be off because the size of the completed items wouldn't match the new calculations.

Any chance that's it? It sounds like it -- and if that's the case it makes perfect sense, the only thing I could do is start the entire encode over when you change your selections. You can't change the size of a file that has already been encoded...

Was the SSD the source (where the original BDMV and CERTIFICATE folders reside). If disc I/O is the limiting factor, that's where you'd have to use it to see a difference.