Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > DVD & BD Rebuilder

Reply
 
Thread Tools Search this Thread Display Modes
Old 30th January 2012, 15:57   #14101  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by Ch3vr0n View Post
I'm curious if there will be a time difference between LAVF & multiprocessing.
If it works out well I'm hoping to add a module to split for LAVF as well. I'm also doing some testing with X264's internal resizing too -- that way LAVF mode could handle some of the ALTERNATE formats that require resizing -- and even use the 64 bit routines to do it. With no filters, LAVF mode should be able to do pretty much anything as long as you aren't trying to process a VC-1 interlaced source.

I think the program has gotten to a point where it is stable (for the most part, as long as the helper apps don't get hosed by some other process) -- so I'm now looking at ways to improve speeds.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 30th January 2012 at 16:03.
jdobbs is offline   Reply With Quote
Old 30th January 2012, 16:19   #14102  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Quote:
Originally Posted by jdobbs View Post
Unfortunately I can't change the number of X264 instances between passes 1 and 2 -- because X264 needs them to be identical. It would also require another split -- which would also add time.
Very likely a stupid question as I'm not a programmer... So it's not possible to set instances to 4 for pass 1, and then during pass 2 drop instances to 2 while tellling the instances to process parts sequentially? For example, instance #1 begins processing part 1 and #2 begins with part 2. Once #1 is done with part 1 it begins processing part 3 and #2 moves to part 4 once its done with part 2.

Or how about starting 4 instances but only giving 2 instances parts to process? Too much of a kludge?

I suppose further means of improving speed would include moving to 64-bit apps and multithreading, like Avisynth-MT.

Last edited by colinhunt; 30th January 2012 at 16:46.
colinhunt is offline   Reply With Quote
Old 30th January 2012, 16:22   #14103  |  Link
Ch3vr0n
Registered User
 
Join Date: Jan 2009
Posts: 1,368
multiprocessing LAVF would be cool to see happening. multiprocessing LAFV x264-64 would be even better. That would definitely give a major speed increase. 80% into the current build and ram usage per instance is holding steady arround 800MB each, total cpu usage at 100%, total ram use at 5.2GB

** edit: recoding for main m2ts complete: job time including splitting and rejoining multiparts 93minutes**

i did notice some weird behavior. LAVF was disabled in the setup. yet x264-64 was launched and cpu usage was approx 100% for 1 pass encodes on the secondairy files. According to the release notes from September 5th, 2010 x264-64 should only be launched when LAVF is selected on a 64bit os. My OS is 64bit but lavf was not selected.

Code:
[Status]
LABEL=FRIENDS_WITH_BENEFITS
VERSION=v0.40.01 (beta)
SOURCE_SIZE=36453851971
SOURCE_VIDEO_SIZE=35969003520
TARGET_SIZE=24641536000
REDUCTION=.671597352858776
RESIZE_1080=0
AUDIO_TO_KEEP=dut;eng;nld;und;
KEEP_HD_AUDIO=-1
SUBS_TO_KEEP=dut;eng;nld;und;
BACKUP_MODE=0
MOVIEONLY_TYPE=0
USE_LAVF=0
MULTIPROCESS=3
QUICK=0
ENCODE_STEP=1.5
COMPLETED=6
** edit 2: well this is even weider. Now x264-64 doenst get loaded on VID_00102. On some it does others it doesnt. Weird"
** edit 3: Iso burning completed at 17.53. Job started on 14.45. So 2hrs for an 125video encoding and 3 blanks" Total size output 23.364.096kb aka 22.2GB
Now redoing same job with LAVF enabled. To check time diff

Job Restarted at 18.00.00hrs

Last edited by Ch3vr0n; 30th January 2012 at 18:01.
Ch3vr0n is offline   Reply With Quote
Old 30th January 2012, 17:46   #14104  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by Ch3vr0n View Post
multiprocessing LAVF would be cool to see happening. multiprocessing LAFV x264-64 would be even better. That would definitely give a major speed increase. 80% into the current build and ram usage per instance is holding steady arround 800MB each, total cpu usage at 100%, total ram use at 5.2GB

** edit: recoding for main m2ts complete: job time including splitting and rejoining multiparts 93minutes**

i did notice some weird behavior. LAVF was disabled in the setup. yet x264-64 was launched and cpu usage was approx 100% for 1 pass encodes on the secondairy files. According to the release notes from September 5th, 2010 x264-64 should only be launched when LAVF is selected on a 64bit os. My OS is 64bit but lavf was not selected.

** edit 2: well this is even weider. Now x264-64 doenst get loaded on VID_00102. On some it does others it doesnt. Weird"
X264-64 might also be loaded for segments that have ATCDelta set. That was implemented because of a glitch in certain instances related to framecounts processed on multipart sources. I'll look at it and see if it is still necessary with the splitting...
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 30th January 2012 at 17:50.
jdobbs is offline   Reply With Quote
Old 30th January 2012, 18:17   #14105  |  Link
varekai
Registered User
 
varekai's Avatar
 
Join Date: Jul 2006
Posts: 528
Thought I'd share this, I'm a not-so-settings-savvy-person.
I just fired up BDRB like I use to do.
Usually I select ENCODE_QUALITY=4 and let it run over-nite.
But for this test I lowered the quality.

The new version 0.40.01 is a lot faster!
Code:
-----------------------
[11:49:38] BD Rebuilder v0.39.07 (beta)
  - Source:  xxx_xxx_xxx_x
  - Input BD size: 43,16 GB
  - Approximate total content: [02:48:21.091]
  - Target BD size: 23,05 GB
  - Windows Version: 6.1 [7601]
  - MOVIE-ONLY mode enabled
  - Quality: Good (Very Fast), Two Pass
  - Audio Settings: AC3=0 DTS=0 HD=1 Kbs=640
[11:49:38] PHASE ONE, Encoding
 - [11:49:38] Processing: VID_00050 (1 of 1)
 - [11:49:38] Extracting A/V streams [VID_00050]
 - [11:56:03] Reencoding video [VID_00050]
   - Source Video: MPEG-4 (AVC), 1920x1080
   - Rate/Length: 23,976fps, 242*184 frames
   - Bitrate: 13*606 Kbs
 - [11:56:03] Reencoding: VID_00050, Pass 1 of 2
 - [13:29:56] Reencoding: VID_00050, Pass 2 of 2
 - [15:14:36] Video Encode complete
 - [15:14:37] Processing audio tracks
   - Track 4352 (xxx): Keeping original audio
 - [15:14:37] Multiplexing M2TS
[15:20:30]PHASE ONE complete
[15:20:30]PHASE TWO - Rebuild Started
 - [15:20:30] Rebuilding BD file Structure
[15:29:24] - Encode and Rebuild complete
[15:29:24]JOB: xxx_xxx_xxx_x finished.
-----------------------

-----------------------
[15:38:40] BD Rebuilder v0.40.01 (beta)
  - Source:  xxx_xxx_xxx_x
  - Input BD size: 43,16 GB
  - Approximate total content: [02:48:21.091]
  - Target BD size: 23,05 GB
  - Windows Version: 6.1 [7601]
  - MOVIE-ONLY mode enabled
  - Quality: Good (Very Fast), Two Pass
  - Audio Settings: AC3=0 DTS=0 HD=1 Kbs=640
[15:38:40] PHASE ONE, Encoding
 - [15:38:40] Processing: VID_00050 (1 of 1)
 - [15:38:40] Extracting A/V streams [VID_00050]
 - [15:45:13] Reencoding video [VID_00050]
   - Source Video: MPEG-4 (AVC), 1920x1080
   - Rate/Length: 23,976fps, 242*184 frames
   - Bitrate: 13*606 Kbs
 - [15:45:13] Reencoding: VID_00050, Pass 1 of 2
 - [16:29:37] Reencoding: VID_00050, Pass 2 of 2
 - [17:13:58] Video Encode complete
 - [17:13:59] Processing audio tracks
   - Track 4352 (xxx): Keeping original audio
 - [17:13:59] Multiplexing M2TS
[17:20:06]PHASE ONE complete
[17:20:06]PHASE TWO - Rebuild Started
 - [17:20:06] Rebuilding BD file Structure
[17:29:10] - Encode and Rebuild complete
[17:29:10] JOB: xxx_xxx_xxx_x finished.
-----------------------
varekai is offline   Reply With Quote
Old 30th January 2012, 18:24   #14106  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by varekai View Post
Thought I'd share this, I'm a not-so-settings-savvy-person.
I just fired up BDRB like I use to do.
Usually I select ENCODE_QUALITY=4 and let it run over-nite.
But for this test I lowered the quality.

The new version 0.40.01 is a lot faster!
Did you have MULTIPROCESS set?
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 30th January 2012, 18:32   #14107  |  Link
varekai
Registered User
 
varekai's Avatar
 
Join Date: Jul 2006
Posts: 528
Quote:
Originally Posted by jdobbs View Post
Did you have MULTIPROCESS set?
No, not set.
I havn't looked into the HIDDENOPTS.TXT until now.
Will read more about it on your site.
I can do some more testings if you like?
varekai is offline   Reply With Quote
Old 30th January 2012, 18:47   #14108  |  Link
Rumbah
Registered User
 
Join Date: Mar 2003
Posts: 480
The new x264 is much faster for me, too. 10-20% without any setting changed in some test encodes of mine (not tested with bdrb)
Rumbah is offline   Reply With Quote
Old 30th January 2012, 19:38   #14109  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Now that one of the three x264 instances finished its job and the process is running two instances only, FPS went from ~17 to ~18.5 (now actually 19.00, whee! ... less than 10% to go and fps is now 19.50.). And while total CPU usage is still at 100%, the PC is much more responsive to use than it was when 3 instances were running.

It will be interesting to see how many instances BD-RB will launch when I set multiprocess to automatic (i.e. 1). On my PC 4 instances is optimal for Pass 1 (as extrapolated from current data) but 2 instances works best for Pass 2.

Last edited by colinhunt; 30th January 2012 at 20:08.
colinhunt is offline   Reply With Quote
Old 30th January 2012, 20:32   #14110  |  Link
Ch3vr0n
Registered User
 
Join Date: Jan 2009
Posts: 1,368
@Jdobbs:

2nd rebuild of Friends with Benefits completed, LAVF enabled

Job 2: LAVF (multithread enabled, though according to you not used in LAVF situations)
Job Start: 18.00hrs
Job completion: 20.26hrs (including iso burning)
Total duration: approx 2.5hrs

Job 1: multithread enabled
Job Start: 14.45
Job Completion: 17.53 (including iso burning)
Total Duration: approx 3hrs 10 minutes

Total timeLOSS with multi thread (including splitting & rejoining) 40 minutes unless my math is off

I'd say that's a pretty significant timeloss for an experimental feature.

Last edited by Ch3vr0n; 30th January 2012 at 20:38.
Ch3vr0n is offline   Reply With Quote
Old 30th January 2012, 20:45   #14111  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
BD-RB just finished Pass 2 of a 290,138 frame main movie. Based on experience I can estimate previous versions of BD-RB would have taken approx. 400-450 minutes to process that Pass 2. This new version of BD-RB ran Pass 2 in 296 minutes with multiprocess enabled... and it would have been even faster if it had been running only 2 instances of x264 from the start instead of 3. Pass 1 took 100 minutes and I'm pretty certain earlier versions would have taken at least twice as long, perhaps more.

I'd say the total time saved is something like 3-4 hours. You know what that is? Awesome, that's what.
colinhunt is offline   Reply With Quote
Old 30th January 2012, 20:49   #14112  |  Link
soneca
Registered User
 
Join Date: Feb 2008
Location: Brazil
Posts: 753
1 instance

- [15:33:42] Reencoding: VID_00000, Pass 1 of 2
- [15:58:33] Reencoding: VID_00000, Pass 2 of 2
- [17:18:10] Video Encode complete

3 instances
ram max pass 1 ~ 4,05GB
ram max pass 2 ~ 3,65GB
- [13:44:10] Reencoding: VID_00000, Pass 1 of 2
- [14:03:00] Reencoding: VID_00000, Pass 2 of 2
- [15:22:17] Video Encode complete

4 instances
ram max pass 1 ~ 5,06GB
ram max pass 2 ~ 4,5GB
- [10:40:58] Reencoding: VID_00000, Pass 1 of 2
- [10:59:39] Reencoding: VID_00000, Pass 2 of 2
- [12:19:40] Video Encode complete

I believe that using two instances in my case would be a good balance between speed on the first pass and consumption of ram.
soneca is offline   Reply With Quote
Old 30th January 2012, 21:04   #14113  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Running the same backup again on v0.39.07 to see what sort of FPS figures I get.

Source: AVC/480i (DEINT processing enabled)
PASS 1
v0.39.07 - 179 fps
v0.40.01 - 381 fps (3 instances)

PASS2
v0.39.07 - 100 fps
v0.40.01 - 117 fps (3 instances)

from ~75% completion onwards:
v0.39.07 - 92 fps
v0.40.01 - 133 fps (2 instances)

Source: AVC/1080p23.976
PASS 1
v0.39.07 - 30 fps (...and dropping)
v0.40.01 - 63 fps (3 instances)

PASS 2
at 10% completion:
v0.39.07 - 13 fps (higher than expected!)
v0.40.01 - 17 fps (3 instances)


I user bd-rebuilder.log to compare how long it took for 0.39.07 and 0.40.01 to make two passes of the same main movie file using the exact same settings.

v0.40.01: 6 hours 36 minutes
v0.39.07: 7 hours 54 minutes

Despite higher fps figures especially for Pass 1, the difference in total time is not quite as big as I originally estimated. I was expecting to see 11-12 fps for Pass 2 from v0.39.07 but the disc I picked for the test was "easier" to re-encode than I thought it would be.

Last edited by colinhunt; 31st January 2012 at 10:01.
colinhunt is offline   Reply With Quote
Old 30th January 2012, 21:15   #14114  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by Ch3vr0n View Post
@Jdobbs:

2nd rebuild of Friends with Benefits completed, LAVF enabled

Job 2: LAVF (multithread enabled, though according to you not used in LAVF situations)
Job Start: 18.00hrs
Job completion: 20.26hrs (including iso burning)
Total duration: approx 2.5hrs

Job 1: multithread enabled
Job Start: 14.45
Job Completion: 17.53 (including iso burning)
Total Duration: approx 3hrs 10 minutes

Total timeLOSS with multi thread (including splitting & rejoining) 40 minutes unless my math is off

I'd say that's a pretty significant timeloss for an experimental feature.
Gotta be something wrong there. At the most you should lose no more than the time associated with splitting... and in most cases that would be 5-10 minutes. The only other possibility might be if you are running short on memory.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 30th January 2012, 21:17   #14115  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by varekai View Post
No, not set.
I havn't looked into the HIDDENOPTS.TXT until now.
Will read more about it on your site.
I can do some more testings if you like?
If you're getting improvements without MULTIPROCESS set I would have to guess the improvement is in the new version of X264. Have you checked the output and it looks good/normal?
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 30th January 2012, 21:20   #14116  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,973
Quote:
Originally Posted by Rumbah View Post
The new x264 is much faster for me, too. 10-20% without any setting changed in some test encodes of mine (not tested with bdrb)
That's good to know. I'll go back through the changelog of X264 and see where it could pick up that much. 10-20% seems likea lot.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 30th January 2012, 21:40   #14117  |  Link
soneca
Registered User
 
Join Date: Feb 2008
Location: Brazil
Posts: 753
Quote:
The new x264 is much faster for me, too. 10-20% without any setting changed in some test encodes of mine (not tested with bdrb)
I got a gain of ~12% using the 64bit version and preset slower.
soneca is offline   Reply With Quote
Old 30th January 2012, 21:47   #14118  |  Link
Rumbah
Registered User
 
Join Date: Mar 2003
Posts: 480
Quote:
Originally Posted by jdobbs View Post
That's good to know. I'll go back through the changelog of X264 and see where it could pick up that much. 10-20% seems likea lot.
There were CABAC Trellis asm optimizations. Must have been the last commits.
Rumbah is offline   Reply With Quote
Old 30th January 2012, 22:06   #14119  |  Link
Ch3vr0n
Registered User
 
Join Date: Jan 2009
Posts: 1,368
Quote:
Originally Posted by jdobbs View Post
Gotta be something wrong there. At the most you should lose no more than the time associated with splitting... and in most cases that would be 5-10 minutes. The only other possibility might be if you are running short on memory.
Doubt it, never ran above 5.5GB. Got 8 gig installed. I'll run another encode tomorrow. Its a bit too late now.
Ch3vr0n is offline   Reply With Quote
Old 30th January 2012, 22:47   #14120  |  Link
dfsooner
Registered User
 
Join Date: Nov 2009
Location: North Las Vegas, NV
Posts: 112
x264-64 and # x264 threads

I set multiprocess=3 but I'm only seeing one instance of x264 and it is 32-bit. What do I need to do to use multiple instances of 64-bit x264?

I'm seeing the desired increase in CPU usage but only modest memory usage increase.

Intel i7 2600K - 8GB Windows 7 x64.

Last edited by dfsooner; 30th January 2012 at 22:54.
dfsooner is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:49.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.