Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th October 2019, 19:31   #17541  |  Link
slalom
Registered User
 
slalom's Avatar
 
Join Date: Jan 2010
Posts: 456
Quote:
Originally Posted by Pauly Dunne View Post
So as great as DE is, there can be a LOT of time that the servers aren't doing anything..so if there was a way to have RB to at least start the next job, once the chunks of the previous job were completed, that could save a reasonable amount of time, and power.
But Temp disk is being used for the previous job, you will slow it down if the program starts a new job simultaneously
__________________
E5 2697 v2 @ 3.0GHz on P9X79 Deluxe 24GB
Xeon E5-2680 v2 @ 3.1GHz 16GB
Sony Vaio VPC-F13Z1E/B
slalom is offline   Reply With Quote
Old 17th October 2019, 19:58   #17542  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 560
Quote:
Originally Posted by slalom View Post
But Temp disk is being used for the previous job, you will slow it down if the program starts a new job simultaneously
Good point. If you're muxing on the temp drive while trying to transfer the video to the job folder at the same time that would only compound things.
You'd have to pre-load the video for the next job once the current job starts to get around that but still not starting while the last job is muxing...tricky timing.

Last edited by byteshare; 17th October 2019 at 20:01.
byteshare is offline   Reply With Quote
Old 17th October 2019, 23:46   #17543  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by byteshare View Post
Good point. If you're muxing on the temp drive while trying to transfer the video to the job folder at the same time that would only compound things.
You'd have to pre-load the video for the next job once the current job starts to get around that but still not starting while the last job is muxing...tricky timing.
You're probably correct, but at least while the muxing is being done, the next job is at least starting it's process, which could save some time & resource's.

Even if the muxing & copying was a little slower, you're not having to wait for one job to be completed.

But here's an idea, what if there was a second temp drive, and you allocated which temp drive to be used, when creating new jobs. (wouldn't work too well with batching)
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 18th October 2019, 03:06   #17544  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 560
Quote:
Originally Posted by Pauly Dunne View Post
You're probably correct, but at least while the muxing is being done, the next job is at least starting it's process, which could save some time & resource's.
Even if the muxing & copying was a little slower, you're not having to wait for one job to be completed.
That is kind of the point, it would slow it down enough that your next job would actually be encoding later than the way it works now. Yes, technically the job would "start" sooner but the time for the video to copy to the temp drive would take longer with the last job muxing at the same time.
The only way it would be "faster" is if once jobs are started the next job loads the video while the other job is still encoding, then muxing would happen and the next job would start, then while that job is still going but after the last muxing is done the next-next job would start copying the video...this only really works if your jobs are both small enough/slow enough encodes that the timing can work...like I was saying, it would be tricky and wouldn't be helpful in many cases.
...Ideally if some symbolic link to the original file could be used for the sharing (in DE) so that you wouldn't have to even copy a file, then it would be much simpler/faster to start the next job while the last job is muxing.
Quote:
But here's an idea, what if there was a second temp drive, and you allocated which temp drive to be used, when creating new jobs. (wouldn't work too well with batching)
Wouldn't work for a lot of people I'd guess. Wouldn't work for me. I'd have to get a 3rd drive.

Last edited by byteshare; 18th October 2019 at 03:15.
byteshare is offline   Reply With Quote
Old 18th October 2019, 04:04   #17545  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by byteshare View Post
That is kind of the point, it would slow it down enough that your next job would actually be encoding later than the way it works now. Yes, technically the job would "start" sooner but the time for the video to copy to the temp drive would take longer with the last job muxing at the same time.
The only way it would be "faster" is if once jobs are started the next job loads the video while the other job is still encoding, then muxing would happen and the next job would start, then while that job is still going but after the last muxing is done the next-next job would start copying the video...this only really works if your jobs are both small enough/slow enough encodes that the timing can work...like I was saying, it would be tricky and wouldn't be helpful in many cases.
...Ideally if some symbolic link to the original file could be used for the sharing (in DE) so that you wouldn't have to even copy a file, then it would be much simpler/faster to start the next job while the last job is muxing.
Good to see you've given this some thought, as well.

So I have to ask, do you do many, if any, 4K movie encodes, and how many pc's can you use for DE ??

I'm just trying to figure out some way to utilise the DE servers more energy efficiently. It just frustrates me to see several servers whirring away, using power, but not doing any encoding

Quote:
Wouldn't work for a lot of people I'd guess. Wouldn't work for me. I'd have to get a 3rd drive.
Well, if I step up to using SDD's or NVMe's for the temp & encodes drives, I'm going to have to buy a LOT

But if a 2nd temp drive could be implemented, I think it would be worth having to buy another drive
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 18th October 2019, 04:13   #17546  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by Atak_Snajpera View Post
Could you test your SSD/HDD in my benchmark tool?
http://www.mediafire.com/file/i630b6...Tester.7z/file

Upload MyLog.csv somewhere.
OK, have done your test's as requested, I hope I have provided the info you wanted to know.

I checked ALL the drives in that PC, just for comparison, some didn't have enough free space to do the 120Gb test, but the speeds are still relevant, I'd say.

Now remembering that these SSD's aren't anything special, they were just a few I had lying around.

http://www.mediafire.com/file/dqlf6c...sktop.rar/file
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 18th October 2019, 04:48   #17547  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 560
Quote:
Originally Posted by Pauly Dunne View Post
Good to see you've given this some thought, as well.

So I have to ask, do you do many, if any, 4K movie encodes, and how many pc's can you use for DE ??

I'm just trying to figure out some way to utilise the DE servers more energy efficiently. It just frustrates me to see several servers whirring away, using power, but not doing any encoding
I do some 4K. I've been doing more and more with time.
I have had up to 6 computers in DE mode, but right now I use 2 since upgrading some systems (made the others ones pointless), and 1-2 other computers with a local only DE mode. I see no reason you can't easily have
8-16 computers being used on one jobs (assuming longer/high CPU jobs).
The main thing is having a fast temp drive for large files.
Quote:
Well, if I step up to using SDD's or NVMe's for the temp & encodes drives, I'm going to have to buy a LOT

But if a 2nd temp drive could be implemented, I think it would be worth having to buy another drive
The main thing would be using the SSD/M.2 for the temp drive (locally mux) then if you still want it somewhere else, have it auto-move to another drive with the new after job scripting to allow the next job will start while it is moving.
byteshare is offline   Reply With Quote
Old 18th October 2019, 15:17   #17548  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by Pauly Dunne View Post
OK, have done your test's as requested, I hope I have provided the info you wanted to know.

I checked ALL the drives in that PC, just for comparison, some didn't have enough free space to do the 120Gb test, but the speeds are still relevant, I'd say.

Now remembering that these SSD's aren't anything special, they were just a few I had lying around.

http://www.mediafire.com/file/dqlf6c...sktop.rar/file
Your SSDs are even worse than your HDDs!
No wonder that muxing takes ages! You really need some good MLC SSD instead of those TLC garbage.








Last edited by Atak_Snajpera; 18th October 2019 at 15:51.
Atak_Snajpera is offline   Reply With Quote
Old 18th October 2019, 16:03   #17549  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by Atak_Snajpera View Post
Your SSDs are even worse than your HDDs!
No wonder that muxing takes ages! You really need some good MLC SSD instead of those TLC garbage
Not that surprised by that...I did say they were all I had...

So, what would you suggest...SSD's or PCI-e mounted NVMe's ??
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)

Last edited by Pauly Dunne; 18th October 2019 at 16:16.
Pauly Dunne is offline   Reply With Quote
Old 18th October 2019, 16:05   #17550  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 560
Quote:
Originally Posted by Atak_Snajpera View Post
Your SSDs are even worse than your HDDs!
No wonder that muxing takes ages! You really need some good MLC SSD instead of those TLC garbage.
Wow, I've never seen that. Maybe they're also near the end of their life?
TY for the graphs.

Quote:
Originally Posted by Pauly Dunne View Post
So, what would you suggest...SSD's or PCI-e mounted NVMe's ??
Those or M.2. Pretty much any drive that does 500MB/s or more read/write, which is pretty common these days.
I have an M.2 that does around 2GB/s for read/write.
byteshare is offline   Reply With Quote
Old 18th October 2019, 16:19   #17551  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by byteshare View Post
Wow, I've never seen that. Maybe they're also near the end of their life?
TY for the graphs.


Those or M.2. Pretty much any drive that does 500MB/s or more read/write, which is pretty common these days.
I have an M.2 that does around 2GB/s for read/write.
Yeah, pretty bad

Thanks, I'll get the best I can that will suit my MB (no native support)
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 18th October 2019, 16:21   #17552  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Wow, I've never seen that. Maybe they're also near the end of their life?
TY for the graphs.
No, it is because SLC cache is tiny (around 3 GiB)

Quote:
So, what would you suggest...SSD's or PCI-e mounted NVMe's ??
Obviously something using NVMe. Downside of PCI-e is that you most likely won't be able to boot from that drive. Before you buy anything make sure that write speed is good across whole capacity. Do not be fooled by that SLC-cache used in TLC/QLC drives.

Last edited by Atak_Snajpera; 18th October 2019 at 16:24.
Atak_Snajpera is offline   Reply With Quote
Old 19th October 2019, 00:28   #17553  |  Link
mparade
Registered User
 
Join Date: Nov 2013
Posts: 577
Quote:
Originally Posted by Atak_Snajpera View Post
1) Source files are on your home PC.
2) Work PC just needs EncodingServer.exe running!
3) The idea is that you connect via VPN to work PC and then Work PC sees shared folder (for example \\HOME-PC\RipBot264temp) on Home PC.
With VPN connection established using my remote PC, servers of the remote PC remain in offline mode even after trying to connect to them from home PC (after pressing ON button it tries to connect to remote servers but after some second it returns to offline mode again). Should I set something else as well? My admin told VPN now should be working. Please help. Thank you.

Last edited by mparade; 19th October 2019 at 00:32.
mparade is offline   Reply With Quote
Old 19th October 2019, 02:18   #17554  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by byteshare View Post
I have an M.2 that does around 2GB/s for read/write.
Hey, well, I think I've decided what I want to try, and that is get a couple of NVMe PCI-e Adapter's, and maybe 500Gb M.2 Samsung 970's (or similar).

Might as well spend a little more, than "settle" for some "fast" SSD's, they are so 2017

And to a point that Atak made, PCI-e mounted NVMe's may not boot, but I won't be using them as the boot / OS drive.
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 19th October 2019, 02:23   #17555  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by Atak_Snajpera View Post
Your SSDs are even worse than your HDDs!

I just noticed that these are depicted as being Green HDD's, but indeed, they were Green SSD's in a RAID 0...

So that show's you how piss weak they are !!!
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 19th October 2019, 07:54   #17556  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 560
Quote:
Originally Posted by mparade View Post
With VPN connection established using my remote PC, servers of the remote PC remain in offline mode even after trying to connect to them from home PC (after pressing ON button it tries to connect to remote servers but after some second it returns to offline mode again). Should I set something else as well? My admin told VPN now should be working. Please help. Thank you.
Can you reach the share manually from the remote PC?
Can you ping the remote PC?
byteshare is offline   Reply With Quote
Old 19th October 2019, 07:59   #17557  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by Atak_Snajpera View Post
Obviously something using NVMe. Downside of PCI-e is that you most likely won't be able to boot from that drive. Before you buy anything make sure that write speed is good across whole capacity. Do not be fooled by that SLC-cache used in TLC/QLC drives.
Could you please give me your honest opinion on these items ??

NVMe PCI-e adaptors

https://www.amazon.com/Silverstone-S...1467552&sr=8-7

or

https://www.amazon.com/SilverStone-T...1467714&sr=8-3

500Gb NVMe's

https://www.amazon.com/XPG-SX8200-Ge...1467837&sr=8-1

or

https://www.amazon.com/Samsung-970-E...s%2C387&sr=8-2

At this stage, I think I would go a 500Gb for the temp drive, and 250Gb for the encode drive, and just "clean up" after every job or 2.
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)
Pauly Dunne is offline   Reply With Quote
Old 19th October 2019, 13:32   #17558  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806


Like I said, if you can afford buy Samsung 970 PRO (MLC)... Adata sx8200 also looks good until 60GiB mark.

Last edited by Atak_Snajpera; 19th October 2019 at 13:41.
Atak_Snajpera is offline   Reply With Quote
Old 19th October 2019, 14:03   #17559  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by mparade View Post
With VPN connection established using my remote PC, servers of the remote PC remain in offline mode even after trying to connect to them from home PC (after pressing ON button it tries to connect to remote servers but after some second it returns to offline mode again). Should I set something else as well? My admin told VPN now should be working. Please help. Thank you.
1) run encodingserver.exe on remote pc and take a screenshot.
2) run ripbot264.exe on your home pc. go to settings->Distributed encoding and show me what client ips are listed under CLIENT IP
3) Are you sure firewall on remote pc does block connection?
Atak_Snajpera is offline   Reply With Quote
Old 19th October 2019, 14:11   #17560  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 692
Quote:
Originally Posted by Atak_Snajpera View Post


Like I said, if you can afford buy Samsung 970 PRO (MLC)... Adata sx8200 also looks good until 60GiB mark.
I would be interested to see this info for the new'ish 970 Plus.

Found some :- not too good

https://www.tomshardware.com/reviews...sd,5608-2.html

Adata is 2nd best, but a fair bit off the mark

I think the Pro is a little too exy for me, the Plus is substantially cheaper, but might be only a little "slower"

And are those adapters suitable ??
__________________
Not poorly done, just doin' it my way !!!
Live every day like it's your last, because one day, it will be !! (M$B)

Last edited by Pauly Dunne; 19th October 2019 at 14:17.
Pauly Dunne is offline   Reply With Quote
Reply

Tags
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.