Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 7th December 2019, 16:44   #17961  |  Link
GZZ
Registered User
 
Join Date: Jan 2002
Posts: 546
Quote:
Originally Posted by Pauly Dunne View Post
So I would imagine that most of you have a reasonable collection of both unprocessed and processed movies & series, etc, which can add up to a LOT of Terabytes.

I would be interested to know what some of you (who are happy to oblige) have & use for storage, of said data ???

I have several reasonable capacity Supermicro storage servers, and a couple of Unraid "box's", but unfortunately, every so often there is a problem, and as a result, I get data corruption.

I don't really have too much backed up, so sometimes it's a complete loss , which can be very frustrating, especially if it was irreplaceable.

Anyway, I'm just curious...

Should be in another thread or properly another group, try search forum as its out off topic for this thread.
GZZ is offline   Reply With Quote
Old 7th December 2019, 17:34   #17962  |  Link
tkwou
Registered User
 
Join Date: Nov 2016
Posts: 30
Quote:
Originally Posted by tkwou View Post
Hi Atak, I recently updated to v1.25.0. I just tried encoding the movie Life in DE mode and it failed with about 8 minutes left in the movie. When I look at the Chunks folder there are no *.264 files for the last 8 chunks. When I try to run each .cmd file manually I get these errors.

y4m[error]: bad sequence header magic
x2464[error]: could not open input file `_'

If I rerun the .cmd files for any of the completed chunks I get the normal .264 output. I don't understand what's going on. Should I just try recoding again from scratch?
Hi Atak, strange this just happened again. Processing stopped with just 2 chunks to go. With the missing chunks, the CombineChunk.cmd fails and video.264 is missing for the mux. I restarted the encode and again it finished the final 2 chunks and everything completed fine.
tkwou is offline   Reply With Quote
Old 7th December 2019, 18:31   #17963  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,317
Quote:
Originally Posted by tkwou View Post
Hi Atak, strange this just happened again. Processing stopped with just 2 chunks to go. With the missing chunks, the CombineChunk.cmd fails and video.264 is missing for the mux. I restarted the encode and again it finished the final 2 chunks and everything completed fine.
You have some network issues or some instability in RAM/CPU...
Atak_Snajpera is offline   Reply With Quote
Old 7th December 2019, 18:57   #17964  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 265
Quote:
Originally Posted by Atak_Snajpera View Post
With latest version of lsmash I recommend to use it for ALL formats! Just forget about ffms. Lsmash is superior in almost every aspect.
So the random frames happening around the end of the DE chunks should be fixed? Works for VC-1 too?
Ryushin is offline   Reply With Quote
Old 7th December 2019, 20:17   #17965  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,317
Quote:
Originally Posted by Ryushin View Post
So the random frames happening around the end of the DE chunks should be fixed? Works for VC-1 too?
yes...
Atak_Snajpera is offline   Reply With Quote
Old 8th December 2019, 01:43   #17966  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Quote:
Originally Posted by GZZ View Post
Should be in another thread or properly another group, try search forum as its out off topic for this thread.
OMG, GZZ !!!

I knew that I was off topic for this thread, but I thought I'd just ask the question of the member's on THIS thread, as they MUST have some form of reliable storage.

So why you reply to state the bleeding obvious is a waste of your time, when for the same effort you could have said something like ....I have a Synology or Unraid backup....
__________________
Not poorly done, just doin' it my way !!!
Pauly Dunne is offline   Reply With Quote
Old 8th December 2019, 07:05   #17967  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Att:- Atak (MDegrain custom filter testing)

Is there a way to confirm that these "Custom" MDegrain filters are / or have been used ??
Other than creating some short test encodes and comparing the fps, kbps (whilst processing), or time taken to encode & final file size ??

Anyway, I did that, and here's some interesting results:-

a test tv episode, mp4, 17 minutes long.

MDegrain1 took 8:34m

MDegrain2 took 11:43m

MDegrain3 took 15:18m

they are ALL with MT enabled.

Then I ran MDegrain1 test again changing the MT settings.

with no MT, it took a "massive" 28:25m.

with MT enable, but using 16 threads took the original 8:34m

with 8 threads it took 8:17m

with 0 (which must be unlimited) it took 8:08m (on my 24 thread pc)

so going on that all the above times would be a touch faster if I had of had it set on 0 threads.

As for file size change:-

original file is 255,856kb @ 1,923kb/s

MD1 encoded is 303,196kb @ 2,192kb/s

MD2 encoded is 289,490kb @ 2,126kb/s

MD3 encoded is 281,275kb @ 2,126kb/s

No other filters or changes where made, using CRF 20, and ONLY the custom MDegrain filters.
__________________
Not poorly done, just doin' it my way !!!
Pauly Dunne is offline   Reply With Quote
Old 8th December 2019, 13:26   #17968  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,317
Zero means number of physical cores If I remember correctly. So If your CPU is 12c/24t then value will be 12.
Regarding file size. You should try with noisy source. Difference will be much higher. MD2 should give you up to 1.8x bitrate reduction.

Last edited by Atak_Snajpera; 8th December 2019 at 13:29.
Atak_Snajpera is offline   Reply With Quote
Old 8th December 2019, 13:57   #17969  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Quote:
Originally Posted by Atak_Snajpera View Post
Zero means number of physical cores If I remember correctly. So If your CPU is 12c/24t then value will be 12.
Regarding file size. You should try with noisy source. Difference will be much higher. MD2 should give you up to 1.8x bitrate reduction.
So does "0" automatically determine how many cores/threads are available ??

As "0" is the fastest of the settings I tried.

Also, I DID do a test on much noisier source (The Fifth Element), and there was a bigger file size/bitrate difference between the three.
__________________
Not poorly done, just doin' it my way !!!
Pauly Dunne is offline   Reply With Quote
Old 9th December 2019, 04:26   #17970  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Random errors, several times during 1 job..

Me again,

I thought I'd run a few job thru RB today, on a pc that I was using successfully just last week, and I just keep getting errors, several time during the same job.

Cannot find file F:\Temp\RipBot264temp\job10\video.264
Error importing F:\Temp\RipBot264temp\job10\video.264:name=The Report 2019: Requested URL is not valid or cannot be found

It got thru approx 12 chunks before the 1st error, then after restarting the job, it got thru to chunk 24, then the next time I think it got thru quite a bit further, and errored again.

After several more attempts with different jobs, restarting the pc, I gave up, and went and did some cooking.
__________________
Not poorly done, just doin' it my way !!!
Pauly Dunne is offline   Reply With Quote
Old 9th December 2019, 15:19   #17971  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,317
Auto-update 2019.12.09
Code:
Added: EncodingClient combines chunks with output file on fly as soon as they are encoded in correct order
Added: Audio profiles in Profile\Audio.txt
Changed: Jobless servers will be suspended only if ETA>60s (reason: reduce unnecessary suspend -> resume cycles)
Fixed: Excessive flickering of encoding status in EncodingClient
Fixed: EncodingClient didn't resume suspended jobless servers before starting second pass.
Fixed: EncodingServer didn't go to suspend mode again after previous resume

Last edited by Atak_Snajpera; 9th December 2019 at 16:07.
Atak_Snajpera is offline   Reply With Quote
Old 9th December 2019, 16:03   #17972  |  Link
ReinerSchweinlin
Registered User
 
Join Date: Oct 2001
Posts: 244
Thanks for the update log. This is very helpful, please continue posting when updates are done
ReinerSchweinlin is offline   Reply With Quote
Old 9th December 2019, 16:54   #17973  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 537
Quote:
Originally Posted by Pauly Dunne View Post
So I would imagine that most of you have a reasonable collection of both unprocessed and processed movies & series, etc, which can add up to a LOT of Terabytes.

I would be interested to know what some of you (who are happy to oblige) have & use for storage, of said data ???
https://www.reddit.com/r/DataHoarder/
I use 1 to 1 backups

Quote:
Originally Posted by Pauly Dunne View Post
Is there a way to confirm that these "Custom" MDegrain filters are / or have been used ??
Other than creating some short test encodes and comparing the fps, kbps (whilst processing), or time taken to encode & final file size ??
Won't give you the benefits of a test encode (seeing the actual file results) but you can check:
(on a job) edit > AviSynth > Show Video Script > Look under the custom section that you see your script, then...
> Preview (first to make sure there are no errors in your script.
If no errors and your script is being loaded > AVS Meter to see the speed and such it runs at.
If you're still skeptical that your script isn't being used because of a mistake in your code that you're not seeing try removing it and then running the AVS Meter again, if there is no difference in speed (FPS) then something could be wrong with your script and then you'd want to run a test encode to see if there are any filesize differences, if none then it is likely your script or you're using a filter that is not having an impact either by design or by an error in your script.
(tip, on your script be sure "video" is somewhere after the "=" otherwise it isn't being using in the final video sent to the HEVC encoder, otherwise it is fine if you're using it for a mask, prefilter, etc...

Quote:
Originally Posted by Pauly Dunne View Post
original file is 255,856kb @ 1,923kb/s
MD1 encoded is 303,196kb @ 2,192kb/s
MD2 encoded is 289,490kb @ 2,126kb/s
MD3 encoded is 281,275kb @ 2,126kb/s
Besides MD1-3 (there are higher ones as well) you can try adjusting the strength as well (thSAD, higher values mean more removal of grain/noise but too high removes noticeable details). Can also try after MD a filter like FineDehalo/HQDeringmod (there are a lot of choices in this area) to remove some created/leftover temporal noise (mosquitoes) depending on your source and filter results.

Last edited by byteshare; 9th December 2019 at 17:13.
byteshare is offline   Reply With Quote
Old 10th December 2019, 03:45   #17974  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Quote:
Originally Posted by Atak_Snajpera View Post
Auto-update 2019.12.09
Code:
Added: EncodingClient combines chunks with output file on fly as soon as they are encoded in correct order
Added: Audio profiles in Profile\Audio.txt
Changed: Jobless servers will be suspended only if ETA>60s (reason: reduce unnecessary suspend -> resume cycles)
Fixed: Excessive flickering of encoding status in EncodingClient
Fixed: EncodingClient didn't resume suspended jobless servers before starting second pass.
Fixed: EncodingServer didn't go to suspend mode again after previous resume
Thanks very much for that, will be a lot easier on the eyes

And how do you use the "audio.txt" ?

Just need to add that I updated a client & 1 server with the latest, and it was running well, it had done 2 3/4 jobs, and Encoding Client just stopped all processing, both Client & Server...had to force RB shutdown with Processhacker, started again, and away it went.

edit addition:-since that error, I have had no more problems, maybe it just needed to "warm up"

I do have a question about the function of the Encoding Client "button" on the very right of the 4 buttons...

Not sure what you've named it, but I did a little fiddling, and noticed that if that button is enabled, as soon as the processing chunk is complete, that server/port goes idle....and if thats the only chunk on a particular server, it would then shut that server down....so my question is why, if there are more chunks for that server to process.

All that comes to my old mind is that if you had a very long queue and you wanted to stop it part way thru, you'd enable that button. Then eventually all the servers would turn themselves off, and you'd only be left with the client.

I actually thought that that button would do similar to the suspend option, BUT turn them off !!

I really like the suspend option, that's great, and seems to work well for me.
__________________
Not poorly done, just doin' it my way !!!

Last edited by Pauly Dunne; 10th December 2019 at 06:27.
Pauly Dunne is offline   Reply With Quote
Old 10th December 2019, 03:51   #17975  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Quote:
Originally Posted by byteshare View Post
https://www.reddit.com/r/DataHoarder/
I use 1 to 1 backups


Won't give you the benefits of a test encode (seeing the actual file results) but you can check:
(on a job) edit > AviSynth > Show Video Script > Look under the custom section that you see your script, then...
> Preview (first to make sure there are no errors in your script.
If no errors and your script is being loaded > AVS Meter to see the speed and such it runs at.
If you're still skeptical that your script isn't being used because of a mistake in your code that you're not seeing try removing it and then running the AVS Meter again, if there is no difference in speed (FPS) then something could be wrong with your script and then you'd want to run a test encode to see if there are any filesize differences, if none then it is likely your script or you're using a filter that is not having an impact either by design or by an error in your script.
(tip, on your script be sure "video" is somewhere after the "=" otherwise it isn't being using in the final video sent to the HEVC encoder, otherwise it is fine if you're using it for a mask, prefilter, etc...


Besides MD1-3 (there are higher ones as well) you can try adjusting the strength as well (thSAD, higher values mean more removal of grain/noise but too high removes noticeable details). Can also try after MD a filter like FineDehalo/HQDeringmod (there are a lot of choices in this area) to remove some created/leftover temporal noise (mosquitoes) depending on your source and filter results.
Thanks for the great reply

I did do a run on the first few minutes of The Fifth Element (which is VERY noisey), and I was pretty happy the MD3 results (thSAD=800).

I do have slightly different thSAD numbers for MD1 & MD2.

I might go back to a much older version of RB, and see what the HQDN3D settings were, and maybe create a custom filter for non 10bit encodes.
__________________
Not poorly done, just doin' it my way !!!
Pauly Dunne is offline   Reply With Quote
Old 10th December 2019, 12:03   #17976  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 265
Quote:
Originally Posted by byteshare View Post
Besides MD1-3 (there are higher ones as well) you can try adjusting the strength as well (thSAD, higher values mean more removal of grain/noise but too high removes noticeable details). Can also try after MD a filter like FineDehalo/HQDeringmod (there are a lot of choices in this area) to remove some created/leftover temporal noise (mosquitoes) depending on your source and filter results.
Do you have a copy of those custom scripts for FineDehalo/HQDeringmod. Originally that is what I was using mdegrain2+hqdn3d. If it does a good job I'm hoping Atak will add those as additional features.
Ryushin is offline   Reply With Quote
Old 10th December 2019, 14:57   #17977  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 265
Sorted Order Reverts on Closing

Just updated and I noticed something, which I'm not sure is related to the last update or not. If I open RipBot264, change the order of the encodes, close RipBot264 and open it again, the order goes back to what it was before I changed the order.
Ryushin is offline   Reply With Quote
Old 10th December 2019, 15:15   #17978  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,317
Quote:
Originally Posted by Pauly Dunne View Post
Thanks very much for that, will be a lot easier on the eyes

And how do you use the "audio.txt" ?

Just need to add that I updated a client & 1 server with the latest, and it was running well, it had done 2 3/4 jobs, and Encoding Client just stopped all processing, both Client & Server...had to force RB shutdown with Processhacker, started again, and away it went.

edit addition:-since that error, I have had no more problems, maybe it just needed to "warm up"

I do have a question about the function of the Encoding Client "button" on the very right of the 4 buttons...

Not sure what you've named it, but I did a little fiddling, and noticed that if that button is enabled, as soon as the processing chunk is complete, that server/port goes idle....and if thats the only chunk on a particular server, it would then shut that server down....so my question is why, if there are more chunks for that server to process.

All that comes to my old mind is that if you had a very long queue and you wanted to stop it part way thru, you'd enable that button. Then eventually all the servers would turn themselves off, and you'd only be left with the client.

I actually thought that that button would do similar to the suspend option, BUT turn them off !!

I really like the suspend option, that's great, and seems to work well for me.
= SUSPEND JOBLESS SERVER (All chunks have been already assigned to other servers)

= SHUTDOWN SERVER AT IDLE STATE (Finish encoding current chunk and then shutdown)

Last edited by Atak_Snajpera; 10th December 2019 at 15:18.
Atak_Snajpera is offline   Reply With Quote
Old 10th December 2019, 18:26   #17979  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 537
Quote:
Originally Posted by Ryushin View Post
Do you have a copy of those custom scripts for FineDehalo/HQDeringmod. Originally that is what I was using mdegrain2+hqdn3d. If it does a good job I'm hoping Atak will add those as additional features.
I usually use SMDegrain rather than straight MDegrain. I don't like HQDN3D on its own. I'd rather use KNLMeansCL with GPU, which is actually the prefilter I use with SMDegrain.
For FineDehalo I either use the default: FineDehalo() or something like:
"FineDehalo(rx=2.0,ry=2.0,thmi=80,thma=128,thlimi=50,thlima=100,darkstr=1.0,brightstr=1.0,showmask=0,contra=0.0,excl=true)"
There are some safer settings if you're using anime/cartoons.

For HQDeringmod I also use the defaults "HQDeringmod()" or something like: "HQDeringmod(nrmode=2,Y=3,U=3,V=3,lsb=False,thr=255.0,darkthr=255.0)"

For those filters and others it depends on the source and what you want.

I'd stay away from thSAD values of 800 and higher for MDegrain, as it can soften the image under closer inspection. SMDegrain values of thSAD are a little different because it combines more filters.
byteshare is offline   Reply With Quote
Old 11th December 2019, 01:01   #17980  |  Link
Pauly Dunne
Grumpy Old Man.
 
Pauly Dunne's Avatar
 
Join Date: Jul 2019
Location: Out There....
Posts: 228
Quote:
Originally Posted by Atak_Snajpera View Post
= SUSPEND JOBLESS SERVER (All chunks have been already assigned to other servers)

= SHUTDOWN SERVER AT IDLE STATE (Finish encoding current chunk and then shutdown)
Thankyou for that, I thought that put that particular server to "sleep" after the encoding was completed, then once the next job queue was starting, they would wake up, and start encoding

However, I really can't see that is all that necessary until towards the very end of the chunk queue, and you want individual servers to turn off.

The only reason I can see for it's use is if you have a LONG queue, and you want to stop it part way thru (for what ever reason).

I think one nice function would be to have a similar option for the client, but when the chunk that was processing completed, it didn't start the next one.

Used in conjunction with you could completely (automatically) pause or stop an encode, without doing it manually.

Anyway, enough of that, it's all a little bit over my head, but I know what works for me.

Could you please explain the use of the new "audio.txt" ?
__________________
Not poorly done, just doin' it my way !!!
Pauly Dunne is offline   Reply With Quote
Reply

Tags
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.