Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th January 2019, 12:45   #16541  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,774
Quote:
Also found something out with the Numa support. If I disabled memory interleaving to enable Numa, the processors would only be partially utilized when running MDegrain. Turning off Numa then the processors would be fully utilized. So I've disabled Numa for myself.
I suppose your issue may be similar to necrosoft's https://forum.doom9.org/showthread.p...81#post1861681
Atak_Snajpera is offline   Reply With Quote
Old 8th January 2019, 14:19   #16542  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 186
Quote:
Originally Posted by Atak_Snajpera View Post
I suppose your issue may be similar to necrosoft's https://forum.doom9.org/showthread.p...81#post1861681
Actually, it did not look that way. Each numa node showed about 30% utilization. Maybe mdegrain2 runs better when it sees many cores instead of numa nodes? This was from two Dell R710s.

When I was first setting up the numa I had the same thing happen to me about both going to the same node. Just closed everything again and the problem never returned.
Ryushin is offline   Reply With Quote
Old 8th January 2019, 14:34   #16543  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,774
I have no access to dual socket machines so I can't help here. I can only infer that more available CPU threads means smaller bottleneck while serving frames to encoder.
Basically single 6 core cpu with mdegrain2 is too slow to feed fast enough x264/x265 encoder. In latest version you have a nice graph in encoding server showing how much of cpu time is spent for each process. I predict that x264/x265 uses less cpu time than decoder.

Last edited by Atak_Snajpera; 8th January 2019 at 14:37.
Atak_Snajpera is offline   Reply With Quote
Old 8th January 2019, 14:47   #16544  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 186
Quote:
Originally Posted by Ryushin View Post
Can the HQDN3D denoiser be added back? Only one of my four encoding machines even has a video card that can do OpenCL. Everything else is CPU only.
Maybe it would be best to HQDN3D out since it is not 10bit (from my understanding) and just add mdegrain3 as you will not have to add any other tools, and just change the options generated in the video script.

Actually, that sounds like a good idea to me. If possible, can you change the denoiser to allow selecting mdegrain, then the toggle arrows to select mdegrain1-6 (if using 2.7) and defaulting to mdegrain2 and adding the toggle arrows or a input box for the thSAD= parameter with a default of 200. If toggle arrows, change the amount by increments of 25.
I'm almost always having to change the thSAD value anytime I choose to use mdegrain. Super heavy noise such as in Blade Runner 4K, I choose 600. For light denoising I'll choose 100. Normal denoising for most content I choose 150-200.

I really think only mdegrain2-3 is necessary. But I'm sure, someone down the line, will ask for 4+

Oh, and add mdegrain* to the advanced setting "Limit to the following filters only"

Last edited by Ryushin; 8th January 2019 at 14:49.
Ryushin is offline   Reply With Quote
Old 8th January 2019, 14:53   #16545  |  Link
Ryushin
Registered User
 
Join Date: Mar 2011
Posts: 186
Quote:
Originally Posted by Atak_Snajpera View Post
I have no access to dual socket machines so I can't help here. I can only infer that more available CPU threads means smaller bottleneck while serving frames to encoder.
Basically single 6 core cpu with mdegrain2 is too slow to feed fast enough x264/x265 encoder. In latest version you have a nice graph in encoding server showing how much of cpu time is spent for each process. I predict that x264/x265 uses less cpu time than decoder.
You are probably right there. This is old hardware anyway. It should be up to us choose the right options for our hardware.

BTW, big thumbs up on the new Encoding Client and Encoding Server UI. Very impressed.

Oh, and I cannot begin to tell you how much I like being able to move jobs around in the queue. Especially with my queue being 50-100 jobs deep most of the time.
Ryushin is offline   Reply With Quote
Old 10th January 2019, 17:04   #16546  |  Link
Balthazar2k4
Registered User
 
Join Date: Mar 2009
Location: Here, There, & Everywhere
Posts: 258
I've recently got back into doing some encoding and RipBot264 is always my first choice. I am presently setup to use my three machines on my wired LAN to utilize distributed encoding. I do a lot of 4K HDR to 1080P SDR conversions so I need all the speed I can get. That said I am perplexed by the amount of time lost demuxing, indexing, copying to shared folder, and indexing again to use distributed encoding. It seems to me that there could be a real improvement in efficiency by consolidating some of this work in the first pass. 4K HDR files sourced from 4K BD are rather big and on a gigabit network connection still takes a rather significant amount of time to move that volume of data. In this case we are touching the data twice. Would it be possible to streamline this so that if we enable distributed encoding that RipBot264 handles the entire demux/copy to shared folder step in one pass?

Last edited by Balthazar2k4; 10th January 2019 at 17:07.
Balthazar2k4 is offline   Reply With Quote
Old 10th January 2019, 17:18   #16547  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,774
You may try enabling "Move video file to shared folder" in Distributed Encoding tab. This will speedup work only if files are stored locally.

Last edited by Atak_Snajpera; 10th January 2019 at 17:21.
Atak_Snajpera is offline   Reply With Quote
Old 10th January 2019, 17:35   #16548  |  Link
Balthazar2k4
Registered User
 
Join Date: Mar 2009
Location: Here, There, & Everywhere
Posts: 258
Quote:
Originally Posted by Atak_Snajpera View Post
You may try enabling "Move video file to shared folder" in Distributed Encoding tab. This will speedup work only if files are stored locally.
Thanks Atak. I'll give that a try and report back.

UPDATE: This doesn't seem to be any faster, in fact, it's slower. It is simply moving the video to the shared folder which then has to be moved back. I have all of my videos housed on a server which is on my wired network. I am beginning to think that is not 'local' enough.

Is it possible to have the demux process also copy the video stream when it is processing the audio and sub tracks during the initial load?

Last edited by Balthazar2k4; 10th January 2019 at 18:27.
Balthazar2k4 is offline   Reply With Quote
Old 10th January 2019, 20:23   #16549  |  Link
slalom
Registered User
 
slalom's Avatar
 
Join Date: Jan 2010
Posts: 310
Quote:
Originally Posted by Balthazar2k4 View Post
UPDATE: This doesn't seem to be any faster, in fact, it's slower. It is simply moving the video to the shared folder which then has to be moved back. I have all of my videos housed on a server which is on my wired network. I am beginning to think that is not 'local' enough.
Obviously, that's not local, that's on the network
__________________
i7-980 @ 4.16GHz 8GB
i5-750 @ 3.36GHz 12GB
Sony Vaio VPC-F13Z1E/B
slalom is offline   Reply With Quote
Old 10th January 2019, 20:53   #16550  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,774
Quote:
UPDATE: This doesn't seem to be any faster, in fact, it's slower. It is simply moving the video to the shared folder which then has to be moved back. I have all of my videos housed on a server which is on my wired network. I am beginning to think that is not 'local' enough.

Is it possible to have the demux process also copy the video stream when it is processing the audio and sub tracks during the initial load?
By saying locally I meant on the same machine.
Atak_Snajpera is offline   Reply With Quote
Old 10th January 2019, 21:30   #16551  |  Link
Balthazar2k4
Registered User
 
Join Date: Mar 2009
Location: Here, There, & Everywhere
Posts: 258
Quote:
Originally Posted by slalom View Post
Obviously, that's not local, that's on the network
Quote:
Originally Posted by Atak_Snajpera View Post
By saying locally I meant on the same machine.
Fair point. I wasn't really quite awake this morning when I was working on it. After a couple cups of coffee it is far more obvious.
Balthazar2k4 is offline   Reply With Quote
Old 10th January 2019, 21:45   #16552  |  Link
pepeq
Registered User
 
Join Date: Dec 2018
Posts: 17
Quote:
Originally Posted by pepeq View Post
...

In the past I've done my encodings with Handbrake and there the Nvidia NVenc option for encoding gives things a boost.

Question: Is it possible to add the Nvidia NVenc/H265 as an encoder option to RipBot264?
Than it would be perfect for me!
@Atak: do you think my proposal can be realized?
Or do you think it makes no sense?
pepeq is offline   Reply With Quote
Old 11th January 2019, 01:29   #16553  |  Link
Balthazar2k4
Registered User
 
Join Date: Mar 2009
Location: Here, There, & Everywhere
Posts: 258
I am not sure what happened, but the latest Encoding Server will no longer open on my 9900K. It was working just fine until I rebooted the machine and now the process just hangs. Tried additional reboots with no success so I rolled back to 1.12.5.0 and it works fine. Anyone else experienced this?
Balthazar2k4 is offline   Reply With Quote
Old 11th January 2019, 11:58   #16554  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,774
Quote:
Originally Posted by pepeq View Post
@Atak: do you think my proposal can be realized?
Or do you think it makes no sense?
I do not own nvidia card and chances I will have one in near future are almost zero.
Atak_Snajpera is offline   Reply With Quote
Old 11th January 2019, 13:37   #16555  |  Link
pepeq
Registered User
 
Join Date: Dec 2018
Posts: 17
Quote:
Originally Posted by Atak_Snajpera View Post
I do not own nvidia card and chances I will have one in near future are almost zero.
oh, that are bad news for me...
But I understand, without testing it isn't possible to implement.
If you think I can test something for you, please respond. I will be willing to do so.
pepeq is offline   Reply With Quote
Old 11th January 2019, 13:55   #16556  |  Link
pepeq
Registered User
 
Join Date: Dec 2018
Posts: 17
Quote:
Originally Posted by Balthazar2k4 View Post
I am not sure what happened, but the latest Encoding Server will no longer open on my 9900K. It was working just fine until I rebooted the machine and now the process just hangs. Tried additional reboots with no success so I rolled back to 1.12.5.0 and it works fine. Anyone else experienced this?
I had the same problem when EncodingServer.exe changed from 1.12.6 to 1.13.0. And the problem existed also with 1.14.0.

I have 4 PCs (3x Win10 Pro and 1x Win7) and only one of my Win10-PCs had this problem.
EncodingServer.exe was started, the process existed, but no GUI/Window was displayed. It blocked also RipBot264.exe, when I tried to exit Ripbot264.exe.

I analyzed with Windows-taskmanager and found, that an already started process everything.exe (a fast NTFS-searchtool, see voidtools.com) blocked EncodingServer.exe. EncodingServer.exe was waiting for a not longer responding everything.exe process.
I don't know why, because the same everything.exe runs on all of my PCs and made no problem with EncodingServer.exe below 1.13.0.
I killed everything.exe and then EncodingServer.exe continued to run and its window came up.

The strange thing was, that this problem did not alway arise, but in about 90% of the calls to EncodingServer.exe. Maybe a Window-update changed somthing.

I updated everything.exe to the latest version and the problem has gone (until now).

Taskmanager showed the process-queue, see attachment.
Now EncodingServer.exe does not wait for another process and is ok.
I have no picture which showed the blockin queue, because the problem is gone now.

Conclusion: check with taskmanager if some other process is blocking the stalled EncodinServer.exe in your case.

Hope that helps.

Peter
Attached Images
 
pepeq is offline   Reply With Quote
Old 11th January 2019, 14:27   #16557  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,774
Quote:
Originally Posted by pepeq View Post
oh, that are bad news for me...
But I understand, without testing it isn't possible to implement.
If you think I can test something for you, please respond. I will be willing to do so.
I'm not interested in hardware encoding at all because hardware encoders suck in fine detail retention at low bitrates. Another problem. Hardware encoding will also be less useful in Distributed encoding mode where most laptops/pcs use non nvidia GPUs.

Summary
  1. I do not have nvidia GPU and my next GPU will be for 100% AMD navi
  2. Hardware encoders produce noticeable more blurry image at lower bitrates
  3. Many machines in DE mode could not be used for encoding due to incompatible gpu.
Atak_Snajpera is offline   Reply With Quote
Old 11th January 2019, 15:20   #16558  |  Link
Balthazar2k4
Registered User
 
Join Date: Mar 2009
Location: Here, There, & Everywhere
Posts: 258
Quote:
Originally Posted by Atak_Snajpera View Post
I'm not interested in hardware encoding at all because hardware encoders suck in fine detail retention at low bitrates. Another problem. Hardware encoding will also be less useful in Distributed encoding mode where most laptops/pcs use non nvidia GPUs.

Summary
  1. I do not have nvidia GPU and my next GPU will be for 100% AMD navi
  2. Hardware encoders produce noticeable more blurry image at lower bitrates
  3. Many machines in DE mode could not be used for encoding due to incompatible gpu.
To back Atak's point, I have tried GPU encoding for several years. Each year the quality does improve, but the overall image is still nowhere near what traditional CPU encoding can do. Often the image is softer with more compression artifacts and larger file size. The only win is the speed. I can encode 1080p at >200fps on my 2080ti which makes the need for distributed encoding null in my opinion, but the result is so underwhelming that I can't justify it. Also, as soon as you throw any additional filters in the mix such as denoise or tonemapping you lose the speed anyways. I'll stick with distributed encoding for now.
Balthazar2k4 is offline   Reply With Quote
Old 11th January 2019, 15:51   #16559  |  Link
byteshare
ByteShare
 
byteshare's Avatar
 
Join Date: Sep 2014
Location: On the Internet
Posts: 365
Quote:
Originally Posted by Ryushin View Post
Maybe it would be best to HQDN3D out since it is not 10bit (from my understanding) and just add mdegrain3 as you will not have to add any other tools, and just change the options generated in the video script.

Actually, that sounds like a good idea to me. If possible, can you change the denoiser to allow selecting mdegrain, then the toggle arrows to select mdegrain1-6 (if using 2.7) and defaulting to mdegrain2 and adding the toggle arrows or a input box for the thSAD= parameter with a default of 200. If toggle arrows, change the amount by increments of 25.
I'm almost always having to change the thSAD value anytime I choose to use mdegrain. Super heavy noise such as in Blade Runner 4K, I choose 600. For light denoising I'll choose 100. Normal denoising for most content I choose 150-200.

I really think only mdegrain2-3 is necessary. But I'm sure, someone down the line, will ask for 4+

Oh, and add mdegrain* to the advanced setting "Limit to the following filters only"
You can already use the other MDegrain modes. I use MDegrain3 a lot. The only catch is you need to add it as a custom script. I've posted such scripts several times in this thread.
byteshare is offline   Reply With Quote
Old 11th January 2019, 16:14   #16560  |  Link
Balthazar2k4
Registered User
 
Join Date: Mar 2009
Location: Here, There, & Everywhere
Posts: 258
Thanks @pepeq. I have a pretty deep queue processing right now so I'll take a look at it again when it wraps.

UPDATE: Analyzed the wait chain as you suggested and sure enough it was the Discord applet in my Logitech Gaming software that was the hold up. Killed the applet and 'Voila!' loaded right up. What I can't figure out is why it worked the first time and then none of the subsequent. Weird, but problem solved. Thanks again @pepeq.

Last edited by Balthazar2k4; 11th January 2019 at 18:27.
Balthazar2k4 is offline   Reply With Quote
Reply

Tags
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:30.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.