Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VirtualDub, VDubMod & AviDemux

Reply
 
Thread Tools Search this Thread Display Modes
Old 27th December 2017, 10:55   #1  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
livelock using External encoder

I have used External encoder for many years in VirtualDub.
Now I often receives "livelock 7749E78C" when checking log at saving.
I then have to use abort. Do any of you have any idea what has happened? I have not changed anything in program.
asaleo is offline   Reply With Quote
Old 27th December 2017, 14:55   #2  |  Link
raffriff42
Retried Guesser
 
raffriff42's Avatar
 
Join Date: Jun 2012
Posts: 1,373
It's got something to do with threads, priorities, race conditions blah blah blah.

An early vdub blog post, not concerning external encoding, but points to the general issue:
Quote:
The infamous frame 9995 MP3 bug
http://www.virtualdub.org/blog/pivot/entry.php?id=88

For years, I've been receiving reports about a mysterious problem with VirtualDub hanging during a save operation. The program's UI was still responsive, so the application hadn't totally died, but the processing pipeline jammed up such that the render couldn't make any progress or be aborted (a "livelock"). I had initially assumed that this was a deadlock caused by thread synchronization issues, which wouldn't lock the UI because VirtualDub's UI runs in a separate thread, and this was reinforced by the livelock log messages in recent versions indicating that the audio thread was stuck in a system call. However, I could never find the culprit or reproduce the problem. This was very frustrating for me, because it was a long-standing issue that made my program unusable for some people, and which I couldn't fix.

Until now.

I recently found out which software triggers the problem and why — it's the Creative Labs MP3 codec (ctmp3.acm), and it's because of peculiar notifications being sent by that codec. It comes with the software that ships with certain SoundBlaster Live! sound cards; in particular, installing the PlayCenter application will also install the codec. Either renaming the driver file temporarily, uninstalling it, or lowering its priority in the Sounds and Audio Devices control panel so that another MP3 codec has priority will work around the problem. I think I have a viable workaround that I can put into VirtualDub itself, but why this codec causes a lockup is an interesting question in itself....
Change codecs, experiment with thread priority.
Quote:
Originally Posted by plugh View Post
I have found that setting 'processing thread priority' to 'normal' makes the livelock issues go away.
raffriff42 is offline   Reply With Quote
Old 27th December 2017, 18:05   #3  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 775
In VDFM I fixed one bug which lead to this.
For the curious: https://github.com/shekh/VirtualDub2....cpp#L780-L784
The expression is either plain wrong or in undefined behavior area.
sfe.mBatchNumber - ofe.mBatchNumber may result as negative number threated as unsigned -> very large positive number.
(sint32) has no effect because then it is compared with unsigned -> converted to unsigned again. In practice this situation may occur very easily if the source file is decoded very fast (faster than some other processing in the chain).
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 27th December 2017, 18:43   #4  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
Livelock may be caused by sound problem

Thanks for your rapid answers. Saving .avi files from VDub works well.
I now suspects that it has something with the sound processing since I got saved mp4 files with bad sound. I will go on testing to see what factors are involved. I will use both VirtualDub and VirtualDub filter mod in tests.
asaleo is offline   Reply With Quote
Old 28th December 2017, 09:19   #5  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
Merging of files in external encoder did not work for big files.

Saving files using external encoders to mp4 from Vdub- Vdub FilterMod worked well for small files but got stuck for bigger files,( 5000 frames) Vdub went into a loop at 99% ready. After abortion I got three files, an empty mp4 file, one sound file and a video file. It seems that merging of these files caused the livelock loop.

Saving with "save as" to mp4 in VirtualDub FilterMod worked well.

I learnt another thing too. Do not save Processing settings in VirtualDub FilterMod wirh Audio set to "Full processing mode" Loading these settings afterwards makes it impossible to change Audio to "Direct stream mode" if you need that. It seems to work since you can change Audio settings in Vdub without any problem but you will not get a proper sound in resulting video. Setting Audio setting "manually" for each process works.

VirtualDub FilterMod is otherwise the best VDub version I have used.
asaleo is offline   Reply With Quote
Old 28th December 2017, 10:40   #6  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 775
Quote:
Originally Posted by asaleo View Post
Saving files using external encoders to mp4 from Vdub- Vdub FilterMod worked well for small files but got stuck for bigger files,( 5000 frames) Vdub went into a loop at 99% ready. After abortion I got three files, an empty mp4 file, one sound file and a video file. It seems that merging of these files caused the livelock loop.
I have another bugfix still unpublished, related to external encoder.
Can you share everything related to this test so I can try to repeat it:
source video
.vdprof (external encoder settings)
job file with processing settings
registry tree export HKEY_CURRENT_USER\Software\VirtualDub.org

Also when it gets stuck, you can press "dump status" in the status window, "perf" page. It may have something interesting.
And everything from the log window.

Quote:
Originally Posted by asaleo View Post
Do not save Processing settings in VirtualDub FilterMod wirh Audio set to "Full processing mode" Loading these settings afterwards makes it impossible to change Audio to "Direct stream mode" if you need that. It seems to work since you can change Audio settings in Vdub without any problem but you will not get a proper sound in resulting video.
Will try this. What do you mean "not proper sound", is it completely destroyed?
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 28th December 2017, 20:09   #7  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
I have made "dump status" from processing windows when VDub is trying to end saving.

First line in log which shows that something is wrong:
"Dub processing thread has not cycled for 30 seconds - possible live lock ......"
(Thread action stalled due to full pipe processing thread) Current adress 775ED1C

I then made "dumb status" which is shown below:
=================
Processing thread
=================
Completed: Yes
Error encountered: No
Video push ended: Yes
Video ended: Yes
Flushing compressor: No
Codec frames buffered: 0
The processing thread is busy and could not be suspended.

I then made an abort.
Vdub FilterMod now shows: "Vdub in progress"
Later text about something stuck appears.
I confirm that Vdub should be closed.

Saving video without sound did not change anything.
Short videos with 1000 frames saves OK. Big videos with 5000 frames will mostly not succed. Sometimes the three resulting files have merged by themself long after VDub has been closed. Strange.

Regarding problem with Audio setting in saved settings: I used saved settings with Audio set to Full processing mode in Vdub FilterMod and changed Audio to Direct stream mode and made a short video with external encoder. My voice in saved video was hoarse. I could understand what I talked about but it was not a pleasant voice.
asaleo is offline   Reply With Quote
Old 28th December 2017, 20:50   #8  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 775
External encoder: the fact that it completes afterwards lets me think the external encoder process is waiting for more input. If you want, try with this version: https://drive.google.com/open?id=1f8...EpxwRHcigrro-6
There is a fix for wrong pipe output.

Audio: sorry, your description is too obscure. Are you saving with external encoder? Which preset for audio? And you say audio is in "direct mode". Is it already compressed? To which format?
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 29th December 2017, 08:32   #9  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
When using external encoder I normaly do nothing with the sound. Default setting in Vdub is Direct stream mode which works with external encoders. When finished with deshaker and other filters I export the result using external encoder. Since direct stream mode is default for external encoder I have to change audio if it set to full processing mode because of the "save as" possibility in VDub FilterMod. If "save as" also could use Direct stream mode as audio input there would be no conflict.
asaleo is offline   Reply With Quote
Old 29th December 2017, 13:25   #10  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
Put everything on an USB-pin and run in Vista computer

I put everything on an USB pin and run in my old Vista computer.
Everything went well. File saved without problem with external encoder.
Thus it is something in my Windows 10 computer which affects saving my virtualDub file with the external encoder.
asaleo is offline   Reply With Quote
Old 29th December 2017, 20:31   #11  |  Link
raffriff42
Retried Guesser
 
raffriff42's Avatar
 
Join Date: Jun 2012
Posts: 1,373
>something in my Windows 10 computer which affects saving my virtualDub file with the external encoder.

Okay, that reminds me - make sure processes like antivirus, search indexing, thumbnail generation and media player library scanning are not enabled for your write folder.
raffriff42 is offline   Reply With Quote
Old 29th December 2017, 21:54   #12  |  Link
asaleo
Registered User
 
Join Date: Mar 2013
Posts: 27
Thanks Raffriff for your answer. I checked processing going on by using Task manager. No processes were running. I tried a saving but it got stuck as before.
I have now to add that files joined after abortion and VDub was not locked either. I have to wait before aborting to see what happens.
I have now marked in external encoders that files should be merge after abortion. It has not worked before but it worked today,

Last edited by asaleo; 29th December 2017 at 22:10.
asaleo is offline   Reply With Quote
Old 29th May 2018, 11:02   #13  |  Link
exonx
Registered User
 
Join Date: May 2018
Posts: 8
I realise that this is an old thread but I was wondering if anyone had yet found a solution to the problem between VirtualDub's external encoders and Windows 10.
As already mentioned in the the thread the external encoders demultiplex the multimedia file and produce a video file and an audio file then the process stalls.
On most occasions if the process isn't aborted, after an hour or more these two files are eventually remultiplexed. No idea why!
Alternatively if the process is aborted, then the audio and video files may be multiplexed immediately into a single multimedia file using something like mkvtoolnix.
This does however produce a non-standard file although on most occasions it will play in in MPC-HC.
I have returned frequently to this problem but never with any success.
My personal workaround is to dual boot with Windows 7 and use Virtualdub with its External Encoders within this operating system.
The remultiplexing works flawlessly within Windows 7.
Any pointers would be greatly appreciated.
exonx is offline   Reply With Quote
Old 29th May 2018, 12:06   #14  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
Quote:
Originally Posted by exonx View Post
As already mentioned in the the thread the external encoders demultiplex the multimedia file and produce a video file and an audio file then the process stalls.
On most occasions if the process isn't aborted, after an hour or more these two files are eventually remultiplexed. No idea why!
Alternatively if the process is aborted, then the audio and video files may be multiplexed immediately into a single multimedia file using something like mkvtoolnix.
This does however produce a non-standard file
The "external encoders" don't demux anything. VirtualDub handles the demultiplexing/decoding and sends raw video and audio to the external encoders. After both video and audio encoding have finished VirtualDub makes a call to a multiplexer like mkvmerge/mkvtoolnix. This isn't different from calling mkvmerge manually from the command-line. It's any more or less "non-standard" than doing in manually.
sneaker_ger is offline   Reply With Quote
Old 29th May 2018, 12:33   #15  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 775
Did you try with VirtualDub2? Last found issue that I remember: there was unnecessary extra data (garbage) sent to encoders. I can imagine how this could result in non-standard files or hang the process. No idea how Win10 is involved (I don't do anything on it).
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 29th May 2018, 14:37   #16  |  Link
exonx
Registered User
 
Join Date: May 2018
Posts: 8
I apologise I made a mistake. I meant to say that external encoders re-encode and then re-multiplex the temporary audio and video files created after VirtualDub has demultiplexed and then decoded the original multimedia file. In my case I call either mkvmerge, mp4box or ffmpeg. to do the re-multiplexing
Mea Culpa.
I have tried VirtualDub2 but have had no success with that either.
I am just really curious to know why in Windows 7 the process runs perfectly every time and in Windows 10 it doesn't re-multiplex at all. I might add that all my other apps run well in windows 10.
exonx is offline   Reply With Quote
Old 10th June 2018, 20:53   #17  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 775
The issue should be fixed now.
Under the hood:
the helper process vdlaunch.exe was not able to exit because some other software injects garbage thread. I don't have exact confirmation, it could be antivirus or even system compatibility layer.
The exit procedure of vdlaunch.exe was unreliable and failed. Now it is simply "ExitProcess" and works in all cases.
The updated vdlaunch.exe can also be used with old VirtualDub.
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 16th June 2018, 16:40   #18  |  Link
exonx
Registered User
 
Join Date: May 2018
Posts: 8
Fixed

Hi shekh
Sorry I'm late updating you but I've been away for several days and only arrived back yesterday.
I've tested the new VirtualDub2 release with a large variety of file formats and codecs and; TA DA! it works like a dream. It really was a pain having to reboot into windows 7 every time I wanted to use External Encoders.
The only operation that fails now in Windows 10 is AC3 passthrough.
The command line I am using in FFMpeg is "-i - -vn -acodec copy". This throws up the error in the log, "invalid data found when processing input". Again this command works in Windows 7.
However please don't bother about this as I can always re-encode the audio. Windows 10 doubles the encoding speed compared to Windows 7
Thank you so much I'm really very grateful for your trouble.

Last edited by exonx; 16th June 2018 at 16:46.
exonx is offline   Reply With Quote
Old 17th June 2018, 00:40   #19  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 775
Thanks for confirmation
shekh is offline   Reply With Quote
Old 17th June 2018, 20:30   #20  |  Link
gvp
Registered User
 
Join Date: Feb 2017
Posts: 1
Quote:
Originally Posted by exonx View Post
The only operation that fails now in Windows 10 is AC3 passthrough.
The command line I am using in FFMpeg is "-i - -vn -acodec copy".
I'm using this conf for ac3 and I have no problems in win10


https://pasteboard.co/Hqm29pU.jpg

Last edited by gvp; 17th June 2018 at 20:39.
gvp is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:53.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.