Synchronising two videos; different audio, frame drops

[Goggen240]

Hi there!

I have a five part eight-hour Soviet war movie I want to watch, (who wouldn't) and I've been trying to figure out a problem.
The movie(s) is available in two versions:
Russian (YouTube) and German (Blu-ray)
The Russian version overdubs all German speech in Russian, and the German versions overdubs all Russian speech in German, naturally.

So here's the problem:
I want the Russians to speak Russian and the Germans to speak German.

Here's the fifth movie (of five) on YouTube:
https://www.youtube.com/watch?v=MqMZSWaiJVE
And at 59:26, you can compare to this dubbed German clip:
https://www.youtube.com/watch?v=h7muqO7kngk

And here's the second problem:
They are not in sync.
The two different versions are also scanned from different sources; the image and audio is different, and both versions have different frame drops at different times (reel changes, it seems); the German drops behind, then less behind, then more behind.
The Russian dub is on top of the German speech, so it's very noticeable when the German version would be appropriate.
The Russian version is also worse quality, so I'd prefer to use the German version where possible.
(On YouTube it's 25FPS for some reason, but that's a quick fix. And the German version doesn't have subtitles.)

So I'd need some procedure to figure out two things:
1. Where there are missing frames, and which version is missing how many frames, so that they can be synced
2. When synced; where the audio is different (enough) that it's easy to see that they're speaking the "wrong" language.
(3. Also sync the subtitles, but once I know the offsets I can do that by hand, if necessary.)

I found this older thread with a slightly similar problem:
https://forum.doom9.org/showthread.p...mparing+videos
I'm not sure how to use this script or if it would even work; from what I understand it's picky about cropping and such, and it doesn't do any of the audio work I'm hoping to do.

I'd preferably automate the whole process, or what's possible to automate, because I'm certainly not going to watch this 8-hour behemoth more than once.

I figure that the video doesn't even have to be re-cut; cutting a few frames worth of audio, or adding a couple frames of silence, should be fine.
If I could sync the audio files, I could just switch languages on-the-fly while watching, but I'd prefer to do it in advance if that's simple. If I get a list of differently-audio'd scenes, I could listen to a second here and there and splice in the "right one" manually. (I'm not too worried about spoilers; the Germans lose.) But I'm not going to listen to the whole soundtrack twice. And then watch the movie.
Naaah...

Technical differences:
The German version seems to be a master copy, but is cropped smaller, particularly on the right side (for the soundtrack?). The two files have different filmgate shake. The Russian version might be a distribution copy; it's dirty in general, and noticeably dirtier at reel changes. There are tinting differences; tints are redder in the German version.
That's the stuff I've noticed scrubbing quickly through.

It's possible that only the German version has frame drops, so that might simplify things. And the frame drops might happen more often during on-screen titles.

I've checked de-sync by frame count:
Russian: German:
Frame #10000: 10000 10000
Frame #20000: 20027 20000
Frame #30000: 30028 30000
Frame #40000: 40029 40000
Frame #50000: 50041 50000
Frame #60000: 60042 60000
Frame #70000: 70043 70000
Frame #80000: 80045 80000
Frame #90000: 80044 90000
Frame #100000: 100048 100000
Frame #110000: 110076 110000
Frame #120000: 120076 120000
Frame #130000: 130077 130000
Total frames: 131797 131883


So, any good suggestions from you fine folks?

[johnmeyer]

I think StainlessS developed some technology for dealing with this. Search using his user name. Perhaps he will check in at some point.

[StainlessS]

Apart from the thread you linked, there are a few others,

MatchClips:- https://forum.doom9.org/showthread.php?t=86691
LineUpScenes:- https://forum.doom9.org/showthread.php?t=95115
https://forum.doom9.org/showthread.p...ighlight=mg262
mg262's plugins with source:- https://forum.doom9.org/showthread.p...ighlight=mg262

Avisynth filter Collection from warpenterprises(OLD plugins on Wiki):- http://www.avisynth.nl/users/warpenterprises/

Generally related:-
https://forum.doom9.org/showthread.php?t=174653
https://forum.doom9.org/showthread.php?t=177065
FindFrame:- http://forum.doom9.org/showthread.ph...22#post1569622
FindCuts:- https://forum.doom9.org/showthread.php?t=135423
MatchFrames/LocateFrames:- http://forum.doom9.org/showthread.php?t=164766
SirPlant:- http://forum.doom9.org/showthread.ph...98#post1787898
General:- https://forum.doom9.org/showthread.php?t=176227
AutoOverlay:- https://forum.doom9.org/showthread.php?t=175247
WaveForm_FilmStrip:- https://forum.doom9.org/showthread.p...63#post1863963
CutFrames:- https://forum.doom9.org/showthread.php?t=135423

What you are trying to do is incredibly difficult, and even if you do manage it, is likely to be a bit awful to listen to
(audio will probably sound completely different when switching from one to the other).
I simply would not even try to do it. [un-scrambling an egg might be easier]
EDIT: Although I guess that if there are long pauses between change of audio, then not so awful after all.



For subtitles, you might find that someone has went to the bother of making a subtitles file, do a google search on

Subtitle "Name of movie"

EDIT: Apparently, there are subtitles in English, Russian, Spanish, Romanian, Portuguese.
google [EDIT: I've added "-Amazon" to below link to remove those entries from listing.

subtitle "Liberation" "The Last Assault" -Amazon

https://www.google.com/search?ei=2kL...4dUDCAo&uact=5

Perhaps some kind of auto translate would suffice. SubtitleEdit has some such feature, but I've never tried it.


EDIT: There are I believe tools to extract/remove human voice from stereo songs, where voice part is equally apparent in both channels,
sometimes used to remove voice for eg karaoke type stuff (In war movie, may not work too well what with all them guns and things).
Also would have problem identifying which is German/Russian, but perhaps somebody has related idea.

EDIT: I watched "My Name is Earl" season 1 DVD's a little while ago, is quite comical with English subtitles switched on,
they were obviously done by a Chinese person for whom English was their 17th language, sometimes little in common
with the original spoken English. [1st few episodes only, after that, was done by native English speaker]

EDIT: According to Wikipedia:- https://en.wikipedia.org/wiki/Liberation_(film_series)

Language Russian, German, English, Polish, Italian, French, Serbo-Croatian, Thai, Khmer, Vietnamese, Mien

Dont know if those languages are spoken in the movie, or movie is dubbed in those languages [presume spoken in movie because Spanish, Romanian, Portuguese missing from list].

And here on IMDB (there are quite a few movies of same name, Liberation [1971]):- https://www.imdb.com/title/tt0198811/

[Goggen240]

Wow, you're putting way more effort into this than I expected, good job, and thanks very much!
I have had basically no time at all to mess around with this project, but I'm crossing my fingers things will calm down a bit. I'd certainly like to read through the mountain of links, heh.

Originally Posted by StainlessS

What you are trying to do is incredibly difficult, and even if you do manage it, is likely to be a bit awful to listen to
(audio will probably sound completely different when switching from one to the other).
I simply would not even try to do it. [un-scrambling an egg might be easier]
EDIT: Although I guess that if there are long pauses between change of audio, then not so awful after all.

Apparently not!
I did a really quick and dirty experiment where I played the two versions side-by-side, and just muted one or the other. The scenes have long pauses of silence, and there's basically no background noise at all except in battle scenes. And those are all in Russian anyway. Plus the audio quality is pretty great for both versions I have.
Convenient thing I've noticed; In any scene that has both Russians and Germans, there's also translators present, so neither version does any dubbing at all.

I think the simple way to go is to just synchronize the audio tracks to whichever version has the best video, and then switch manually while watching. Most dialogue scenes are really long, and it's pretty obvious when a bunch of Germans and Russians show up which they're going to be speaking.

Originally Posted by StainlessS

For subtitles, you might find that someone has went to the bother of making a subtitles file, do a google search on

Subtitle "Name of movie"

EDIT: Apparently, there are subtitles in English, Russian, Spanish, Romanian, Portuguese.
(...)
Perhaps some kind of auto translate would suffice. SubtitleEdit has some such feature, but I've never tried it.

Actually, the official Mosfilm YouTube version has the English subtitles. I'm not sure if other DVD or Blu-ray releases are the same timing-wise, though. The English subs on the Cantonese DVD might sync better with the German Blu-ray video, for instance.
There's a bunch more languages of subtitles that definitely exist, but I'll be good as long as I have English. (Which I do, sorry you wasted your time...)

Originally Posted by StainlessS

EDIT: There are I believe tools to extract/remove human voice from stereo songs, where voice part is equally apparent in both channels,
sometimes used to remove voice for eg karaoke type stuff (In war movie, may not work too well what with all them guns and things).
Also would have problem identifying which is German/Russian, but perhaps somebody has related idea.

The major dialogue scenes are mostly generals-talking-in-big-silent-rooms, so they would be excellent for such a tool.
Identifying which is German or Russian cooould be possible. In the Russian dub, they start speaking German, and then the dubber talks over them. So if the Russian/German dubs match intermittently, that scene's in German, and if they don't match at all, that's Russian. Maybe something a tool like a voice-to-text subtitle maker could do...?

Probably not worth the effort over just switching manually for the one time I'll watch this thing...

*EDIT*
I didn't think of it, but the subtitles might be really useful. That's all the speech in the movie, and when it's spoken.
There's no obvious way to figure out what language they're supposed to be speaking though, but the when is solved.
And I suppose it could fit with the unrealistic autosubtitle idea, hmmm...

Originally Posted by StainlessS

EDIT: According to Wikipedia:- https://en.wikipedia.org/wiki/Liberation_(film_series)

Dont know if those languages are spoken in the movie, or movie is dubbed in those languages [presume spoken in movie because Spanish, Romanian, Portuguese missing from list].

The spoken languages I've noticed while scrubbing through are:
Russian (almost all)
German (almost all that's left)
English
Polish
Italian
Maybe more?

(There's a reeeal bad Churchill in there, but he's overdubbed in the two versions I've found.)

There might be a few more languages here and there; the movie was a Soviet / East German / Polish / Romanian / Italian / French co-production, but I don't imagine that there's going to be much Vietnamese on the Eastern Front.

Originally Posted by StainlessS

And here on IMDB (there are quite a few movies of same name, Liberation [1971]):- https://www.imdb.com/title/tt0198811/

Yep, there's five separate movies (one in two parts), roughly 1969-1971. But it's basically just a single reaaally long movie, it seems.

I also accidentally downloaded a movie of the same name also from the '70s:
Turns out "Liberation" and "Deliverance" is the same word.
Makes sense when you think about it, heh.

[Goggen240]

Progress report!
(In case anyone's interested...)

I'm still digging my way deeper and deeper into this Google-hole, and I've just started on the image-based filters suggested by StainlessS... (Thanks again!)
MatchClips and AutoOverlay look really interesting; that should synchronise the German to the Russian version, and get a little more image since the German version is framed further to the right than the Russian version.
Unfortunately, my AviSynth installation is messed up, so I'll have to fix that first.

Also, I've found both a Polish and a Czechoslovakian version.
These are both dubbed on top of the Russian soundtrack, continuing the fun. But the Polish is not dubbed over, but I'm not sure there's any Czecholovakian speech.

So now I have four soundtracks...
I did find clips of the Spanish and Thai versions; the Spanish is the subtitled Russian version, and the Thai is dubbed on top of the Russian version, so those are useless. I haven't found any French, Italian, Romanian or Finnish versions, but I'm not holding my breath...


I also did a test on auto subtitlers/transcribers. It failed.

Originally Posted by Goggen240

Identifying which is German or Russian cooould be possible. In the Russian dub, they start speaking German, and then the dubber talks over them. So if the Russian/German dubs match intermittently, that scene's in German, and if they don't match at all, that's Russian. Maybe something a tool like a voice-to-text subtitle maker could do...?

I tried this with a scene with English speech (the Teheran conference) and both the Russian and German dub is on top of English. The Russian dub lets the English line get almost to the end before overdub, and the German starts almost right away (but finishes earlier), and there's a lot of undubbed Russian speech in the Russian dub.

The experiment was to see if a voice-to-text thing could find common speech (English) and ignore overdub (German/Russian). Hypothetically that could have separated the two by speech. But since the actual Russian speech is dubbed over in German, that's a false negative.

So I spent a couple hours on this voice-to-text website:.
https://speech-to-text-demo.ng.bluemix.net/

And... Well...
Let's just say that the vegetable chips with Jim Belushi was problem.

My conclusion is that doing anything at all with the audio is probably never going to work. So a Scene Detector to sync up the two videos and switching manually is probably the best option, still.

If you do want to check my experiment, see below.

WARNING!
This is going to be a great wall of text!
Nobody (else) should read the whole thing!

So here's the Russian audio transcribed into English:
Italic means that there's no Russian overdub (English/Russian speech).
Bold means that the Russian transcript and the German transcript actually agree (English).
Underlined is where neither soundtrack has overdub.

Speaker 1:
    Let's not look where to put the blame Sir sooner or later we have met 
please never discussed on August but installing thank you national status however. 
Speaker 1:
    Where it not for that. 
Speaker 1:
    Threats have assassination from **** agent least opinion that's to ski again 
the sickle grows a portrait now as your guest yet nevertheless from custom Vikas thing you really. 
Speaker 2:
    Which is against the. 
Speaker 1:
    Sir if such a thing with a guy who was it uses just about hiding in the Russian 
media when if that'll set your little scheme you see there are wait here 
for fifty years what is he deals nest of Anya the mobile but it when is this after the battle of the Kursk bulge the Russians are in. 
Speaker 1:
    Title. 
Speaker 0:
    To special respect. 
Speaker 1:
    From the present mostly because the beats little scheme it but I would also be watching the president the sitting on the couch that. 
Speaker 0:
    Me rest utility. 
Speaker 0:
    Now US room with this. 
Speaker 0:
    Any issue please. 
Speaker 1:
    We'd better get down to. 
Speaker 0:
    Our problem. 
Speaker 1:
    The vegetable chips with Jim Belushi was problem.

Yeah, so that didn't work.
It never agrees on Russian speech, and barely ever agrees on English. All underlined speech should also be bold (and mostly isn't) and all non-underlined italics should also be bold (separated from German).
Still a fun experiment, but I'll switch over to experimenting on video only and keep chugging along.