Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 31st December 2008, 13:07   #1  |  Link
Morte66
Flying Skull
 
Morte66's Avatar
 
Join Date: Jan 2005
Posts: 397
Deinterlacing "Made In Hong Kong"

I've been working at deinterlacing my old Malaysian DVD of Fruit Chan's "Made in Hong Kong", since it doesn't look like a better version is going to go on sale any time soon. I'm stumped, and I'd appreciate some help/advice/commentary. I should say in advance that I don't particularly understand this stuff, so some of the things I've tried/concluded may be hilariously wrong...

Samples are here:
http://rapidshare.com/files/178424374/MIHK1.m2v
http://rapidshare.com/files/178424376/MIHK2.m2v
http://rapidshare.com/files/178424378/MIHK3.m2v

Here's what I've been through so far, not that it's made me any wiser...

The encode is 29.976 FPS in combed fields, and it's apparently made from a 24 FPS cinema source (e.g. exhibition print).

MeGUI says it's hybrid Film/Interlaced, mostly interlaced. The suggested deinterlacer works poorly, lots of flicker and some judder plus residual combing.

So I ran a two pass job with TFM/TDecimate to get timecodes and make a VFR file. It reported 32.30% film 67.70% video, the files are uploaded. The VFR file looked better, but it had some judder at times and some residual combing. Here are the diagnostics for the whole rip:
http://rapidshare.com/files/178424381/stats.txt
http://rapidshare.com/files/178424382/tfm.txt
http://rapidshare.com/files/178424383/timecodes.txt

Since it's NTSC and it really ought to be a telecine, I began to wonder whether it might be a mechanical telecine so awful that TFM just couldn't match the fields. I ran TFM (order=1).TDecimate() to see what happened, and got sharper video with lousy judder in places.

Finally I figured I'd run it through straight TDeint() for a lark. It gave me the smoothest video of all, was sharper than everything except the IVTC, and had residual combing within the abilities of vinverse(). I saw occasional ghosts, but rare enough that they might be bad edits rather than frame blending. This is the best looking output on balance, even though getting 29.976 FPS from a film is obviously wrong.

I wonder if anybody can come up with a better answer. All help gratefully received.
Morte66 is offline   Reply With Quote
Old 31st December 2008, 15:29   #2  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,922
You have blended fields. Look at deblending solutions.
Guest is offline   Reply With Quote
Old 1st January 2009, 12:48   #3  |  Link
Jawed
Registered User
 
Join Date: Jan 2008
Location: London
Posts: 156
It's 25fps that's been butchered to 29.976fps.

So you can:

Code:
bob() # use your favourite bobber
RePAL()
or you can use:

Code:
SeparateFields()
ScriptClip("subtitle(string(current_frame))") 
n=-167
selectevery(199,n+0  ,n+3  ,n+5,  n+8,  n+10, n+12, n+15, n+17, n+20, n+22, n+24, n+27, n+29,
\               n+32 ,n+34 ,n+36, n+39, n+41, n+44, n+46, n+48, n+51, n+53, n+56, n+58,
\               n+60 ,n+63 ,n+65, n+68, n+70, n+72, n+75, n+77, n+80, n+82, n+84, n+87, n+89,
\               n+92 ,n+94 ,n+96, n+99, n+101,n+104,n+106,n+108,n+111,n+113,n+116,n+118,
\               n+120,n+123,n+125,n+128,n+130,n+132,n+135,n+137,n+140,n+142,n+144,n+147,n+149,
\               n+152,n+154,n+156,n+159,n+161,n+164,n+166,n+168,n+171,n+173,n+176,n+178,
\               n+180,n+183,n+185,n+188,n+190,n+192,n+195,n+197)

bob() # replace with bobber of choice
trim(69,0) # remove surplus frames at start
which looks pretty good. There are some residual blends, e.g. around 1h12m. I suppose you could try to find the ends of the reels and re-sync on each reel start, hohoho.

I generated this by examining frames (fields from the original video file) 656-1105, creating a list of desired frames and then searching for the cycle. Then I manually searched for the offset, n. All fairly tedious...

The result is a file that plays for exactly the same time (1h44m10.289s) as the original video (1h44m10.277s). The RePAL() version plays for 1h44m10.244s.

Jawed
Jawed is offline   Reply With Quote
Old 1st January 2009, 13:11   #4  |  Link
Jawed
Registered User
 
Join Date: Jan 2008
Location: London
Posts: 156
Having played momentarily with yadif() to bob this in these two scripts, my use of bob() in the second script is a bit different...

My main interest was in identifying the good fields (least motion artefacts) after separating them, not in getting the best bobbing/height-doubling/field-jitter-cleanup.

Jawed
Jawed is offline   Reply With Quote
Old 2nd January 2009, 11:46   #5  |  Link
Morte66
Flying Skull
 
Morte66's Avatar
 
Join Date: Jan 2005
Posts: 397
Thanks.

Either of these seemed to work fairly well...

Code:
yadif(mode=3)
repal()
Code:
yadif(mode=3)
SeparateFields()
n=-167
selectevery(199,n+0  ,n+3  ,n+5,  n+8,  n+10, n+12, n+15, n+17, n+20, n+22, n+24, n+27, n+29,
\               n+32 ,n+34 ,n+36, n+39, n+41, n+44, n+46, n+48, n+51, n+53, n+56, n+58,
\               n+60 ,n+63 ,n+65, n+68, n+70, n+72, n+75, n+77, n+80, n+82, n+84, n+87, n+89,
\               n+92 ,n+94 ,n+96, n+99, n+101,n+104,n+106,n+108,n+111,n+113,n+116,n+118,
\               n+120,n+123,n+125,n+128,n+130,n+132,n+135,n+137,n+140,n+142,n+144,n+147,n+149,
\               n+152,n+154,n+156,n+159,n+161,n+164,n+166,n+168,n+171,n+173,n+176,n+178,
\               n+180,n+183,n+185,n+188,n+190,n+192,n+195,n+197)
Morte66 is offline   Reply With Quote
Old 2nd January 2009, 16:11   #6  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Quote:
Originally Posted by Jawed View Post
The result is a file that plays for exactly the same time (1h44m10.289s) as the original video (1h44m10.277s). The RePAL() version plays for 1h44m10.244s.

Jawed
Hey Jawed,

You seem to be an expert in this field, could you help me as well?
I have a video I can't get along with: http://www.badongo.com/file/12382006 (sample, 12 Mbytes)
(To remain faithful to the topic, this movie is also made in Hong Kong )

I'm 100% positive it's a 25->29.97i conversion too (running time is PAL time). Last time I've tried fixing with RestoreFPS with no luck, I couldn't find the right phase.
zee944 is offline   Reply With Quote
Old 2nd January 2009, 18:27   #7  |  Link
Jawed
Registered User
 
Join Date: Jan 2008
Location: London
Posts: 156
I've never heard of RestoreFPS

I would recommend you try:

Code:
yadif(mode=3)
repal()
I tried it here, it didn't look perfect but it does get rid of a lot of ghosting. You might like it better than other things you've tried.

Today I discovered that China is a PAL country (while Hong Kong and Taiwan are NTSC). That might explain a good portion of all the blended crap out there...

Jawed
Jawed is offline   Reply With Quote
Old 2nd January 2009, 21:04   #8  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Quote:
Originally Posted by Jawed View Post
I tried it here, it didn't look perfect but it does get rid of a lot of ghosting. You might like it better than other things you've tried.
I tried it first but I didn't believe it would work, and the result seemed jerky indeed, so I sent my post. At my second try I've realized only the playback was jerky. Man, this totally blew me away. It's almost perfect. I can't believe my eyes.

One little problem though: RePal decimates a bit more fields than it'd needed. My 29.97i source from DVD is 179,565 frames long. I'm 100% sure it is not a 25->30, but a direct 25->29.97i conversion. So I'm also sure the PAL version should be 179,565*(25/29.97)--> 149,787 frames. But RePal decimates it to 179,565*(25/30) --> 149,637 frames. Any idea how to fix it?

But this is really just for the perfectionism. The result is already unbelieveable. Thank you very much, Jawed.
zee944 is offline   Reply With Quote
Old 2nd January 2009, 23:01   #9  |  Link
Morte66
Flying Skull
 
Morte66's Avatar
 
Join Date: Jan 2005
Posts: 397
I think what you'll find is that RePAL has converted from 29.976 (i.e. 30000/1001) to 24.975 (i.e. 25000/1001), and flagged the output as such.

If you're using a PC for playback then you can just play your 149,637 frames at the indicated 24.975fps and they'll have the same duration as 149,787 frames at 25fps would (or the original 179,565 at 29.976fps).

If you need 25fps, e.g. to make a DVD to play in a standalone, then I suggest you treat the RePAL output video as 25fps which will make it slightly shorter. Then use AviSynth or a program like BeLight to very slightly speed up the audio (by 1001/1000) for synch and resample it to 48KHz for DVD authoring.
Morte66 is offline   Reply With Quote
Old 2nd January 2009, 23:03   #10  |  Link
Jawed
Registered User
 
Join Date: Jan 2008
Location: London
Posts: 156
I'm glad that helped, it's always a great moment when something comes out better than expected.

The shortfall in framecount in the RePAL processed version is caused by the extremely long cycle of the conversion from PAL to NTSC, which RePAL tries to undo. Basically there's 1 too many frames deleted by RePAL, in a cycle that lasts 1001 frames!

The complicated processing I did with SelectEvery(199,...) gets the correct number of frames. So in Made in Hong Kong, RePAL is 156101 frames while SelectEvery is 156259.

If you really need PAL framerate (e.g. to burn a DVD in PAL) then you could change the length of the film. That would mean changing the framerate of the video and adjusting the audio.

Or, you can undertake the misery of working out "n", while using the SelectEvery function as I've written it. In the film Made in Hong Kong, there's a long long sequence where the camera tracks along a wire fence in close-up. The wire shapes of this fence, with their reasonably smooth left-right movement, made working out "n" relatively easy

Jawed
Jawed is offline   Reply With Quote
Old 3rd January 2009, 00:06   #11  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Morte66, thanks, but speeding up wouldn't be a problem. I'm worrying about the few lost frames.

Quote:
Originally Posted by Jawed View Post
The complicated processing I did with SelectEvery(199,...) gets the correct number of frames. So in Made in Hong Kong, RePAL is 156101 frames while SelectEvery is 156259.
I'm confused about this SelectEvery() thing. If we leave out Yadif() and especially RePal() from the script, how we get rid of ghosting? There's no chance to eliminate as much ghosting as Yadif()+RePal() does whatever 'n' value I try, is there?
zee944 is offline   Reply With Quote
Old 3rd January 2009, 01:40   #12  |  Link
Jawed
Registered User
 
Join Date: Jan 2008
Location: London
Posts: 156
Hmm, I was a bit imprecise earlier, the code you should use is:

Code:
source=last
yadif(mode=3)

# These three lines reduce some of the interlacing-induced flicker on sharp horizontal edges
spline36resize(last.width,last.height*2)
vinverse()
spline36resize(source.width,source.height)

# this must be adjusted for each clip - start with 0 and try 
# other values, keep negative to avoid chopping off start of clip
n=-167

# the following line should always work
selectevery(199,n+0  ,n+3  ,n+5,  n+8,  n+10, n+12, n+15, n+17, n+20, n+22, n+24, n+27, n+29,
\               n+32 ,n+34 ,n+36, n+39, n+41, n+44, n+46, n+48, n+51, n+53, n+56, n+58,
\               n+60 ,n+63 ,n+65, n+68, n+70, n+72, n+75, n+77, n+80, n+82, n+84, n+87, n+89,
\               n+92 ,n+94 ,n+96, n+99, n+101,n+104,n+106,n+108,n+111,n+113,n+116,n+118,
\               n+120,n+123,n+125,n+128,n+130,n+132,n+135,n+137,n+140,n+142,n+144,n+147,n+149,
\               n+152,n+154,n+156,n+159,n+161,n+164,n+166,n+168,n+171,n+173,n+176,n+178,
\               n+180,n+183,n+185,n+188,n+190,n+192,n+195,n+197)

# Adjust trim so that the clip starts in the right place
trim(69,0)
Put that in your script in place of the yadif() and repal() lines, as this other technique is a complete replacement.

This selectevery() statement is doing the same kind of thing as RePAL. It's throwing away fields. RePAL seeks out the fields with the least blending. selectevery() is just a dumb pattern used to extract fields - but it can at least get you back to 25fps. That pattern needs to be lined up with how the video was converted, which is why n needs to be adjusted. You have to seek out the correct phase.

There's 199 different values for n, sadly, and only 1 will work. And if the PAL->NTSC conversion was done one reel of film at a time (admittedly, extremely unlikely) and the video of the separate reels joined together, the value of n could be different for every reel.

So, it's a lot of grief for, at best, the most minor improvement over RePAL.

Jawed

Last edited by Jawed; 3rd January 2009 at 01:42.
Jawed is offline   Reply With Quote
Old 3rd January 2009, 13:21   #13  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Quote:
Originally Posted by Jawed View Post
This selectevery() statement is doing the same kind of thing as RePAL. It's throwing away fields. RePAL seeks out the fields with the least blending. selectevery() is just a dumb pattern used to extract fields - but it can at least get you back to 25fps. That pattern needs to be lined up with how the video was converted, which is why n needs to be adjusted. You have to seek out the correct phase.
Oh, I see know. Perhaps the resulting video of each 'n' value could be analyzed somehow. How could we decide if a frame is blended or not? If we could, then the result with the least blended fields would be the right one.
zee944 is offline   Reply With Quote
Old 3rd January 2009, 14:01   #14  |  Link
manono
Moderator
 
Join Date: Oct 2001
Location: Hawaii
Posts: 7,349
Quote:
Originally Posted by zee944 View Post
Morte66, thanks, but speeding up wouldn't be a problem. I'm worrying about the few lost frames.
There aren't any lost frames. RePAL gives the correct framerate of 24.975fps.
manono is offline   Reply With Quote
Old 3rd January 2009, 14:27   #15  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Quote:
Originally Posted by manono View Post
There aren't any lost frames. RePAL gives the correct framerate of 24.975fps.
Why are you sure there are no lost frames, manono?
I'm pretty convinced the original PAL source was 149,787 frames and not 149,637.
zee944 is offline   Reply With Quote
Old 3rd January 2009, 14:53   #16  |  Link
manono
Moderator
 
Join Date: Oct 2001
Location: Hawaii
Posts: 7,349
I might ask you what makes you so sure you know the original frame count? Do you know within a margin of 6 seconds the original PAL length?

Anyway. Is 24fps film 24fps on an NTSC DVD? No, it's 23.976fps (after being Force Filmed, IVTC'd, or unblended back to its base framerate).
Is 30fps video 30fps on an NTSC DVD? No, it's 29.97fps.
By the same token, a 25fps PAL source becomes 24.975fps on an NTSC DVD (after being unblended or "IVTC'd" back to its base framerate).
manono is offline   Reply With Quote
Old 3rd January 2009, 15:17   #17  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Quote:
Originally Posted by manono View Post
I might ask you what makes you so sure you know the original frame count? Do you know within a margin of 6 seconds the original PAL length?
I have the movie from a different, also uncut 25 fps source. I've compared six longer scenes (counted the frames actually). In 5 of 6 cases, the numbers showed the 29.97i source had 19.88% more frames for the same scene, instead of 20.00%. Moreover, at first I assumed it was a 25->30 conversion, but calculating with that, the figures were only close to perfect. After I've changed my assumption to 25->29.97, my calculations became 99.5% accurate.
zee944 is offline   Reply With Quote
Old 3rd January 2009, 17:18   #18  |  Link
manono
Moderator
 
Join Date: Oct 2001
Location: Hawaii
Posts: 7,349
If the framerate is really supposed to be 25fps, then there shouldn't be any duplicate frames (given a perfect unblend). And using RePAL, there should be a missing frame roughly every 40 seconds, producing a noticeable jump or stutter.

My contention is that making it 25fps will produce duplicate frames. I've used RePAL probably more than anyone else on the planet (many hundreds of times, approaching 1000 now), and there was only one instance where I decided it should really be 25fps. Even in that case the problem could have been a not-so-good unblend.

Because of the drop frame flag 30fps becomes 29.97fps, 24 becomes 23.976fps, and 25 becomes 24.975fps. You can read a layman's explanation here:

http://www.doom9.org/synch.htm

However, if you'd like to make it 25fps, SRestore can easily do it, and there's also a modified version of RePAL floating around somewhere that unblends for 25fps. I briefly looked for it and couldn't find it, but I'm sure someone can point to it.
manono is offline   Reply With Quote
Old 3rd January 2009, 18:08   #19  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
Quote:
Originally Posted by manono View Post
And using RePAL, there should be a missing frame roughly every 40 seconds, producing a noticeable jump or stutter.
There really are missing frames in the resulting video. Between 9549..9550 and 10504..10505. Although the difference isn't accurately one thousand frames, it must be it.These missing motions are there in the original 29.97 and in the other 25 fps source.

I see now where are you coming from. All conversion you've done back was a 25->30 job, and mine is an exception.

You mean I could substitute RePAL with SRestore? I thought RePal is something exceptional.

(Is this the modified RePAL you told me about?)

Last edited by zee944; 3rd January 2009 at 18:13.
zee944 is offline   Reply With Quote
Old 3rd January 2009, 18:48   #20  |  Link
zee944
Registered User
 
Join Date: Apr 2007
Posts: 240
The modified (29.97 Hz) RePAL isn't working right. It selects blended frames, the same when I tried to use ChangeFPS(30) or ChangeFPS(60) after Yadif with the original RePAL.

...While SRestore(Frate=25) seems to be perfect. I haven't done enough testing yet, but the frame count is right (149,787) , the video seems progressive etc. If this is true, let me ask, why do you use RePal() if SRestore is for the same thing and capable of more?
zee944 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:38.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.