Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th June 2019, 08:04   #1  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 634
Interlaced encoded as progressive and repeated fields

Hi,
I'm in troubles.
The service who shot and edited a beach soccer match delivered a mess: the file has been encoded (by them) in H.264 60fps progressive, but it's actually interlaced.
The problem is that fields are screwed up and I can't solve the problem by simply bobbing everything.
By applying plain and brute bobbing, I notice that chroma goes outside the luma boundaries and it looks simply awful. TDeint fails to deinterlace it properly and creates many artifacts with still noticeable fields in the picture. Libav Deinterlace with linear blending gives a somewhat better result, however I noticed that there are still repeated frames left.

Here's why I'm in trouble: air time is tomorrow and my target is 25i, so either 25fps progressive flagged as interlaced or 50fps progressive divided in fields to get a truly interlaced 25i. I've already contacted the service to send another file that is not so screwed up, however they won't be able to send it 'till tomorrow, therefore I gotta try to make something out of what I currently have.

I have uploaded a few sample: a lossless uncompressed one (but beware that the source they sent was already compressed) of 10 and 20 seconds and a long sample that I re-encoded in H.265 10bit leaving everything as it was (fields as they were) so that you can play with it.

Long Sample (lossy): https://we.tl/t-qaUrbdD8ns
Short Sample 1 (lossless): https://we.tl/t-yuNOTvgnzG
Short Sample 2 (lossless): https://we.tl/t-IUKdRvriLf
Short Sample 3 (lossless): https://we.tl/t-dQdrJm6Mq0
Short Sample 4 (lossless): https://we.tl/t-bocAED6sJb

Thank you in advance,
I have faith in you,
Frank.
__________________
Broadcast Encoder
LinkedIn
FranceBB is offline   Reply With Quote
Old 18th June 2019, 09:03   #2  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 957
There is probably a better way. But this looks at least more or less ok. Or lets say it a start...

Code:
clip = core.lsmas.LWLibavSource(source=r"D:\Download\Sample_short.mkv")
clip0 = clip.std.SeparateFields(0).std.SelectEvery( cycle=2, offsets=0)
clip1 = clip.std.SeparateFields(1).std.SelectEvery( cycle=2, offsets=1)

clipi = core.std.Interleave([clip0, clip1]).std.SelectEvery( cycle=2, offsets=0)
clip = core.std.DoubleWeave(clipi, tff=True)

clip = haf.QTGMC(clip, TR2=1, TFF=False).std.SelectEvery( cycle=2, offsets=0)
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database || https://github.com/avisynth-repository

Last edited by ChaosKing; 18th June 2019 at 09:12.
ChaosKing is offline   Reply With Quote
Old 18th June 2019, 09:42   #3  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 634
Quote:
Originally Posted by ChaosKing View Post
There is probably a better way. But this looks at least more or less ok. Or lets say it a start...

Code:
clip = core.lsmas.LWLibavSource(source=r"D:\Download\Sample_short.mkv")
clip0 = clip.std.SeparateFields(0).std.SelectEvery( cycle=2, offsets=0)
clip1 = clip.std.SeparateFields(1).std.SelectEvery( cycle=2, offsets=1)

clipi = core.std.Interleave([clip0, clip1]).std.SelectEvery( cycle=2, offsets=0)
clip = core.std.DoubleWeave(clipi, tff=True)

clip = haf.QTGMC(clip, TR2=1, TFF=False).std.SelectEvery( cycle=2, offsets=0)

I'm not familiar with VapourSynth.
Translated in Avisynth language would be:

Code:
#Indexing
LWLibavVideoSource("Sample.mkv")

#Separate fields
clip0=SeparateFields(0).SelectEvery( cycle=2, offsets=0)
clip1=SeparateFields(1).SelectEvery( cycle=2, offsets=1)

#Interleaving both fields
Interleave(clip0, clip1)
SelectEvery(cycle=2, offsets=0)

#Weave
DoubleWeave(tff=True)

#Deinterlace
QTGMC(TR2=1, TFF=False)

SelectEvery(cycle=2, offsets=0)
Right? I'm gonna try it asap.
__________________
Broadcast Encoder
LinkedIn
FranceBB is offline   Reply With Quote
Old 18th June 2019, 10:04   #4  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 957
I think this should be the correct translation:

Code:
ffms2("D:\Download\Sample_short.mkv")

clip0=AssumeBFF().SeparateFields().SelectEvery(2, 0)
clip1=AssumeTFF().SeparateFields().SelectEvery(2, 1)

#Interleaving both fields
Interleave(clip0, clip1)
SelectEvery(2, 0)

#Weave
AssumeTFF()
DoubleWeave()

#Deinterlace
QTGMC() ## use a preset here!
SelectEven()

I also noticed that the chroma has blends in SeparateFields(). Not sure what a good fix would look like. And I think every 6th frame is a double frame!? But it doesnt seems to be a fixed pattern.


EDIT:
I tested just qtgmc alone again and now it doesn't look "jumpy" like the first time. I don't understand why... the code above is unnecessary and not entirely correct anyway. I will post new code soon.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database || https://github.com/avisynth-repository

Last edited by ChaosKing; 18th June 2019 at 11:02.
ChaosKing is offline   Reply With Quote
Old 18th June 2019, 11:56   #5  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 1,804
What a mess. I can't see any pattern to the repeats.

Try this, though:

Code:
tdecimate(mode = 1, cycler = 7, cycle = 12)
You could also try mode = 0, or cycler = 35 and cycle = 60, although that will be a lot slower.

Unfortunately tdecimate only works in 8 bit, and there will always be remaining dupes which result in a stutter, but I think it's as good as you're going to get unless I can work out the exact dupe pattern (it's so random I think it was done analytically).

Also the chroma is completely blended so there's no hope of fixing that.

Also tell them they suck at compositing!
__________________
My AviSynth filters / I'm the Doctor

Last edited by wonkey_monkey; 18th June 2019 at 11:59.
wonkey_monkey is offline   Reply With Quote
Old 18th June 2019, 12:10   #6  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 1,804
ChaosKing:

Code:
clip0=AssumeBFF().SeparateFields().SelectEvery(2, 0)
clip1=AssumeTFF().SeparateFields().SelectEvery(2, 1)
I'm not sure what your intent was, but clip0 and clip1 are identical.
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 18th June 2019, 13:26   #7  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,012
Quote:
Originally Posted by wonkey_monkey View Post
there will always be remaining dupes which result in a stutter
Duplicty2/DropDeadGorgeous could get rid of that [EDIT: at least with 8 bit].

from:- https://forum.doom9.org/showthread.p...08#post1873808

Quote:
Perhaps this explains better
Code:
abcXdefgDDhi       Src, Two dupe frames preceded (within a set range) by a jerk frame. Jerk frame X, has greater motion to its predecessor than do d, e, f, or g.
abc12Xdefghi       DropDeadGorgeous dupe removal and Jerk interpolation

Lower case non dupe non Jerk
X     = Frame of greatest motion from predecessor (Jerk)
D     = Dupe Frames
Digit = Interpolated frames between X and its predecessor c (Same Count as D dupes)
EDIT:
Duplicity/DropDeadGorgeous only 8 bit, but could convert to 8 bit for duplicity DBase creation,
and hack DropDeadGorgeous script for 2nd pass with 10 bit and DropDeadGorgeous().
Code:
Function ZZ_AverageLuma(clip c, int "n") {   ### Replace RT_AverageLuma() calls to ZZ_AverageLuma() EDIT: Simple Search/Replace in text editor.
    c # EDIT: Oops, added
    current_frame = Default(n,current_frame)
    return AverageLuma
}
Untested, should work I think
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 18th June 2019 at 13:51.
StainlessS is offline   Reply With Quote
Old 18th June 2019, 13:50   #8  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 1,804
No interpolation is needed, just precise removal of exact (barring compression differences) duplicates. But there might be some scenes with static graphics, so there needs to be a limit on how many frames are removed per second, at least on average.

It seems that, at most, there are three duplicate frames in a row (original correct frame, plus three duplicates).
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 18th June 2019, 13:54   #9  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,012
Perhaps, but given the alternative of [EDIT: left over dupes/] jerks, might be an option.
Duplicity has MaxDupeLen (where if greater then is presumed static scene).
EDIT: Duplicity does not have a 1 in n decimation mode [well, when creating Frame files, it does not actually decimate at all].
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 18th June 2019 at 14:05.
StainlessS is offline   Reply With Quote
Old 18th June 2019, 16:52   #10  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,865
You mentioned h.264 60fps progressive, but what was it? 4:2:2 8bit h264 ? 4:2:0 ?

The short sample is 8bit 4:2:2 ffv1, but long sample is 10bit 4:2:0

Why didn't you just upload the cut original ?
- smaller filesize compared to ffv1
- less chance of introducing more errors
a) maybe other issues like decoding, timecodes causing the problem, such as repeat fields. Maybe other decoding methods are better for the actual source ? Certain versions of ffms2 / lsmash/ ffmpeg are known to give problems , e.g. interlaced AVC transport streams
b)possibly introducing other interlaced/progressive up/downsampling errors exacerbating the chroma ghosting .

It probably doesn't matter here, but you should always upload the original , untouched as much as possible
poisondeathray is offline   Reply With Quote
Old 18th June 2019, 20:21   #11  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 634
Quote:
Originally Posted by poisondeathray View Post
You mentioned h.264 60fps progressive, but what was it? 4:2:2 8bit h264 ? 4:2:0 ?

The short sample is 8bit 4:2:2 ffv1, but long sample is 10bit 4:2:0
The original is H.264 4:2:0 yv12 8bit planar "fake" 60fps, encoded as progressive but with an interlacing mess inside. I encoded a few lossless samples to give you an idea of what it was and a lossy one 'cause I wouldn't have been able to upload the original one. Still, in this case, ffmpeg didn't introduce any new weird decoding issue and I left the fields exactly as they were.

For everyone: I've tried the code posted above and I encoded a few samples. I also tried various modes and parameters of tdecimate, chained tdecimate, single tdecimate, SRestore for dups only, ApparentFPS, Duplicity2 and I spent hours and hours on this flipping mess. I even tried to manually detect and trim out dups and then blend the missing frames to get a truly interlaced 25i (50fps) and although it took a lot of time, the result was promising. In order to try to get transition between missing frames smoother, I even tried linear interpolation but it was so inconstant that it was creating artifacts and it wasn't worth it. I called the producer and I showed them each and every result of the various encoding test, but he said that the "automatic solutions" were not good and although the manually decimated and blended one was promising, he recognised that it was too much effort to do it for a whole game of over 1h (I did it for several minutes but I was far from the end and my eyes were crossing already...).
Anyway, the producer informed the playout that we're gonna miss the scheduled deadline and that he's gonna talk to the service. (I don't know about what contract we have with them; it's not my field, I'm an encoder).

Alright, I would like to thank you all for the support as you tried to help me (well, "help us") and my boss is also thanking you as he knows about doom9 and he's totally in favour of open source and reciprocal help.

Thank you all anyway.

Alright, fellas, time to go home now and have dinner.
What a day at work today has been...

EDIT: They actually managed to send the real untouched masterfile in AppleProRes during the night and we encoded it straight away so we will be able to air in today, which also made me solve the "mystery" of what happened to the provided file: the original masterfile was an 25i truly interlaced (50fps) 4:2:2 yv16 10bit planar, however it has been edited by the guy who sent the file in a timeline that was set as 60fps progressive and that screwed up the fields. Then, he also brought it from 10bit to 8bit by truncation and it converted it from 4:2:2 planar to 4:2:0 planar while interlaced but treating it as progressive, which made the chroma expand from the fields to the frames... What a mess. Luckily the original untouched master has been provided just in time.
__________________
Broadcast Encoder
LinkedIn

Last edited by FranceBB; 19th June 2019 at 06:21.
FranceBB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:12.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.