Strange FILM/VIDEO Percentage in Captured Cable/Broadcasts Shows

[CarlEdman]

I have been transcoding digital (mostly HD) MPEG-2 broadcasts for archiving in a more efficient format (i.e., x264 -> h264) and the FILM/VIDEO percentage (as reported by dgindex) somewhat puzzles me.

1. Many broadcasts have percentages like you'd expect (i.e., close to 100% video or 100% film). Fine.

2. Some broadcasts have 90%-100% film--I can understand that as film material with perhaps a like interlaced/progressive format intermixed for CGI/etc.

3. However, many broadcasts have very weird reported mixtures of 40-60% FILM/VIDEO.

A few possibilities can be excluded:

1. I am not talking about intermixed other material (like commercials). All of these materials are pure movies/tv shows with the extraneous material at the beginning or end cut off.

2. I'd be very surprised if there actually was a substantial intermixture of FILM/VIDEO in the source material. They are typically theatrically released movies with little or no noticeable CGI, or materials which have been fully generated in the same manner (e.g., fully CGI movies or hand-drawn animation).

3. I'm not talking about beginning or end material (e.g., opening or closing credits). Lots of material will hoover around 60% VIDEO or FILM during its entire run.

So how am I getting these weird percentages? Do broadcasters (or cable movie channels) partially hard-telecine 24p movies?

And what is the proper way of treating such material? I have been tfm.tdecimating them (after honoring pull-down flags in dgindex) on the assumption that the material was just partially hard-telecined, but I am not sure.

[Blue_MiSfit]

It's quite common for all material to have hard 3:2 pulldown. I've never seen a scenario as you describe. Broken cadence, sure...

Is this all 1080i60 stuff?

[CarlEdman]

Yes, it is all 1080i60 material. Good to know that hard 3:2 pulldown is not uncommon (which presumably will be fixed by tfm.tdecimate).

[MrVideo]

Originally Posted by CarlEdman

Yes, it is all 1080i60 material. Good to know that hard 3:2 pulldown is not uncommon (which presumably will be fixed by tfm.tdecimate).

All 23.976/24 fps source material is transferred to 29.97i/59.94p video via 2:3 pulldown.

And yes, if you want to get it back to 23.976, you will have to inverse telecine it.

You might want to look into the author's newer dgindexnv program: http://www.neuron2.net/dgdecnv/dgdecnv.html

I don't pay any attention to video/film content values. You have to know you source material.

2. Some broadcasts have 90%-100% film--I can understand that as film material with perhaps a like interlaced/progressive format intermixed for CGI/etc.

No such thing. All movies are produced at 23.976 fps, or true 24 fps. As mentioned above, when transferred for broadcast, 2:3 pulldown is used.

Keep in mind that pretty much all scripted TV shows are produced at 23.976 fps, or PsF. So, the 1080p23.976 masters are converted to 1080i29.97 using 2:3 pulldown. Same thing if converted to 720p59.94.

[Ghitulescu]

Animes, I presume? Then it's normal, go hunt for deinterlacing techniques specifically used for animes.

[jmac698]

I think you're missing something here, it's normal for shows to be time-compressed so random fields may be dropped. You can get some extremely odd patterns and the only way to IVTC them is to count through a hundred frames by hand and look for it.

[CarlEdman]

Originally Posted by MrVideo

All 23.976/24 fps source material is transferred to 29.97i/59.94p video via 2:3 pulldown. And yes, if you want to get it back to 23.976, you will have to inverse telecine it.

I understand that this is necessary. However 2:3 pulldown can be done soft- (i.e., by marking certain fields as identical to previous ones and not repeating them) or hard (i.e., by repeating the extra fields in identical or nearly identical form). The former can easily be detected by dgindex and is, if I understand correctly, what is reflected in the dgindex reported FILM/VIDEO percentage. The latter can only be detected by visual inspection compensated for by cleverer teleciders (like tfm.tdecimate).

You might want to look into the author's newer dgindexnv program: http://www.neuron2.net/dgdecnv/dgdecnv.html

I have and would switch, except:
(1) I have a Radeon GPU, rather than an NVidia one, so dgdecnv would not be accelerated.
(2) Dgindex (or dgdecode) barely manages to move my CPU usage from idle, even with 1080p material being served off a very fast (400-500 MByte/second) SSD. Any performance improvement in dgdecnv would be negligible, compared to the time consumed by other filters and x264 encoding.
(3) Dgdecnv costs money. While I have used dgindex so much over the years that I'd gladly pay several times for having a few minor bugs fixed or features add (and in fact have offered bounties to do so), I don't really want a buy a program that I can't use and that wouldn't help me at all, even if I could.

I don't pay any attention to video/film content values. You have to know you source material.

I have used VIDEO/FILM content percentages for years to choose the processing of DVD-content with great success. And it works just fine on a majority of broadcast content, just not all of it.

[CarlEdman]

Originally Posted by Ghitulescu

Animes, I presume? Then it's normal, go hunt for deinterlacing techniques specifically used for animes.

A small number of animes, but far from all or even most cases, in which I've observed this.

[CarlEdman]

Originally Posted by jmac698

I think you're missing something here, it's normal for shows to be time-compressed so random fields may be dropped. You can get some extremely odd patterns and the only way to IVTC them is to count through a hundred frames by hand and look for it.

Really? When paid, premium movie channels broadcast commerical-free high-def versions of recent movies, they do stuff this hackish just to make them fit neatly into time-slots? That is very disappointing and in such a caase, I have no idea how to properly IVTC, much less keep the audio/subtitle/etc. information.

[jmac698]

http://forum.doom9.org/showthread.php?t=122472&page=3

Jericho ... it's keeping 38(dropping 9) out of every 47 frames, which would work out to 24.231 FPS ... on CBS

Also problems with Grey's Anatomy on NBC, etc.
And I agree, sometimes the flags aren't accurate, it's hard telecined.

[Ghitulescu]

Originally Posted by CarlEdman

Really? When paid, premium movie channels broadcast commerical-free high-def versions of recent movies, they do stuff this hackish just to make them fit neatly into time-slots? That is very disappointing and in such a caase, I have no idea how to properly IVTC, much less keep the audio/subtitle/etc. information.

The broadcasters don't care about this stuff. They rely on HW gear, they are supposed to do the job, for the rest - who cares?

Secondly, at least in Germany, there's not enough bandwidth to carry out HD audio (there are several HD channels on the same transponder), and the video is reencoded to a much lower bitrate than on a Blu-ray. They simply feed the stream/s into a multi-stream encoder (an HW encoder that encodes several programmes at once to be directly aired on a single transponder, or on the allocated bandwidth, for subrented transponders) then air it.

Who cares about the quality of a premium channel? You? You are even not suppose to record them in the first place.

And if a restoration has been performed (ie someone took a lil'bit more care about the settings), well, this is a marketing tool, a bonus. "I do my job lousily, I'll get my paycheck. I'll do my job well, hmm, I'll ask you twice the money". Why would they do this for almost nothing?

[MrVideo]

Originally Posted by CarlEdman

I understand that this is necessary. However 2:3 pulldown can be done soft- (i.e., by marking certain fields as identical to previous ones and not repeating them) or hard (i.e., by repeating the extra fields in identical or nearly identical form). The former can easily be detected by dgindex and is, if I understand correctly, what is reflected in the dgindex reported FILM/VIDEO percentage. The latter can only be detected by visual inspection compensated for by cleverer teleciders (like tfm.tdecimate).

For material that comes from a DVD, you are correct, soft marking can be done. But, for broadcast material, either OTA, cable, or DBS, that can't be done. Each and every field (for 1080i) and each and every frame (for 720p), must be there, so only hard 2:3 pulldown can be used.

(1) I have a Radeon GPU, rather than an NVidia one, so dgdecnv would not be accelerated.

The cards are cheap.

(3) Dgdecnv costs money. While I have used dgindex so much over the years that I'd gladly pay several times for having a few minor bugs fixed or features add (and in fact have offered bounties to do so), I don't really want a buy a program that I can't use and that wouldn't help me at all, even if I could.

$15 isn't going to break the bank and it is a one-time cost for all updates to the program.

When you start dealing with H.264 sources, DGdecNV will be a must. DGindex will not handle H.264 video.

I have used VIDEO/FILM content percentages for years to choose the processing of DVD-content with great success. And it works just fine on a majority of broadcast content, just not all of it.

DVD content != Broadcast content, as stated above.

As mentioned previous, all scripted dramas are 2:3 pulldown converted for air. You can count on that being the case. I've yet to find one that wasn't.

[MrVideo]

Originally Posted by jmac698

I think you're missing something here, it's normal for shows to be time-compressed so random fields may be dropped. You can get some extremely odd patterns and the only way to IVTC them is to count through a hundred frames by hand and look for it.

The only network that time compresses a lot of their scripted shows is ABC. They definitely screw up the 2:3 cadence with the crap they pull.

I don't watch CBS, so I do not know about them. NBC, FOX and The CW do not time compress. The 2:3 cadence is intact for each segment.

I've not heard any complaints about the major cable sources.

[MrVideo]

Originally Posted by CarlEdman

Really? When paid, premium movie channels broadcast commerical-free high-def versions of recent movies, they do stuff this hackish just to make them fit neatly into time-slots?

I've not heard of the Pay channel times compressing. I don't know about movies, but original programming has not had any issues with IVTC.

[MrVideo]

Originally Posted by jmac698

Also problems with Grey's Anatomy on NBC, etc.

I've not tried to IVTC the few Jericho episodes that I got off air, yet. Most are from another source (better).

Sorry, but Grey's Anatomy is on ABC, the only offender that I know of the time compresses.

[MrVideo]

Originally Posted by Ghitulescu

Secondly, at least in Germany, there's not enough bandwidth to carry out HD audio (there are several HD channels on the same transponder), and the video is reencoded to a much lower bitrate than on a Blu-ray. They simply feed the stream/s into a multi-stream encoder (an HW encoder that encodes several programmes at once to be directly aired on a single transponder, or on the allocated bandwidth, for subrented transponders) then air it.

What are you considering HD audio? The only audio allowed in the states is AC3, DD2.0 or DD5.1.

What you are talking about is a stat-mux encoder/multiplexer.

You are even not suppose to record them in the first place.

Not true in the states. Look up the famous Sony-Betamax case.

[Vurbal]

Originally Posted by MrVideo

When you start dealing with H.264 sources, DGdecNV will be a must. DGindex will not handle H.264 video.

DgDecNV is not the only option for H.264. You can purchase DiAVC and DGAVCDecDI or use FFMS2, which is free but less reliable. Since I don't have an nVidia card right now, I've gone with DGAVCDecDI. The fact that I'll automatically have access to DGDecNV when I do eventually get a supported card made it an easy decision.

[jmac698]

I don't mean to sidetrack too much, but I don't see the point of dgindex anymore. Even a simple directshowsource works for me, and certainly ffms2 does. If it's a straight n-pass from start to finish, that is. Why bother with the extra step?

[jmac698]

Some mentions in that thread,

I'm running into these strange cadences, House has them, Bones has them, Men In Trees has them, etc...

and Lost. There's talk of ABC affiliates owned by Belo who convert from ABC's 720p to 1080i and messing things up.
Mentions of ABC doing slowdowns and speedups. TNT-HD. More shows: Grey's Anatomy, House, Lost, Jericho.
Findings that cadence breaks at scene changes.
The odd show on CBS messed up.

I might mention also that I found a technique to improve the quality from HD captures. Record it twice, then take the best frames from each. You need ffms2 to do this, to see if it's I,B, or P. I found a complicated pattern that depends on the movie and compression cadence. Usually the B after an I looks good.

[CarlEdman]

Originally Posted by MrVideo

For material that comes from a DVD, you are correct, soft marking can be done. But, for broadcast material, either OTA, cable, or DBS, that can't be done. Each and every field (for 1080i) and each and every frame (for 720p), must be there, so only hard 2:3 pulldown can be used.

Really? So what does it mean if dgindex reports that a broadcast captured show is, e.g., 90% or 100% FILM, as it sometimes does. How does dgindex detect that if there are no pulldown flags in broadcast content?

When you start dealing with H.264 sources, DGdecNV will be a must. DGindex will not handle H.264 video.

Maybe. But as my principal reason for bothering with any of this is to convert bulky mpeg-2 video to slender, high-quality h.264 video through x264, that'll be a while.