Intel QuickSync Decoder - HW accelerated FFDShow decoder with video processing

[Blight]

cruncher:
When eric said he designed the the scaler, he should have given a bit of background.
Prior to Intel, he worked on scaling algorithms (noise reduction as well if I remember correctly) at a company that designed image processing chips.
The company was bought out by Intel and the scaler algorithm work was integrated into future Intel chips.
A few years later... and now it's part of Sandy Bridge

[CruNcher]

So the Scaler inside the first Atoms is also from Eric ?
they also bough in the beginning of the year Silicon Hive to improve most probably the power consumption part of future Atoms (and maybe even allready Ivy Bridge or @ least Haswell)

Silicon Hive processors save tremendous area and power by moving almost all control from run-time to compile time.

In a statement Intel said that the acquisition would bring better still-imaging and multimedia video processor technology, compilers and software tools to the Atom processor portfolio. "The Silicon Hive capabilities will aid in the delivery of more differentiated Atom-processor based SoCs as multimedia and imaging grow in importance across the mobile smart device segments," Intel said.

Ahh i see Oplus Technologies Inc is the Deinterlacing Oplus Pixel-Entropy Deinterlacing ?

hehe

Image enhancement filter - enhances images and provides spatial noise reduction
Color transition improvement - improves picture sharpness by emphasizing color transitions
3D motion noise reduction filter
Motion adaptive de-interlacing
Color control - providing independent saturation and luminance control per color compounent
Skin tone enhancement - providing detection and improvement of skin tones
Adaptive contrast enhancement - dynamically adjusts the contrast curve, enhancing image detail
Pixel mode recognition algorithm - enables seamless 3:2 (NTSC) and 2:2 (PAL) reverse pulldown, at the pixel level, of TV-adapted film material.

Image detection, adjustment and positioning
High-quality, patented, linear and non-linear scaling
Patented keystone correction algorithm for high-quality electronic correction for projectors
Flexible windowing technology supporting any combination of video and graphics or video and HDTV, with size, position and order control.

At least i know what im comparing here

So basically the equivalent of HQV from Israel

hehe have to find that patent

hehe thats cool what he done with his Skin tone Research

Segman examined thousands of photos of people
with different skin tones. “The aim was to establish
if a certain skin color could indicate illness. Could a
certain skin tone, for example, be influenced by liver
problems? He uncovered a relationship between skin
colors that indicate normal health or illness.
Segman found that when people are tense, their skin
tone changes. “
The first trials of Cnoga’s technology will soon be
tested at Bnei Zion Hospital in Haifa, supervised by
Prof. Eli Zuckerman.

[egur]

Originally Posted by CruNcher

So the Scaler inside the first Atoms is also from Eric ?

No, It's a completely GPU, I'm not familiar with the details on the Atom...

[egur]

Originally Posted by CruNcher

...Also Deinterlacing and IVTC work only Efficient on EVR with EVR-CP MBAFF Deinterlacing and IVTC are failing currently...

Can you upload failing clips?

[CruNcher]

Originally Posted by egur

Can you upload failing clips?

http://www.mediafire.com/?4gcnazd6y78c9yp
http://www.mediafire.com/?wlnkh80oaou827h

Sure works fine on EVR only EVR-CP is the Problem

It really seems there are only 2 ways to use the Adaptive Deinterlacing either using EVR or EVR-CP + DXVA Decoder

PS: I got Desktop Capturing to work via Quicksync trying to capture though 60 fps on DWM with BitBlt seems not really that efficient



Performance droop with Firefox is Extreme here on both sides IE is much better doing, obviously drooped frames in the EVR but not as bad as with Firefox (they are still far away from Microsofts D2D Engine Performance)

[egur]

Originally Posted by CruNcher

Ahh i see Oplus Technologies Inc is the Deinterlacing Oplus Pixel-Entropy Deinterlacing ?

Before 2005 (when Oplus was bought), the technology went into video processors for flat panel TV and projectors.
The SNB algorithms where done when Oplus was already owned by Intel almost 2 years later. Segman was one of the Oplus founders and he invented the algorithms for the early Oplus video processors, a little before my time and I don't want to comment on his work. He didn't get along with Intel and left very quickly after the acquisition. He got a few million dollars from the sale like all the founders and now he pursues his exotic algorithms elsewhere...

I had a bug with identifying the interlace flag correctly, hence the sample-mbaff.ts clip didn't deinterlace. fixed.
The other clip telecine-test.ts plays fine (EVR).

Regarding EVR-CP. Most video clips will not show video using this on a GT1. I think GT2 works, I'll recheck on Sunday when I come to work.
Almost all renderers disable Aero when MPC-HC starts playback (EVR doesn't ).
Does anyone have a solution for this?
Why is this happening?

[CruNcher]

No such problems (though i know that black frames also like to happen if a bitstream is wrong marked for example as High 5.1, some decoder in the past didn't liked that) though my Win 7 is very Virgin and i didn't yet installed some optional updates that have todo with the D2D subsystem.

And please look carefully @ the telecine one, the problem happens @ the fade (the cut after the man gets dragged out of the frame by the bodyguard and the camera switches back to her on stage) its almost invisible but the difference between EVR and EVR-CP in that fade is visually recognizable.

[nevcairiel]

Originally Posted by egur

Almost all renderers disable Aero when MPC-HC starts playback (EVR doesn't ).
Does anyone have a solution for this?
Why is this happening?

Its a option. You probably applied some settings preset which included this option.

Check the renderer options in the right-click menu.

[CruNcher]

ouhh yeah that nasty thing i totally ignored that, since i first turned it off don't wanna remember it

[Blight]

I know that at least with MadVR, the reason for the option to disable Aero with fullscreen exclusive mode is that it makes the switch between exclusive/window mode faster.
Which is good and fine if you're running a standalone HTPC which never sees the desktop.

Cruncher:
I'm interested to see a visual comparison between SNB DeInterlacing and the NVIDIA/ATI/Software modes.
I believe the eric's plan is to look into adding the hardware deinterlacing support after the decoding is fully stable

egur:
What other PP does the SNB support? Denoise? Sharpen? Weren't you talking about facial color correction a while back?

[egur]

The various EVR CP clack screen issues were solved by resetting the renderer to its default settings.

EVR CP give a jitter or 7-8ms and an offset of ~8ms very constantly. I have no clue why, changing the EVR buffer count doesn't change anything. If someone has an idea, let me know.
CruNcher:
Regarding the IVTC drop (telecine-test.ts) – I can’t reproduce in my current version. I’ve placed a breakpoint in the code that drops IVTC and it doesn’t trigger. BTW IVTC isn’t activated on the first frame, I look for a field doubling flag and only when I see one I activate IVTC. The IVTC dropping heuristic is to count 4 frames from the last frame that had the field doubling flag. After 4 frames the 3:2 cadence is broken and IVTC is dropped. If someone has a better way of doing this, don’t be shy.

SNB video processing blocks are:
*Motion Detection based deinterlacer
*Film cadence detection (for interlaced content) – works on raw video not part of decoder
* Denoise (temporal and spatial)
* Context adaptive sharpening
* Context adaptive 8 tap polyphase scaler. Supports Non Linear Adaptive Scaling (4:3 -> 16:9)
* Total Color Control – something like digital vibrance .
* Automatic Contrast Enhancement
*Skin tone correction
*Color space conversion
* Surface conversion (e.g. YUY2 -> NV12)
* Frame rate conversion.
Note that most are pure ASIC implementation and do affect CPU or EU utilization.
Other algorithms might have been added via kernels (shaders)

[CruNcher]

Egur i see it seems there is indeed a deinterlaced frame that doesn't got flaged as such and the adaptive deinterlacing catches it on EVR
yep that Latency i guess has something todo with the nature of EVR-CP there is a lot going on you could try Jan's current test builds and see if it lowers the latency.

Blight from the first looks just from the experience of it with different interlaced content it does a very good job also look @ what it does to this telecine clip http://forum.doom9.org/showthread.ph...39#post1526139 where it catches that 1 fade very accurately of course as it is blended it's very hard to get that right.

Egur yeah that is the big + vs Nvidia/AMDs shader implementations having most important performance taking PP stuff aside of decoding/encoding Natively and adaptive

Though my first test with * Context adaptive sharpening didn't looked so good versus SharpenComplex2 PS implementation though obviously SharpenComplex2 almost kills the GT1 with High Resolution content but with SD (and for me i use it very unlikely anyways with HD content) it does a good job and the GT1 survives it easily there, the results from Intels Hardware Sharpening via the Control Panel doesn't look as good on the first sight with Heavy compressed streams (even pushing the slider to the edge) but obviously it does much better when you push it a not so compressed image then what SharpenComplex2 does then with it

[egur]

CruNcher:
I looked closer at the problematic telecine-test.ts clip. Indeed it has several interlaced frames right where you said it has.
The fix isn't easy and I don't know if it will be in the next release. At least I root caused the problem. My fix will include a much quicker switch from telecined to non telecined material.
There' also the PAL telecining 24->25 frames by repeating the 24th frame, but it's low priority now.

I'm having issues with time stamps. Several clips produce out of order time stamps. Currently I take the 1st good (non garbage) time stamp and use it as a reference. Following time output stamps are derived from that first one. This isn't perfect and I'd like to know if anyone has a good (bulletproof) solution for this.

[nevcairiel]

I've never had any real issues with H264 or MPEG2 timestamps, they usually are rather consistent. (VC-1 is another matter, different splitters give different results, but there are luckily only 2 ways they do it)
If you just look at the incoming timestamps, they will of course be out of order, because those are presentation timestamps. They belong to the frame they are attached to, and frames are delivered in decode order, not in presentation order.

Any specific clip that gives you trouble?

[CruNcher]

Originally Posted by Blight

I know that at least with MadVR, the reason for the option to disable Aero with fullscreen exclusive mode is that it makes the switch between exclusive/window mode faster.
Which is good and fine if you're running a standalone HTPC which never sees the desktop.

Cruncher:
I'm interested to see a visual comparison between SNB DeInterlacing and the NVIDIA/ATI/Software modes.
I believe the eric's plan is to look into adding the hardware deinterlacing support after the decoding is fully stable

egur:
What other PP does the SNB support? Denoise? Sharpen? Weren't you talking about facial color correction a while back?

Blight lets start with what the user currently sees and has direct access to, that are 7 types of Output configurations.

Color
Denoise
Sharpening
Telecine
Skin tone
Adaptive Contrast
Scaling


in that order the Denoise and Sharpening have a automatic setting and a manual one. Telecine has like the others (AMD/Nvidia) only a on/off (fully adaptive)

they are in the Media section with 3 different subsections in the UI (Color,Picture,Scale) and Deinterlacing being fully Adaptive like Telecine and only 1 Quality therefore is not configurable as well and not mentioned @ all in the UI (makes perfect sense).

Color:


Picture:


Scale:


Everyone of these 3 Sections has 2 Preview Pictures to chose from (basic still picture)

Originally Posted by egur

CruNcher:
I looked closer at the problematic telecine-test.ts clip. Indeed it has several interlaced frames right where you said it has.
The fix isn't easy and I don't know if it will be in the next release. At least I root caused the problem. My fix will include a much quicker switch from telecined to non telecined material.
There' also the PAL telecining 24->25 frames by repeating the 24th frame, but it's low priority now.

Nice can't wait to see this in action

[egur]

Originally Posted by nevcairiel

I've never had any real issues with H264 or MPEG2 timestamps, they usually are rather consistent. (VC-1 is another matter, different splitters give different results, but there are luckily only 2 ways they do it)
If you just look at the incoming timestamps, they will of course be out of order, because those are presentation timestamps. They belong to the frame they are attached to, and frames are delivered in decode order, not in presentation order.

Any specific clip that gives you trouble?

The DC.ts is a good example:
Time stamps are fine when LAV's "Enable VC1 timestamp correction" is checked. Out of order otherwise (including the default 'auto' option) or when using MPC internal splitter. Haali splitter is also fine (although MPC-HC refuses to use it for mpeg2 transport for some reason).

I don't have access to the DShow graph from within my DLL to query the connected filters and from a design point of view I think it's a terrible idea to patch my code according to the connected filters. Ill behaved filters should be handled by the DShow decoder filter if at all. I'll do a more deep scan of my clips to see if non VC1 formats behave well enough.

nevcairiel:
BTW LAV Splitter produces incorrect (or inconsistent with other splitters) frame rates, interlaced or telecined clips receive 59.97 AvgTimePerFrame in the initial VIDEOINFOHEADER. This is very confusing.

[CruNcher]

Originally Posted by egur

The DC.ts is a good example:
Time stamps are fine when LAV's "Enable VC1 timestamp correction" is checked. Out of order otherwise (including the default 'auto' option) or when using MPC internal splitter. Haali splitter is also fine (although MPC-HC refuses to use it for mpeg2 transport for some reason).

I don't have access to the DShow graph from within my DLL to query the connected filters and from a design point of view I think it's a terrible idea to patch my code according to the connected filters. Ill behaved filters should be handled by the DShow decoder filter if at all. I'll do a more deep scan of my clips to see if non VC1 formats behave well enough.

nevcairiel:
BTW LAV Splitter produces incorrect (or inconsistent with other splitters) frame rates, interlaced or telecined clips receive 59.97 AvgTimePerFrame in the initial VIDEOINFOHEADER. This is very confusing.

Egur are these inconsistencies vanishing if you run it in Source mode ? so instead of the default File Source Async way ?
That is still a big difference to how the Internal MPC-HC splitter works and i saw major differences here especially with partly broken streams.
That also got fixed when running Lav Splitter in Source Mode instead of going through "File Source Async" first, especially with evil trees.ts.

[nevcairiel]

Originally Posted by egur

nevcairiel:
BTW LAV Splitter produces incorrect (or inconsistent with other splitters) frame rates, interlaced or telecined clips receive 59.97 AvgTimePerFrame in the initial VIDEOINFOHEADER. This is very confusing.

The information in that header is really not interesting.
Its usually just taken from the container, if its stored there. If not, it'll try to measure it. I would never assume that its 100% reliable with any splitter. Its more of a "hint" then a reliable source of information. Decoding must work without it, a media type is perfectly valid with it set to 0.
The measured values can be off, depending on how its encoded. But luckily, its only measuring it on MPEG-TS and similar formats, but MPEG-TS is tightly timestamped, so that the value is not needed to calculate missing timestamps.

MKV might contain timestamp gaps, but it does include a proper fps header in the container instead.

It never was a problem before for me.

PS:
Filters are not "ill behaved", there is just two ways to handle VC1. I believe i explained that in a PM before. There is PTS and DTS timestamps. MPC-HCs MPEG Splitter outputs PTS, Haali outputs DTS, LAV can output both, depending on which decoder is connected (and ffdshow expects PTS)
PTS timestamps would look out of order if you don't handle them properly, thats perfectly normal and valid. H264 and MPEG2 always use PTS.

To top the VC1 problem off, VC1 in MKV or in WMV only contains DTS, while MPEG-TS contains both DTS and PTS.

[egur]

CruNcher:
I use LAV/Haali/MPC splitters only as source filters.
BTW, I doubt if I can decode both evil_trees.ts and telecine-test.ts using a common logic. Both clips stop marking the frames as telecined (field doubling) mid playback. evil_trees.ts looks best if I don't drop out of IVTC immediately and telecine-test.ts looks bad no matter what I do, but only for a fraction of a second, almost impossible to notice. Even if i switch correctly back to interlaced, the renderer doesn't handle this well and shows the same frame 3 times and then 2-3 frames that should have been interlaced but have not.
Since both clips have identical behavior (same flags on the decoded frames) it's impossible to tell them apart. DGindex reports that evil trees is 12% interlaced, 88% film...
I'd like to hear your opinion on this.

nevcairiel:
How can I tell if get DTS or PTS, missing time stamps. Sounds like hell
Is there code for dealing with this some where?
To avoid license problems (is there a problem?), I can put the time stamp normalizing code in ffdshow, not in my DLL.
BTW expecting a valid value for AvgTimePerFrame is legit. A zero value might mean I need to measure (need a few frames to do so), but a wrong value is misleading. Microsoft's documentation doesn't say you can ignore this value.

[nevcairiel]

Even if its legit, its not required. Just use the timestamps you get from the source filter, thats what they are for!

Zero AvgFrameTime means it could as well be variable frame rate, which sounds like your code would crash and burn with.
The fun about VFR is that it will most likely not be 0, it'll probably be whatever frame rate the first segment is.

@timestamps in general:

I'm sure the MSDK offers an ability to handle timestamps by itself. This will probably be fine with all PTS timestamps. At least thats the case with NVIDIAs API and DXVA2.
Just for DTS timestamps, you need to manually map the incoming times to the outgoing frames. Since the number of frames coming in and going out is usually the same, that shouldn't be much of a problem. I have that implemented in LAV Video using a FIFO buffer for the CUVID decoder (because i don't know its exact processing delay), and a fixed circular buffer in the avcodec decoder (because there i know its exact decoding delay).