Stereoscopic 3D using MVC with x264 ?

[vsv]

"TDVision's patented technology has also been implemented as Intra-view
coding in the MVC MPEG-4 H.264 extension recommendation initiative by
comparing the difference between left and right channels. This eliminates
redundant information and optimizes the bandwidth required for a given
infrastructure such as cable, satellite, ATSC 1.0/2.0 and Blu-ray disc."
http://3dcinecast.blogspot.com/2009/...irst-3d2d.html

A version of the reference software for MVC including both encoder and
decoder source code is publicly available too:
http://wftp3.itu.int/av-arch/jvt-sit.../JVT-AB207.zip

Panasonic, TDVision and third must be X264

[Dark Shikari]

It's insane to try to implement such a spec when they're still finding bugs in it every other week.

[vsv]

What better for x264 encoding with two views
side-by-side or over-under ?

[dukey]

i would say if you are going to encode 2 images in the stream
i would go with one image on top of the other as opposed to side by side. If you want to render the video with a 3d api direct x etc, side by side you'll run out of texture resolution for older cards, where as on top you won't

[Manao]

As far as I can tell, MVC is just AVC with a bunch of SEI messages. No new coding tools have been added (so your post is outdated). So you don't need to modify x264 internals, you just need to create those SEI messages properly.

[BeyondTheEyes]

Well, it's a little more than that: while it's true that MVC does not add new encoding tools (e.g. new macroblock modes) on top of AVC, MVC does specify new high-level syntax (from NAL units to slice headers) for inter view prediction (an MVC picture is composed of N >= 2 view components). Therefore I believe MVC could be supported by x264 core engine but (especially) DPB (e.g. reference picture lists) management would require specific adaptation changes.

[LoRd_MuldeR]

Originally Posted by vsv

What better for x264 encoding with two views
side-by-side or over-under ?

Given that both views show a pretty similar picture, I'd say:
Interleave the views before encode and separate them after decode (at playback time).
That should give more efficient encoding and avoid ultra-long motion vectors...

[vsv]

Originally Posted by Manao

As far as I can tell, MVC is just AVC with a bunch of SEI messages.

Originally Posted by LoRd_MuldeR

Interleave the views before encode and separate them after decode (at playback time).

Yes! Now we need to compare efficience of multiview encoding (MVC) vs over-under and side-by-side with
same stereo3D source.

[Zerofool]

Sorry for bringing that old topic up, but now that 3D Blu-Ray standard is finalized (using MVC), will you guys consider adding MVC support in x264 at some later time? First we'll have to wait because "The completed specification will be available shortly and provides individual manufacturers and content providers with the technical information and guidelines necessary to develop, announce and bring products to market pursuant to their own internal planning cycles and timetables". So it's finalized, but not out yet . |sources: 1 2 3 4 5 |
There's an Annex H in H.264 standard about MVC and the "Multiview High profile", but I guess you already know that .
Considering 2010 will be the year of Stereoscopic 3D, this will be a great addition to x264.

[shon3i]

Originally Posted by Zerofool

will you guys consider adding MVC support in x264 at some later time?

They should first finish that current x264 became full blu-ray compilant :P

[Biggiesized]

Can you imagine 3-D blocking artifacts?! Trippy!

[Zerofool]

Originally Posted by Biggiesized

Can you imagine 3-D blocking artifacts?! Trippy!

Yes, it's strange, but your brains deal with it better than in 2D, I've seen some old side-by-side XviDs . For low quality, blurry 3D H.264 encodes just head over to YouTube .

Originally Posted by shon3i

They should first finish that current x264 became full blu-ray compilant :P

Well, of course, but I'm not talking about BD compatibility, it's about efficient storage and playback of 3D videos on the PC. In all these news about MVC, everybody's mentioning about 50% more space needed for the second image (in most cases). And from my CRF test with current x264 builds in side-by-side and over/under modes (both almost identical in size) the results are 96-100% additional bitrate consummation compared to only the left or right video (which are again very close in size to each other). So for the second image we have additional size ~98% (current x264, no MVC) vs. ~50% (H.264 with MVC, on paper ).
If someone's interested I could post some results data, used settings and sources.
And yes, updated decoders will be needed, but obviously right now Nvidia is working with Arcsoft, Corel and CyberLink (devs of TotalMdedia Theater, WinDVD and PowerDVD), and AMD - with CyberLink. Both companies will host demonstrations of Blu-Ray 3D playback at the upcoming CES (HW Accelerated in the case of nv, no word about amd's). Most likely other decoders will soon follow.

[juGGaKNot]

VideoLeft = AVIsource("C:\x264\movie.avi", audio=false)
VideoRight = AVIsource("C:\x264\movie3D.avi", audio=false)
VideoStacked = StackHorizontal(VideoRight,VideoLeft)
ConvertToYV12(VideoStacked)

works for me

[jakor]

Originally Posted by LoRd_MuldeR

Given that both views show a pretty similar picture, I'd say:

Sorry to say that but your statement above is fundamentally wrong.
Given that two pictures represent the same scene - but even if the cameras are parallel - it is not possible to perform efficent MC between them. For example - with 2 views given - the ratio between two P frames - one from the same view - the next in time axis and the other - the same scene, but other view is 1:20.

[Blue_MiSfit]

I wouldn't say I've observed 2x more bitrate being necessary in the split-screen 3D test clips I've encoded for various HDTV manufacturers using x264. More, sure, but definitely not 2x more. For example, I did an encode of a segment from some really awful movie about dinosaurs and giant snakes attacking a city in 3D, and got quite good results at ~8mbps CBR for 1080p, with a 1 second buffer ... streaming is a real bummer lols

~MiSfit

[BlackSharkfr]

I use x264 for encoding stereoscopic 3D content.
My #1 issue at the moment isn't really the bitrate or the increase of bitrate required for stereo video.

The main issue I have at the moment is media player not being able to detect the stereo video format automatically.
At the moment, watching stereo video systematically requires user input to find out what format is being used.

Many different stereo video formats are being used :
-Separate files for L and R views
-Dual streams L+R in single file
-line interleave (encoded as progressive)
-line interleave (encoded as interlaced)
-Stacked L+R views in single video frame
>-stacked horizontally
>-stacked vertically
>->- left eye view then right eye view
>->- right ee view then left eye view

or any ther possible way of storing two eye views in a video file.

This requires tagging the files/streams in a standard way, and then the stereoscopy enabled media player to reckognize these tags.
I have seen the stereoscopy tag for matroska files in mkvmerge-gui, but not only it is unsufficient (it only supports dual streams orhas a vague "both views in one stream") but no media player is capable of actually using them.


I am not a developer and i can't code, but i wish that the arrival of MVC will bring more attention to the issues encountered by stereo video users.

[moviefan]

One short question concerning this 3D stuff for Blu-rays as I find it very interesting: There's for example the movie "My Bloody Valentine 3D", is this movie based on MVC? I guess not, since the standard isn't in use for product sale right now correct? What is the difference between this MVC and 3D Blu-rays that are available by now?

[Reimar]

Originally Posted by BlackSharkfr

I am not a developer and i can't code, but i wish that the arrival of MVC will bring more attention to the issues encountered by stereo video users.

I don't think this kind of "nobody can be bothered to fix the small and easy issues, so let's make a really difficult solution"-thinking is really going to work.
If stereo doesn't work well with the current stuff it IMHO is because nobody cares, and MVC sure isn't going to help there.
And concerning bandwidth: I find it hard to imagine that MVC will do significantly better than encoding the views as successive frames and increasing the number of reference frames...

[Manao]

I find it hard to imagine that MVC will do significantly better than encoding the views as successive frames [...]

That is exactly what MVC is intended for. As BlackSharkfr noticed, what is missing in his case is mostly signalization stuff (like AR, frame rate, overscan, but for 3D material), and MVC does just that, and nothing else. Originally, they did want to improve coding efficiency, but that would have broken compatibility and cost too much.

[BeyondTheEyes]

Originally Posted by Manao

That is exactly what MVC is intended for. As BlackSharkfr noticed, what is missing in his case is mostly signalization stuff (like AR, frame rate, overscan, but for 3D material), and MVC does just that, and nothing else. Originally, they did want to improve coding efficiency, but that would have broken compatibility and cost too much.

I would like to clarify something, as already stated in a previous post (here): MVC is not about SEI messages, but also specifies a new decoding process (cf. Annex H of the H.264/AVC recommendation) that enables the interview prediction of one view (for instance the right-eye view) from the so-called base view (in that example the left-eye view). A legacy H.264/AVC decoder (i.e. that is capable of decoding High profile Elementary Streams), even if some sort of signalization is present, will not be able to completely handle an MVC Elementary Stream (i.e. decode both views), since new NAL unit types will be present in that stream, and (among other things) a modified (in comparison with the one specify in legacy H.264/AVC profiles) process for reference picture lists will be signaled in that stream. While it's true that the currently specified MVC profiles do not specify any new encoding tools at the macroblock level beyond legacy H.264/AVC ones, MVC does specify a bunch of new stuffs compared with H.264/AVC. BTW, coding efficiency is also improved compared with simulcast or legacy interleaving methods, but obviously not by a very significant percentage. What is key is the backward compatibility of MVC with H.264/AVC (both for the video specification and the MPEG-2 TS specification), which will enable 3D Blu-ray titles to be decoded on legacy 2D Blu-ray players (their integrated H.264/AVC High profile decoder will only output the H.264/AVC compatible base view).