Help me understand rate distortion (subme 6/7)

F J Walter · 21st July 2014, 01:04

Hi

For curiosity I'm interested in learning more about how rate distortion fits into the motion estimation process in x264, and specifically in subme 6/7.

I understand that rate distortion factors in bitrate efficiency (quality vs bitrate) into the usual metrics used to make decisions, and it does so by calculating both a quality metric as well as an estimated bit cost.

Now I also know that in subme 6/7, rate distortion optimisation is only done at some late stage of motion estimation - various sources say it's the decision of partition types only.

Now what I don't know:

Partition type decision: does that mean the decision of whether to encode a single motion vector for the whole 16x16 block or divide it into partitions down to 8x16,8x8,8x4 and 4x4? Is this the only step that RD in subme 6/7 affects?

Or does RD in subme 6/7 also affect the calculation of whether to refine the motion vector to half or quarter pixel precision?

Or does RD in subme 6/7 also affect the decision of whether to encode a block as a I or P block?

Or does it also affect some other calculation?

I understand that the RDO in subme 8/9 also uses a rate distortion calculation when actually finding the motion vectors - does this mean the full pixel search AND the decision to go to half or quarter pixel precision AND the hpel/qpel search?

And lastly, if RDO for subme 6/7 only affects the partition type decision, how is it able to have such a significant positive effect as it does? Particularly when using Psy RDO, which also introduces a "complexity" metric into the mix - how is that able to have such a significant effect when all it is doing is affecting the partition type decision? How is the partition type decision all that important?

Thanks in advance.

21st July 2014, 01:04	#1 \| Link
F J Walter Registered User Join Date: Oct 2007 Posts: 47	Help me understand rate distortion (subme 6/7) Hi For curiosity I'm interested in learning more about how rate distortion fits into the motion estimation process in x264, and specifically in subme 6/7. I understand that rate distortion factors in bitrate efficiency (quality vs bitrate) into the usual metrics used to make decisions, and it does so by calculating both a quality metric as well as an estimated bit cost. Now I also know that in subme 6/7, rate distortion optimisation is only done at some late stage of motion estimation - various sources say it's the decision of partition types only. Now what I don't know: Partition type decision: does that mean the decision of whether to encode a single motion vector for the whole 16x16 block or divide it into partitions down to 8x16,8x8,8x4 and 4x4? Is this the only step that RD in subme 6/7 affects? Or does RD in subme 6/7 also affect the calculation of whether to refine the motion vector to half or quarter pixel precision? Or does RD in subme 6/7 also affect the decision of whether to encode a block as a I or P block? Or does it also affect some other calculation? I understand that the RDO in subme 8/9 also uses a rate distortion calculation when actually finding the motion vectors - does this mean the full pixel search AND the decision to go to half or quarter pixel precision AND the hpel/qpel search? And lastly, if RDO for subme 6/7 only affects the partition type decision, how is it able to have such a significant positive effect as it does? Particularly when using Psy RDO, which also introduces a "complexity" metric into the mix - how is that able to have such a significant effect when all it is doing is affecting the partition type decision? How is the partition type decision all that important? Thanks in advance.