developer discussion: custom pixel shaders -> color space, black/white definition

[madshi]

Hello devs,

now we have multiple renderers (VMR9, EVR-CP, madVR) which support custom pixel shaders. I think it's time to explicitly define a few things which are currently not clearly defined:

(A) Which colorspace should custom pixel shaders run in?

1. R'G'B' (gamma corrected RGB)
2. RGB (linear light RGB)
3. BT.709 Y'CbCr (gamma corrected YCbCr)
4. BT.709 YCbCr (linear light YCbCr)
5. something else

Currently VMR9 and EVR seem to use R'G'B'. Should we stick to that? That would be fine with me. But we should clearly define this.

(B) Where is black and where is white?

1. black 0.0, white 1.0
2. black 16/255, white 235/255
3. black (16 * 256)/65535, white (235 * 256)/65535
4. something else

Currently VMR9 and EVR seem to run pixel shaders with black at 0.0 and white at 1.0. Should we stick to that?

Black at 0.0 has some clear advantages. One key advantage is that the typical RGB <-> YCbCr matrixes simply work as expected. With black at 16/255 the usual matrixes would not work correcty, AFAIK. So if a shader needs to convert RGB to YCbCr, having black at 0.0 makes things much easier. The same applies when a shader wants to convert R'G'B' to RGB because the proper conversion is done with "pow" when black is at 0.0.

However, there's one big disadvantage with having black at 0.0: When using cardinal texture formats for temp storage, BTB and WTW are clipped. Which is what currently actually happens with VMR9 and EVR. Having black at 16/255 and white at 235/255 would preserve BTB and WTW even when using cardinal textures.

-------

I'm not really sure what my preferred solution is. I would vote for either R'G'B' with black at 0.0, or for R'G'B' with black at 16/255. In the first case I'd have to use float textures to preserve BTB/WTW which would probably hurt performance a bit. In the 2nd case I could stick to cardinal textures, however life would be more difficult for shader writers.

Comments / votes, anyone?