Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 Doom9's Forum range_in_s settings on bitdepth conversion ?
 Register FAQ Calendar Search Today's Posts Mark Forums Read

 24th July 2020, 16:31 #1  |  Link poisondeathray Registered User   Join Date: Sep 2007 Posts: 4,770 range_in_s settings on bitdepth conversion ? What is math or generic explanation for this observation? Start with YUV420P8 "bars"; Y=0,16,235,255 . Then convert to YUV420P10 , using either range_in_s full or limited Y values read with vsedit picker, but double checked with planestats (crop to a single color) Limited Y=0,64,940,1020 Full Y=0,64,943,1023 Code: ```a = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[0, 128, 128]) b = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[16, 128, 128]) c = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[235, 128, 128]) d = core.std.BlankClip(width=320, height=720, format=vs.YUV420P8, fpsnum=24, fpsden=1, length=24, color=[255, 128, 128]) clip = core.std.StackHorizontal([a,b,c,d]) clip10 = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited") #clip10 = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="full") clip10.set_output()```
 24th July 2020, 19:13 #2  |  Link feisty2 I'm Siri     Join Date: Oct 2012 Location: void Posts: 2,619 I always use fmtc for this kinda stuff, I never encountered any problem. you can specify "fulls" and "fulld" to enforce the assumed source range and the result range.
24th July 2020, 20:25   #3  |  Link
poisondeathray
Registered User

Join Date: Sep 2007
Posts: 4,770
Quote:
 Originally Posted by feisty2 I always use fmtc for this kinda stuff, I never encountered any problem. you can specify "fulls" and "fulld" to enforce the assumed source range and the result range.
It's not a "problem" per se ; I'm asking about the underlying explanation

This is not a full=>limited or limited=>full discussion ( it's not clamping or expanding levels. )

To clarify - When you convert YUV 8bit to YUV 10bit - What are the expected values? Why should the range parameter change the results when the setting is same for input and output

range_in_s="limited", range_s="limited"
Y=0,64,940,1020

range_in_s="full", range_s="full"
Y=0,64,943,1023

core.fmtc.bitdepth (clip, bits=10, fulls=False, fulld=False)
Y=0,64,940,1020

core.fmtc.bitdepth (clip, bits=10, fulls=True, fulld=True)
Y=0,64,943,1023

Last edited by poisondeathray; 24th July 2020 at 20:31.

 24th July 2020, 20:35 #4  |  Link feisty2 I'm Siri     Join Date: Oct 2012 Location: void Posts: 2,619 when the input range is "limited", the bit depth conversion is a bit shift operation, concretely result = input << (output_depth - input_depth) and as you can see, 235 << (10-8) = 235 * 4 = 940 for full range input, the pixel range is first normalized to [0, 1], then linearly scaled to the target range 235 / 255 * 1023 = 942.76470588235294117647058823529 ≈ 943 Last edited by feisty2; 24th July 2020 at 20:39.
24th July 2020, 21:07   #5  |  Link
poisondeathray
Registered User

Join Date: Sep 2007
Posts: 4,770
Quote:
 Originally Posted by feisty2 when the input range is "limited", the bit depth conversion is a bit shift operation, concretely result = input << (output_depth - input_depth) and as you can see, 235 << (10-8) = 235 * 4 = 940 for full range input, the pixel range is first normalized to [0, 1], then linearly scaled to the target range 235 / 255 * 1023 = 942.76470588235294117647058823529 ≈ 943
Ahhh!
Thanks f2!

 24th July 2020, 21:25 #6  |  Link feisty2 I'm Siri     Join Date: Oct 2012 Location: void Posts: 2,619 as defined in BT601-7 (page 4), the upper and lower bounds of limited range for bitdepths > 8 themselves are the bit shifted values of 8-bit bounds, and since the conversion is a linear function, the bit shift operation must also apply to everything in between. and since bit shift fails to map the full range bounds of different bitdepths (apparently 255 << (10-8) = 255 * 4 = 1020 ≠ 1023), we follow the regular "normalize then scale" routine for full range bitdepth conversions now I dont usually worry about bullshit like this because I only work with normalized fp32 inputs (0.0 = black, 1.0=white), sample types other than fp32 or fp64 should just f**kin' die Last edited by feisty2; 24th July 2020 at 21:29.