Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd October 2015, 06:50   #1  |  Link
sephirotic
Registered User
 
Join Date: Apr 2008
Posts: 19
Enconding in 444 YV24 from RGB source Avisynth.

I color graded a m2ts episode from a BD I own on Premiere CS6. Only truly lossless export Option I had was mov with animation codec (RLE).
Now, I wanted to convert to a lossless x264 file (qp0 preset slow) with 4:4:4 subsampling.

I'm truly lost as how to effectively generate a qp0 x264 file with the less possible loss in color acurancy. I understand that a 100% lossless RGB->YV24 conversion isn't possible, but all the attempts I made so far seem to give me a file that was converted to 4:2:0 at some point.

Initially I tried inputing the file directly on x264.exe CLI, but I didn't figure out how to make the mov file acceptable by x264 (and even if it is). --input-fmt uses lav filter, right? Should it be configured as default spliter on windows for this parameter to work? My default filter is ffdshow.
Then I heard Avisynt version 2.60 now accepts YV24 output format, but I'm not sure, the regular version do or just some beta build?
Anyway, I just added ConvertToYV24 on avs script and encoded with megui with the following settings:

program --preset slower --qp 0 --b-pyramid none --no-weightb --qpmax 51 --merange 24 --psy-rd 0.00:0 --no-dct-decimate --no-fast-pskip --output-csp i444 --output "output" "input"

I manually added the output-csp i444 line. This generated me a true 444 output file but the colors seems just like 420. I suspect avisynth is indeed outputting 420.

I further tought about exporting a TIF image sequence from Premiere and trying to feed it into x264.exe directly but I have no idea how to do that.

After a lot of reading around Doom's9 and other websites, I also saw people saying that I should specify the input color space for encoding on x264. Original m2ts was Rec.709 limited range, the RGB conversion on Premiere seemed to be fully lossless as the output mov files looks identical to the original 4:2:0 m2ts file. So should I just specify Rec 709 anyway?

I'm not even sure this should be in the avisynth section or x264 section.
Thanks a lot.

Last edited by sephirotic; 2nd October 2015 at 06:59.
sephirotic is offline   Reply With Quote
Old 2nd October 2015, 06:53   #2  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
"I understand that a 100% lossless RGB->YV24 conversion isn't possible"
nah nah nah, see "YCgCo"

--output-csp i444 --input-csp i444 --profile high444
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 2nd October 2015, 08:23   #3  |  Link
sephirotic
Registered User
 
Join Date: Apr 2008
Posts: 19
Quote:
Originally Posted by feisty2 View Post
"I understand that a 100% lossless RGB->YV24 conversion isn't possible"
nah nah nah, see "YCgCo"

--output-csp i444 --input-csp i444 --profile high444
I have researched about YCgCo and it seems that it is only compatible with Madvr, with is not good enough for me.

I discovered why Avisynth wasn't converting properly, even tough the input was RGB, I still should have specified the matrix: ConvertToYV24 (matrix="rec709")
By just adding the parameter I managed to get 100% lossless result.
Thanks for the quick reply, tough.
sephirotic is offline   Reply With Quote
Old 2nd October 2015, 08:30   #4  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
Quote:
By just adding the parameter I managed to get 100% lossless result.
YCbCr (601, 709, 2020..) <---> RGB IS NOT mathematically lossless under finite precision
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 2nd October 2015, 09:41   #5  |  Link
vivan
/人 ◕ ‿‿ ◕ 人\
 
Join Date: May 2011
Location: Russia
Posts: 649
sephirotic,
Color being different is a different issue.
For RGB->YCbCr conversion x264 uses BT.601 matrix, which is wrong for HD video.
By default avisynth's ConvertTo uses it too, but it can and should be specified ("Rec709" is the correct one if you want tv-range video).

Do you really need 4:4:4 YCbCr? x264 supports RGB encoding which is a bit less effective but way more convenient (and is actually lossless).

Quote:
Originally Posted by feisty2 View Post
nah nah nah, see "YCgCo"
Don't forget to add that it also requires 9 bit precision for chroma to be lossless. Which means 10 bit encoder, Dither tools and 16 bit output from avisynth.

Quote:
Originally Posted by feisty2 View Post
YCbCr (601, 709, 2020..) <---> RGB IS NOT mathematically lossless under finite precision
Of course it's lossless, under higher but still finite precision.
vivan is offline   Reply With Quote
Old 2nd October 2015, 10:54   #6  |  Link
Jamaika
Registered User
 
Join Date: Jul 2015
Posts: 608
Quote:
Originally Posted by sephirotic View Post
I color graded a m2ts episode from a BD I own on Premiere CS6. Only truly lossless export Option I had was mov with animation codec (RLE).
Now, I wanted to convert to a lossless x264 file (qp0 preset slow) with 4:4:4 subsampling.
At the very beginning you need to be aware that m2ts input material was compressed YUV BT709, that you have lost some information.
Then you added effects and wanted to make the movie lossless at x264.
Quote:
Originally Posted by sephirotic View Post
I further tought about exporting a TIF image sequence from Premiere and trying to feed it into x264.exe directly but I have no idea how to do that.
I don't know what you created TIFF images MOV. You can download a free plugin VP9 (sRGB, i444 undef).
Chroma sampling 4:4:4 (YUV i444) will be converted to 8bit sRGB monitor. ):
BT2020 is for depth 10bit files and currently isn't recognized by most programs only TV.
sRGB isn't recognized by most programs.

Last edited by Jamaika; 2nd October 2015 at 11:03.
Jamaika is offline   Reply With Quote
Old 2nd October 2015, 11:49   #7  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
Quote:
Originally Posted by vivan View Post

Of course it's lossless, under higher but still finite precision.


there's something called "Infinite decimal", which is a mathematic concept even known to kids from elementary school.
and iirc, floats can do "finite decimals", not fractions
edit: typo
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated

Last edited by feisty2; 2nd October 2015 at 11:54.
feisty2 is offline   Reply With Quote
Old 2nd October 2015, 13:36   #8  |  Link
Motenai Yoda
Registered User
 
Motenai Yoda's Avatar
 
Join Date: Jan 2010
Posts: 701
Yet but it's lossless like, as float precision is enough to get same values for 8bit RGB -> (16bit RGB) -> >9bit YUV -> (16bit RGB) -> 8bit RGB.
as IIRC you need about 9.6bit YUV to have all 8bit RGB values, ergo 10 or more bit.
__________________
powered by Google Translator
Motenai Yoda is offline   Reply With Quote
Old 2nd October 2015, 13:49   #9  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
Quote:
Originally Posted by Motenai Yoda View Post
Yet but it's lossless like, as float precision is enough to get same values for 8bit RGB -> (16bit RGB) -> >9bit YUV -> (16bit RGB) -> 8bit RGB.
as IIRC you need about 9.6bit YUV to have all 8bit RGB values, ergo 10 or more bit.
the YCbCr -- RGB stuff will make infinite decimals, and the infinite decimal gets truncated to some finite decimal and then converted to an even lower precision and BY CHANCE it's "lossless" at such low precision, that's not "mathematically" lossless, that's coincidence
and YCgCo is free from all this at the mathematic level, no infinite decimal will be produced long as the input is not infinite decimal, and that's "mathematically" lossless
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 2nd October 2015, 15:22   #10  |  Link
vivan
/人 ◕ ‿‿ ◕ 人\
 
Join Date: May 2011
Location: Russia
Posts: 649
Quote:
Originally Posted by feisty2 View Post
there's something called "Infinite decimal", which is a mathematic concept even known to kids from elementary school.
and iirc, floats can do "finite decimals", not fractions
edit: typo
there's also something called "precision". I know, they don't teach it to kids so it's totally ok not to know about it.

Quote:
Originally Posted by feisty2 View Post
the YCbCr -- RGB stuff will make infinite decimals, and the infinite decimal gets truncated to some finite decimal and then converted to an even lower precision and BY CHANCE it's "lossless" at such low precision, that's not "mathematically" lossless, that's coincidence
Yeah, sure, 100% chance is really just a coincidence.

Code:
void RGBtoYCbCr (unsigned char* RGB, unsigned short* YCbCr) {
	YCbCr[0] = (unsigned short)(( 16.0 + 0.257 * RGB[0] + 0.504 * RGB[1] + 0.098 * RGB[2]) * 257.0 + 0.5);
	YCbCr[1] = (unsigned short)((128.0 - 0.148 * RGB[0] - 0.291 * RGB[1] + 0.439 * RGB[2]) * 257.0 + 0.5);
	YCbCr[2] = (unsigned short)((128.0 + 0.439 * RGB[0] - 0.368 * RGB[1] - 0.071 * RGB[2]) * 257.0 + 0.5);
}

void YCbCrToRGB (unsigned short* _YCbCr, unsigned char* RGB) {
	double YCbCr[3] = {_YCbCr[0] / 257.0 - 16.0, _YCbCr[1] / 257.0 - 128.0, _YCbCr[2] / 257.0 - 128.0};
	RGB[0] = (unsigned char)(1.164 * YCbCr[0] + 0.000 * YCbCr[1] + 1.596 * YCbCr[2] + 0.5);
	RGB[1] = (unsigned char)(1.164 * YCbCr[0] - 0.392 * YCbCr[1] - 0.813 * YCbCr[2] + 0.5);
	RGB[2] = (unsigned char)(1.164 * YCbCr[0] + 2.017 * YCbCr[1] + 0.000 * YCbCr[2] + 0.5);
}

int _tmain(int argc, _TCHAR* argv[]) {
	for (int R = 0; R < 256; R++)
	for (int G = 0; G < 256; G++)
	for (int B = 0; B < 256; B++) {
		unsigned char RGB[3] = {R, G, B};
		unsigned char newRGB[3];
		unsigned short YCbCr[3];
		RGBtoYCbCr (RGB, YCbCr);
		YCbCrToRGB (YCbCr, newRGB);
		if (newRGB[0] != RGB[0] || newRGB[1] != RGB[1] || newRGB[2] != RGB[2]) {
			printf ("%i %i %i %i %i %i\n", newRGB[0], RGB[0], newRGB[1], RGB[1], newRGB[2], RGB[2]);
		}
	}

	return 0;
}
Run it and tell me what you see (or, more precisely, don't).
vivan is offline   Reply With Quote
Old 2nd October 2015, 16:27   #11  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
yeah, yeah
something is only lossless when you manually clip the extra imprecise "lsb" part (aka. error part) off, right, 0.123456 = 0.123000 when you zero the "456" lsb out and that is just mathematically lossless??
back to the point, it's lossless cuz you were rounding a short to a char, which almost never ever happens in the wild, cuz you know what? people will dither a short to a char, and boom, the lossless magic is gone with dithering, so you gonna tell everyone, just rounding 16bits YUV to 8bits RGB, don't dither? and I highly doubt if anyone will ever follow that advice
and YCgCo is ALWAYS lossless, dithering or truncating or rounding, cuz you know what, it's mathematically lossless (it has no lsb (no error), so nothing to dither)
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated

Last edited by feisty2; 2nd October 2015 at 16:41.
feisty2 is offline   Reply With Quote
Old 2nd October 2015, 16:52   #12  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
and don't even bother to write c programs to test all these
dither package + mt_makediff + mt_lut (diff, "x 128 - 20 * 128 +") are everything you need
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 3rd October 2015, 04:08   #13  |  Link
sephirotic
Registered User
 
Join Date: Apr 2008
Posts: 19
Quote:
Originally Posted by vivan View Post
sephirotic,
By default avisynth's ConvertTo uses it too, but it can and should be specified ("Rec709" is the correct one if you want tv-range video).
yest, that is what I've done after further digging.

Quote:
Originally Posted by vivan View Post
Do you really need 4:4:4 YCbCr? x264 supports RGB encoding which is a bit less effective but way more convenient (and is actually lossless).
I've done a second encode with x264 rgb for keeping the master, I only needed YCBCR for the final 10 bit encode.

Quote:
Originally Posted by Jamaika View Post
At the very beginning you need to be aware that m2ts input material was compressed YUV BT709, that you have lost some information.
Then you added effects and wanted to make the movie lossless at x264.
Well, Premiere interpreted the color matrix correctly so lost information was minimal.

Quote:
Originally Posted by Jamaika View Post
I don't know what you created TIFF images MOV. You can download a free plugin VP9 (sRGB, i444 undef).
Chroma sampling 4:4:4 (YUV i444) will be converted to 8bit sRGB monitor. ):
BT2020 is for depth 10bit files and currently isn't recognized by most programs only TV.
sRGB isn't recognized by most programs.
sRGB would be interesting for the master, but then again, premiere doesn't export it Afaik so taht would only add another conversion and more rounding losses in the process.

Thanks for all the help, I was able to attain virtually imperceivable loss in color information just by using ConverTToYV24 (matrix="rec709")

Here a sample to anyone curious.
http://diff.pics/ssFVuuBc4Agi/1
Lossless in this case is qp0 veryslow no bframes x264 yv24 encode, but I'm in the middle of doing a RGB x264 encode right now as a V2.

Last edited by sephirotic; 3rd October 2015 at 04:23. Reason: Missed quote mark
sephirotic is offline   Reply With Quote
Old 4th October 2015, 03:25   #14  |  Link
vivan
/人 ◕ ‿‿ ◕ 人\
 
Join Date: May 2011
Location: Russia
Posts: 649
Quote:
Originally Posted by feisty2 View Post
yeah, yeah
something is only lossless when you manually clip the extra imprecise "lsb" part (aka. error part) off, right, 0.123456 = 0.123000 when you zero the "456" lsb out and that is just mathematically lossless??
First, you should learn the definition of lossless. Lossless transform is a transform that can be reversed so that you will get original data. And no, "can" is not about probability - it's about conditions, which is higher precision.

Second, you should really learn something about precision. 0.123 and 0.123000 have different meaning - second implies much greater precision. What original "0.123" actually means is that value lies in the +- 0.0005 range. And this is how you're supposed to compare numbers.

Quote:
Originally Posted by sephirotic View Post
I've done a second encode with x264 rgb for keeping the master, I only needed YCBCR for the final 10 bit encode.
But can't you encode from your x264 rgb master?
vivan is offline   Reply With Quote
Old 4th October 2015, 04:21   #15  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
I don't wanna waste my time on this pointless argument no more
Code:
colorbars (pixel_type="rgb32")

ycgco = dither_convert_rgb_to_yuv (output="yv24",lsb=true,matrix="ycgco",tv_range=false)
ycbcr = dither_convert_rgb_to_yuv (output="yv24",lsb=true,matrix="601",tv_range=false)

rgb48ycgco = dither_convert_yuv_to_rgb (ycgco, output="rgb48y",matrix="ycgco",tv_range=false,lsb_in=true)
rgb48ycbcr = dither_convert_yuv_to_rgb (ycbcr, output="rgb48y",matrix="601",tv_range=false,lsb_in=true)

rgb48cgco = mergergb (rgb48ycgco.selectevery (3,0), rgb48ycgco.selectevery (3,1), rgb48ycgco.selectevery (3,2)).subtitle ("ycgco")
rgb48cbcr = mergergb (rgb48ycbcr.selectevery (3,0), rgb48ycbcr.selectevery (3,1), rgb48ycbcr.selectevery (3,2)).subtitle ("ycbcr")

StackHorizontal(rgb48cgco,rgb48cbcr)


just stare at the LSB part of "ycgco" and "ycbcr" real carefully and tell me what's the difference, which one is "lossless"
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 4th October 2015, 04:34   #16  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
0.123000 (ycgco) = 0.123 is lossless, cuz 0.123000=0.123 mathematically
0.123456 (ycbcr) ≈ 0.123 is NOT lossless, ≈ is still some kind of != obviously
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 4th October 2015, 13:14   #17  |  Link
colours
Registered User
 
colours's Avatar
 
Join Date: Mar 2014
Posts: 312
It's vaguely ironic to see feisty2 post about "mathematical precision" and using Dither tools to illustrate that, when Dither tools has off-by-one errors scattered throughout the full-range colour conversion functions.

Also, good job missing vivan's point. How surprising it is that you find the argument "pointless" when you just ignore what everyone else says.
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!
colours is offline   Reply With Quote
Old 4th October 2015, 15:04   #18  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
Quote:
Originally Posted by colours View Post
It's vaguely ironic to see feisty2 post about "mathematical precision" and using Dither tools to illustrate that, when Dither tools has off-by-one errors scattered throughout the full-range colour conversion functions.
there's no such thing as infinite precision for computers, I don't see any other better option to do it, anyways, I'll explain what I was saying with the following

Quote:
Originally Posted by colours View Post
Also, good job missing vivan's point. How surprising it is that you find the argument "pointless" when you just ignore what everyone else says.
I get that stuff vivan or you stated

suppose:
1. there was infinite precision for computers
2. all rgb-yuv444-rgb stuff was done correctly under infinite precision
3. the input value was 0.123

the output value of both "ycgco" and "ycbcr" should be 0.123000000... or 0.1229999999... both are mathematically equal to 0.123, the input value
perfect, end of the story

now the reality is, there's no infinite precision, only higher but still finite precision

say, the higher precision here is "6 significant figures"
the output of ycgco would be like, 0.123000, still mathematically equals to 0.123, cuz there're only things like /2 and /4 inside ycgco stuff, and they won't produce infinite decimals long as the input is not infinite precision

the ycbcr output might be like 0.123456, which != 0.123, cuz obviously ycbcr requires infinite precision for intermediate values to stay the zero error status, some intermediate value gets truncated to 6 significant figures and that leads to the error presented in the output value

the disagreement starts right from here

you and vivan think 0.123456 will become 0.123 if rounded down to 3-significant-figure precision and will =0.123 again so it's lossless

and I think it's cheating, the mathematically lossless method should have an output like ycgco, lossless without rounding
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated

Last edited by feisty2; 4th October 2015 at 15:18.
feisty2 is offline   Reply With Quote
Old 4th October 2015, 16:26   #19  |  Link
Rumbah
Registered User
 
Join Date: Mar 2003
Posts: 460
Well, I'd define lossless like this in this case:

I have an image in space A.
I have a transformation f from space A to space B.
I have a transformation g from space B to space A.

If for every pixel x in A: x = g(f(x)) is true then the transformation is lossless.

It does not matter if there is rounding or definite presicion involved as long as every starting pixel has the same value as a transformed and back tranformed pixel.

With this you see that the Dither color space tranformations are not lossless (because of dithering they can be off by +-1).

But indeed you can do a lossless RGB8 to YCbCr10 to RGB8 transformation as vivan has shown.
__________________
x264 full help - x264 --fullhelp r2345
Cuttermaran HCEnc provider - Support for HCEnc in Cuttermaran
DualDVDRB - Dual core support for DVD-RB free

Last edited by Rumbah; 4th October 2015 at 16:29.
Rumbah is offline   Reply With Quote
Old 4th October 2015, 16:43   #20  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,133
Quote:
Originally Posted by Rumbah View Post
Well, I'd define lossless like this in this case:

I have an image in space A.
I have a transformation f from space A to space B.
I have a transformation g from space B to space A.

If for every pixel x in A: x = g(f(x)) is true then the transformation is lossless.

It does not matter if there is rounding or definite presicion involved as long as every starting pixel has the same value as a transformed and back tranformed pixel.

With this you see that the Dither color space tranformations are not lossless (because of dithering they can be off by +-1).

But indeed you can do a lossless RGB8 to YCbCr10 to RGB8 transformation as vivan has shown.
and that "lossless" for ycbcr is just useless like flopped celebrities going to reality shows (the voice, ...) trying to regain attention

will you ever "round" or "truncate" a 16bits YUV vid to <= 14bits RGB? guess not

and the losslessness of ycgco will survive thru rounding or truncating or error diffusing, long as no random noise is applied, ycgco takes its crown by being mathematically lossless at finite precision
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Reply

Tags
444, avisynth, color subsampling, x264, yv24

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:54.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.