.ti3 Parser / .3dlut Generator for madVR + .3dlut toolset

[cyberbeing]

Originally Posted by nand chan

Can you detect any visible change when switching between the two?

Hard to say, but interestingly using the 'reset' one, dispcalGUI reports it as only 99.6% of grayscale values. For 0-128 output=input. For 129-255 output=input-1. Using the 'linear' shows 100% of grayscale input=output 0-255. For all we know, that could just be a bug in dispcalGUI though.

I'm thinking the 'reset' style ramp one must be a legacy thing or misinterpretation, since looking back, that is what older calibration programs like ProfileMaker 5 use.

Maximum value for each color is 65280 instead of 65535. Why? Maximum number of 8-bit number is 0xFF (255). Maximum number of gradations for each channel is 256.

255 * 256 = 65280

A Google search came up with the above by a forum poster, but I'm not sure that is correct logic. 8-bit values range from 0-255 (256) which matches up with the 0-255 (256) gradations. Maybe there was a bug in old GPU hardware or drivers which made 8-bit 0x0 null and mapped 0x1 to 0 or was it just a case of poor documentation by Microsoft confusing people? Not something I think you need to worry about now, since there seems to have been a transition at some point to using 65535 as the maximum value in modern calibration software.

[alph@]

version 0,6 work , no error message,the function 'save as 3dlut ' is not available,only 'save ycms script',it's normal?

[nand chan]

Originally Posted by cyberbeing

Hard to say, but interestingly using the 'reset' one, dispcalGUI reports it as only 99.6% of grayscale values. For 0-128 output=input. For 129-255 output=input-1. Using the 'linear' shows 100% of grayscale input=output 0-255. For all we know, that could just be a bug in dispcalGUI though.

I'm thinking the 'reset' style ramp one must be a legacy thing or misinterpretation, since looking back, that is what older calibration programs like ProfileMaker 5 use.


A Google search came up with the above by a forum poster, but I'm not sure that is correct logic. 8-bit values range from 0-255 (256) which matches up with the 0-255 (256) gradations. Maybe there was a bug in old GPU hardware or drivers which made 8-bit 0x0 null and mapped 0x1 to 0 or was it just a case of poor documentation by Microsoft confusing people? Not something I think you need to worry about now, since there seems to have been a transition at some point to using 65535 as the maximum value in modern calibration software.

I'm just going to follow along with it for now, I have higher things on my priority list (such as getting version 0.6 out) than breaking my head over an imperceptible difference.

Originally Posted by alph@

version 0,6 work , no error message,the function 'save as 3dlut ' is not available,only 'save ycms script',it's normal?

Tried Update -> Update yCMS? If that doesn't work, it may be a restriction with Windows' security policy. I'll see if I can find a way around that.

[alph@]

ok work,i have updated ycms and launch the 'TI3ParserGUI.exe' with administrator Rights,you are correct,this week end I start the test with my sony lcd 46 hx 700 and the combo xrite -colorhcfr,thank very match.

[nand chan]

Going to release version 0.6 as-is for now, so you don't get caught up in the delays of a bunch of behind the scenes changes. I will be making greater use of the improved pipeline in 0.7.

Download link for 0.6: http://www.mediafire.com/?112ct1ijm2e2edz

[nand chan]

Experimental mock-up design of my proposed 3dLUT Creation Studio.

What do you think? I could also allow stuff like YCbCr -> RGB conversion blocks, or gamut mapping blocks. Then we wouldn't even need yCMS anymore.

Or you could do fun stuff like this:

[yesgrey]

Originally Posted by nand chan

Which means that I am now being bottlenecked by the limited capacity of the .3dlut file standard, even though my code is perfectly capable of processing something irregular like 10-bit LUTs (as opposed to 8, 16 or 32).

That "limited" capacity you referred was intentional.

The benefit of allowing bit depths other than those would be only a lower 3DLUT file size. We think that a 96 MB (8/16 3DLUT) is a pretty manageable size according to current computer standards, so we favoured a simpler file format structure. In addition to that, I performed some precision tests that showed which input/output bit depth combinations would be useful. For an 8 bit input bit depth using more than 16 bit output is not worth it. The 32 FP was added for future use, when/if we decide to use 9/32 3DLUT files (1.5 GB), but before that we need to have graphic cards with a minimum of 2 GB ram to contain the 3DLUT...

Furthermore, 32 bit FP was preferred over 32 bit integer because the precision is almost the same but 32 FP is much more compressible than 32 bit integer.

[nand chan]

Originally Posted by yesgrey

That "limited" capacity you referred was intentional.

The benefit of allowing bit depths other than those would be only a lower 3DLUT file size. We think that a 96 MB (8/16 3DLUT) is a pretty manageable size according to current computer standards, so we favoured a simpler file format structure. In addition to that, I performed some precision tests that showed which input/output bit depth combinations would be useful. For an 8 bit input bit depth using more than 16 bit output is not worth it. The 32 FP was added for future use, when/if we decide to use 9/32 3DLUT files (1.5 GB), but before that we need to have graphic cards with a minimum of 2 GB ram to contain the 3DLUT...

Furthermore, 32 bit FP was preferred over 32 bit integer because the precision is almost the same but 32 FP is much more compressible than 32 bit integer.

Yeah, it's reasonable enough. Something like a 8->10 bit LUT would be horrendously hard to work with if you just write 10 bits into the file at a time, and if you pad the data to 16 bit to make reading easier, might as well just use 16 bit precision.

That being said, the .3dlut file format has some other, ridiculously major flaws imo, because it's lacking a lot of meta-tags.

While these tags are not strictly speaking necessary for the operation of a LUT as such, many tools *do* want to be more content-aware (eg. is the input or output range 0-255 or 16-235?), a prime example being madVR itself - which scans through the parameters file to find out information.

Of course, this breaks as soon as it encounters a .3dlut not created by any known tool, even if the LUT itself would be in the correct format, simply because it isn't searching for the correct strings.

The result is that I have to pad my parameters data with “madVR compatibility hacks”.

Inclusion of this information, as well as stuff like gamut, gamma, and white point would be *critical* to a fully functional 3dLUT format.

Ps. Floating point can have another advantage: Since value ranges are normalized from 0 to 1, that makes gamut mapping independent of the level range. madVR arguably already works this way because it only works with limited range /output/ LUTs (and transforms these further), but it's a bit silly regardless.

Floating point has another advantage of that you're strictly limited to 0-1, you can introduce whiter than white / blacker than black information.

Edit: Also, what about 9->16 bit LUTs for eg. Hi10P content? Though I suppose it doesn't matter because madVR doesn't dither down to 8 bit precision, it interpolates the .3dlut results up to 10 bit instead.

I don't suppose it should make a significant difference.

[nand chan]

Concept of a proposed scripting language I was toying with, as an analog to the above flowchart GUI:

!Input_Range 16 235
!Input_Depth 8 8 8
!Output_Range 0 255
!Output_Depth 16

# Import the CalFile namespace, which gives us access to the CalFile() filter
!Import(Files.CalFile)

# Example .3dlut generation script
!Pixel {
  
  # Since this is going to be a YCbCr script, first convert to RGB using the BT.709 primaries
  ConvertToRGB(BT709)
  
  # Now we want to adjust for our screen's somewhat darker gamma curve
  Gamma(2.2,2.35)
  
  # And apply the calibration info from an argyll file
  CalFile("path_to_file.ti3")
  
  # And just for fun, grayscale it
  Grayscale()
  
}

Things to consider:

- Allow custom scripting logic or not? Eg. declaration of variables, creation of custom transform filters or just allow loading these via managed assemblies
- Maybe include the pixel pulldown / pullup logic into the !Pixel pipeline itself? The reason doing this might be bad though is it would disencourage creation of meta-data, which renderers such as madVR would rely on.

Interesting realization:

If I went through with this scripting language, it obviously wouldn't be limited to just .3dluts. I could, for example, save them as transformation pipeline packages (.tpp?) and apply these to images etc. manually.

I could also introduce filters such as ChangeGamut() or ChangeWhitepoint().

I've also put some thought into how to do multiplexing. In the most basic idea, single-pixel filters would be distinguished by some sort of extra markup in the name, for example a period or $ or _ or something.

# Single filters could be applied like this
Multiplex(BlockR, BlockG, BlockB)

# For example
Multiplex($Gamma(1.0), $Gamma(0.9), $Gamma(0.8))

# A null or noop operation could either be represented as “null”..
Multiplex($Gamma(0.5), null, null)
# ..left out..
Multiplex($Gamma(0.5), ,)
# or perhaps done with $Cat() or $Default() or something
Multiplex($Gamma(0.5), $Cat(), $Cat())

# Larger blocks of multiple filters would be joined by { }
Multiplex({ $Gamma(0.5) $Add(0.1) }, { $Gamma(0.8) $Add(0.2) }, $Cat())

# Which can be stretched over several lines
Multiplex({
  $Gamma(0.5)
  $Add(0.1)
}, $Cat(), $Cat())

# The basic idea here is that newlines are ignored
Gamma(
  2.2,
  2.35 )

Further thoughts: Maybe extend my scripting format to custom amounts of channels? For example, a 3 -> 4 channel convertor that would convert from RGB to CMYK.

This obviously wouldn't be saveable as a .3dlut due to the limitation of the file format, but I could still save them as .tpp and use for proofing or something.

Feedback / thoughts?

[yesgrey]

Originally Posted by nand chan

That being said, the .3dlut file format has some other, ridiculously major flaws imo, because it's lacking a lot of meta-tags.
...
Inclusion of this information, as well as stuff like gamut, gamma, and white point would be *critical* to a fully functional 3dLUT format.

3DLUT format spec is not complete yet. In the beginning the current state was more than enough, but now I'm perfectly aware that it's not. Unfortunately I currently don't have time to finish it, so we have to discuss it later.

Originally Posted by nand chan

Ps. Floating point can have another advantage: Since value ranges are normalized from 0 to 1, that makes gamut mapping independent of the level range.

No, it's not. The black and white with FP is not always 0.0 and 1.0. That happens only with levels 0-255.

Originally Posted by nand chan

Also, what about 9->16 bit LUTs for eg. Hi10P content?

Using a 9/16 (768MB) 3DLUT for a small increase in precision it's not worth it. The 9/32 (1536MB) shows a significant theoretical increase, but probably no one would be able see any difference, so it might not be worth it too...

[nand chan]

Originally Posted by yesgrey

No, it's not. The black and white with FP is not always 0.0 and 1.0. That happens only with levels 0-255.

I don't see what the motive for this decision is (I've included support for it regardless), any reason?

Since you're pulling down from 16-235 in the first place, why not go all the way and just pull them into 0-1? You actually increase precision instead of losing it, since values don't have to be rounded. The only motive for not normalizing integer 3dLUTs to full range is because you lose WTW/BTB information that way, but with FP it's not an issue.

Using a 9/16 (768MB) 3DLUT for a small increase in precision it's not worth it. The 9/32 (1536MB) shows a significant theoretical increase, but probably no one would be able see any difference, so it might not be worth it too..

Depends on how good the interpolation algorithm is I suppose. I'm currently using linear interpolation (between individual steps) which causes aliasing, but if you were to find out the gamma curve (again, meta-tags) and do a correct non-linear interpolation, the quality of interpolation should be significantly better.

I suppose it would make less of a difference the better the screen is.

[alph@]

i create a 3dlut file for ycms (curve +matrice-120cd-gamma2,35 -testchart 912-profil hight quality) Here are the results

left no ycms-right with,the deltaE after is best



the correction of gamma is very good


color correction is different,the delta luma red is very good,but the blue going from 9 to 15??,

[nand chan]

Originally Posted by alph@

color correction is different,the delta luma red is very good,but the blue going from 9 to 15??

Might just be a limitation of the screen, eg. you can't get correct colors unless you set a certain channel to be too bright (especially for black point correction).

At any rate, this would probably be more appropriate in the yCMS topic.

[alph@]

I use a gamma of 2,35 instead of 709 bt,it's a probleme? and why the diagramme cie is out in the green,ycms must compromise?
'you can't get correct colors unless you set a certain channel to be too bright (especially for black point correction)'
Excuse me, but what is the relation with the black point

[nand chan]

Originally Posted by alph@

I use a gamma of 2,35 instead of 709 bt,it's a probleme? and why the diagramme cie is out in the green,ycms must compromise?

I don't know this.

'you can't get correct colors unless you set a certain channel to be too bright (especially for black point correction)'
Excuse me, but what is the relation with the black point:helpful

Well, when you're on an LCD monitor, you have a certain level range (0-255) which you can't go outside of.

So if your black point (of 0/0/0) is too red, the logical thing you'd have to do is decrease the red channel (-5/0/0). However, this is simply physically not possible.

To compensate, to fix the tone balance, you have to increase the other channels instead, eg. (0/5/5).

However, this results in a “brighter” black than 0/0/0. In other words, fixing the black point's tone balance can offset its luminance (brightness).

That's why the luma error increases.

[nand chan]

Version 0.7 is out!

Changelog is in the first post as usual.

Download link: http://www.mediafire.com/?ugaa1y0o5a1452g

Since this release introduces LutScript files (.3ls), I will post an example file here until the formal documentation is complete:

######################################
## Example .3dlut generation script ##
######################################

# Set the properties properly
!Input_Range(Limited)
!Input_Depth(8,8,8)
!Output_Range(Limited)
!Output_Depth(16)

# An example transformation pipeline. Note that this !Pixel pipeline applies pulldown and range adjustment
# so you don't have to do it manually. All numbers are normalized to the range 0-1 within here.
!Pixel( {
  
  # Load the logic from an existing .3dlut
  LutFile("some_3dlut.3dlut")
  
  # And apply the calibration info from an argyll file
  CalFile("path_to_file.ti3")
  
  # And just for fun, grayscale it
  Grayscale()
  
} )

List of current functions: (All are case-insensitive)

Meta / top-level functions:
!Input_Range(Full | Limited): Sets the range appropriately
!Input_Depth(Number, Number, Number): Sets the depth appropriately
!Output_Range(Full | Limited): Sets the range appropriately
!Output_Depth(Number): Sets the depth appropriately
!Pixel(Function): Creates a pixel transformation pipeline (with appropriate adjustments) from the given function.
                  Multiple of these can be “glued” together to form a block by using { and }

Pixel functions:
Grayscale(): Converts to grayscale
CalFile(String): Applies calibration information from a given .cal or .ti3 file
LutFile(String): Applies a LUT transformation using a given .3dlut
Constant(Number, Number, Number): Outputs a constant value, for example (1, 1, 1) to create a LUT that blanks to white
Invert(): Inverts the input

The .3dlut is calculated by appending each !Pixel() function together (without further transformations), then mapping each possible value through the list.

Do note that to use the generated .3dlut with madVR you need to tag it with BT.709 primaries first using tag3dlut -7

All LutFile() functions are completely independent of each other and detect ranges/depth for their own .3dlut, that is, you could mix a 16-bit 3dlut with a 10-bit 3dlut and an 8-bit 3dlut freely, or mix together limited and full range .3dluts. It really doesn't matter one bit, as long as they are tagged correctly.

To-do / planned functions:

ConvertFromYCbCr(Primaries): Would convert from YCbCr to RGB
Gamma(): Would adjust values using a given gamma coefficient
Swap(Number, Number): Would swap the two channels
Cat(): Would return the input as output, useful for where you want a .3dlut to do nothing

To show how this LutScript format gradually replaces the dedicated functions for .3dlut generation / modification, I have shown some example !Pixel scripts that implement the functionality currently found in other programs:

applycal.exe:

!Pixel({
  LutFile("input.3dlut")
  CalFile("calfile.cal")
})

merge3dlut.exe:

!Pixel({
  LutFile("file1.3dlut")
  LutFile("file2.3dlut")
  LutFile("file3.3dlut")
  # ...
})

changedepth:

# Set the appropriate range/depth for the new .3dlut

!Pixel(LutFile("old.3dlut"))

[alph@]

'Do note that to use the generated .3dlut with madVR you need to tag it with BT.709 primaries first using tag3dlut -7'

it gets complicated,So you can not use the 3dlut directly in madvr?a small tuto would cool,because I'm confused
Your software is great,but utilities annexes are mysterious,I wish I could use it if necessary.

[nand chan]

Originally Posted by alph@

'Do note that to use the generated .3dlut with madVR you need to tag it with BT.709 primaries first using tag3dlut -7'

it gets complicated,So you can not use the 3dlut directly in madvr?a small tuto would cool,because I'm confused
Your software is great,but utilities annexes are mysterious,I wish I could use it if necessary.

It's an issue with madVR and the .3dlut format in particular rather than the programs, so I'm waiting on madshi to add native compatibility checking so you won't have to do it manually. I've chosen to not add madVR compatibility hacks for now because I don't want to botch up the format further than it has to be.

If he still hasn't added support for it, then I'll add a flag to lutscript.exe to add madVR compatibility (or maybe include it by default).

For now, just do this:

lutscript.exe -i input.3ls -o output.3dlut
tag3dlut.exe -i output.3dlut -7

Or for a shorter version:

lutscript -i input.3ls -o - | tag3dlut -7 -i - -o output.3dlut

The command line utilities are meant for power users that know exactly what they're doing and delivers programs to support the necessary logic so even those people who aren't programmers with free time can benefit from it.

The GUI is intended for simple usage. It's just that I haven't gotten around to actually updating it yet due to lack of interest. I have far more planned for the GUI.

[alph@]

thank,
is what you plan to support icm files in TI3Parser,It would be very convenient.

[nand chan]

Originally Posted by alph@

thank,
is what you plan to support icm files in TI3Parser,It would be very convenient.

I am planning on that, yes. An estimate would be around version 0.9.