.ti3 Parser / .3dlut Generator for madVR + .3dlut toolset

[nand chan]

TI3 Parser (v0.16)

Information

I hereby present my TI3 Parser toolset. This collection of programs is intended to provide a somewhat easier front-end for color management using ArgyllCMS¹, yCMS and madVR, as well as a small collection of command line tools. The motivation for creating this was due to the steps involved in locating, opening and extracting the information from a .ti3 file, then properly formatting them so madVR or yCMS accept them was a bit too daunting to explain well in my madVR CMS guide. This program initially started out as a script to simply format the values properly, but I decided to refactor it and enlarge the scope to include a largely automatized 3dLUT generation process. This also makes it easy to generate multiple .3dLUTs for different environments and choose a certain one easily using madVR (instead of having to copy/paste over the new primaries + re-generate it every time).

All logic is also abstracted into a managed .dll so that those who wish to use it in other projects can do so, and there is a collection of (somewhat) useful command line tools included.

1. At the moment, ArgyllCMS is still not automatized - you will have to calibrate+profile manually using dispcalGUI

Features:

This program will:

• Read and parse a .ti3/.cal file as generated by ArgyllCMS
• Present the gamut/grayscale measurements in madVR-friendly form
• Provide a yCMS generation script
• Create a madVR-compatible .3dlut file
• Modify an existing .3dlut by swapping channels, inverting it or applying calibration data from an ArgyllCMS .cal file
• Create new .3dluts
• Allow inspection and modification of .3dluts (to check for errors etc)
• Automatically update yCMS
• Scale/adapt the output bit depth or value range of .3dluts
• Merge multiple .3dluts into a single one
• Inspect, Reset, Load or Save the device adapter's CLUT tables
• Allow application of 3dlut files to images manually
• Create .3dluts from linking ICC profiles together
• Perform, gamma correction, gamut mapping, chromatic saturation (eg. to increase the gamut range while preserving colors) and chromatic adaptation
• Join together .3dluts using a composite algorithm (eg. to fix the black levels)
• Work with both .3dlut and .3dl2 files
• Be fully accessible from the command line console

Usage guide:

I intend for this section to be obsolete by version 1.0, but for now, a quick guide towards calibrating your monitor / profiling it properly can be found here or here.

If you just want the GUI (shown in the screenshot), after extracting, run TI3Parser\bin\LinkICCGUI.exe.

A list of current command line utilities:

• applycal: Applies an ArgyllCMS .cal/.ti3 or a CLUT file to a .3dlut, thus calibrating it for a specific display device
• changedepth: Changes the output bit depth of a .3dlut or adjusts the levels (PC or TV) - note that this transformation is lossy
• dispclut: Allows inspection, changing, resetting, saving or loading of a display adapter's CLUT tables (globally or per-monitor)
• gen3dlut: Generates a new, linear or grayscale .3dlut
• genscript: Generates a yCMS script from .ti3 source value, using the same base as madVR
• imagecal: Apply a .cal file or .3dlut to a bitmap image manually, useful for proofing
• inspect3dlut: Inspects a .3dlut and prints information about the meta-tags or creation script
• join3dlut: Composites together two .3dluts, switching between the two based on luminance.
• linkicc: Links two ICC profiles together and generates a .3dlut from the result
• lutscript: Generates a 3dlut from a provided .3ls file or .3ls2 file
• make3dlut: Generates a .3dlut from a .ti3 file, using yCMS (can also apply calibration data)
• merge3dlut: Merges multiple .3dluts together into a single file
• parseti3: Opens a .ti3 file and presents the primaries / grayscale information in madVR-friendly form. Also allows copying to clipboard
• swapchan: Allows swapping or inverting channels of a .3dlut
• tag3dlut: Modifies a 3dlut's attached parameters to change the declared levels or primaries (this does not modify the 3dlut itself, unlike changedepth)
• updateycms: Updates this toolset's copy of yCMS
• version: Prints the overall packaging version number

Help for each program can be obtained by running <program name> --help.

To-do (planned features list):

• Find a better name for the project, since soon it won't be limited to just parsing TI3 files
• Integration and updating of the binaries provided by ArgyllCMS
• Automation for ArgyllCMS, similar in nature to dispcalGUI but tailored for video playback
• Self-update system (will likely need a decent web host for this first)
• If possible, interface with madVR and set the .3dlut files automatically (perhaps per display device)

Download and license information

All code is currently in public domain but I withhold the rights to change this license at any time (license changes will not apply retroactively).

• Executables and source code: Download from mediafire (packaging v0.16)

All executables can be found in the TI3Parser\bin folder. The program requires the .NET library version 4.0 or above to run.

Feedback

Please post any bugs, issues, suggestions or other sorts of feedback in this thread. If you want to submit source code patches you can do that too.

I'm also not quite satisfied with the current interface. What would you prefer - the current, a tab-based layout or a bunch of buttons like it was before? I'm also probably going to make the interface much larger, add more regions for everything, allow loading multiple files and generally expand the program into a larger, fullscreen management studio instead of a simple tool.

I'm also taking suggestions for a better name, since it is no longer a simple TI3 Parser

Changelog

Moved to second post due to length issues

[alph@]

very good,when you use 'rec709' curve, you use an adjustement of ambient light level ?

[nand chan]

Originally Posted by alph@

very good,when you use 'rec709' curve, you use an adjustement of ambient light level ?

No, instead, I make sure my ambient light level is as close to zero as possible. In other words, I watch anime in a nearly perfectly dark room.

In fact, I've discovered that calibrating to a Rec.709 level makes madVR adjust more using its 3dlut than calibrating to an sRGB or even 2.35 tone curve, so I'm thinking of changing that. I have it calibrated to the D65 sRGB curve right now and the 3dlut doesn't make a huge difference, so I know it's only minor corrections.

I'm thinking of having the program calibrate to a set gamma instead (allowing you to choose between “bright, medium and dark” room or something)

Changelog

Version numbers refer to the packaging version, not the version numbers of the individual components.

*all new changes will be in the release post, vBulletin caps the length of posts unfortunately*

Version 0.10:
! Added basic support for ICC profiles
+ Color space information is extracted from the ICC profiles and written into the resulting .3dl2
+ New program: linkicc.exe
+ New LutScript function: IccTransform(String, String, Intent), intents: Perceptual, Relative, Saturation or Absolute
+ New LutScript function: !Filetype(Type), possible values: 3DL2 or 3DLUT
+ New LutScript function: !Parameters(String), overrides the parameters data (not recommended for 3DLUT filetype)
+ Added support for LZO compression (CompressionMethod=1). This is used by default.
? Note: Legacy LUTs do not support compression, it will always be disabled when using --legacy mode in tag3dlut
+ Added a --compression (-c) flag to tag3dlut, valid values are: None, LZO
+ New LutScript function: !Compression(Method), possible values: None, LZO
+ LutScript now computes the intermediate gamut after each step (this changes when using eg. IccTransform())
  It is recommended to *NOT* set the !Input_Primaries or !Output_Primaries tags when performing gamut mapping!
  LutScript will auto-detect these correctly for you, eg. when mapping together multiple ICC profiles
~ LutScript now defaults to Limited range when not specified, since it just makes more sense for video content
? Changelog v0.9 was flawed, !Input_Primaries did not accept a string but an /enum/
  (eg. “!Input_Primaries(BT709)” and not “!Input_Primaries("BT709")”)

Version 0.9:
! Created and added support for the new .3dl2 format, changes include:
  + Added support for 64-bit floating point LUTs
  + Added support for color-space aware LUTs
  + Added support for RGB/YCbCr -> XYZ LUTs for profiling purposes, similar to how ICC device profiles work
  + Added support for file-encoded value ranges
* Utilities such as gen3dlut and changedepth which create new LUTs now use the 3dl2 by default,
  so you no longer need to fix tagging using tag3dlut
? Note: Since programs such as madVR still only support .3dluts, you need to “convert” these files for those programs
+ --legacy (-l) tag added to tag3dlut in order to convert to a legacy .3dlut, this also overwrites the params
  in order to correctly tag everything for madVR etc.
+ --color-space (-c) flag added to inspect3dlut
+ !Input_Primaries(rx, ry, gx, gy, bx, by, wx, wy) tag added to LutScript
+ !Input_Primaries(ColorSpace) overload added to LutScript, acceptable values are currently: BT709 and None
+ Same as above for !Output_Primaries() added as well
? I aim to make the input_primaries and output_primaries tags obsolete sooner or later though, when I introduce
  gamut conversion functions
+ --ycbcr (-y) flag added to gen3dlut to generate a blank YCbCr LUT
* ColorTriple<T> is now aware of its encoding, in order to ensure transformations are properly chained
  eg. the grayscale filter now simply sets Cr and Cb to 0 if the input is already YCbCr
~ Significantly increased performance here and there by operating on a single value in memory instead of copying it
  over and over, however the old style of doing it is still available for cases where it's needed
- Removed the static methods .FromFile() and .FromStream(), moved logic into constructor instead
* !Pixel() no longer has to be the last function in a .3ls, meta-tags can appear after it
* make3dlut now automatically converts the yCMS output to a .3dl2, use tag3dlut to convert it back if desired
* Disabled the “Update program” field of the GUI, this is not supposed to do anything yet

Version 0.8:
* 3dluts now detect input and output range independently, allowing for a TV -> PC or PC -> TV 3dlut.
* Limited range floating point 3dluts work correctly now, the white level was set too high
- Removed the level override flag from applycal, it was in the wrong place - if you want to override levels, use
  tag3dlut to modify the parameters data
- Removed many functions from _3dLut since transformation factories deprecate them, changes affect:
  .Map(), .Assign(), .Modify(), .ChangeDepth(), .ChangeRange(), .Transform()
- Removed all of the overloaded Pulldown / Pullup functions from ColorTriple (which has been renamed Color). Instead of
  using these functions, the proper method is to create a Pulldown or Pullup transformer (available in Triple variants)
+ Due to the above changes, level limits for pulldown/pullup are now precomputed on demand instead of at program start
+ LUTs are now theoretically unlimited, you could create a 32-bit LUT if you wanted to (and had 950737 YB of RAM)
- Removed the CalFileTransformer and _3dLutTransformer classes, implemented ITransformationProvider instead
  To transform an arbitrary color, first get the transformation using .GetTransformer(), then .Invoke() that
* Updated changedepth to use a transformer to handle range conversions which uses the internal transformer; due
  to this change, range conversions will *no* longer result in banding.
? Do note that since interpolation is still linear you will encounter macro-banding when performing extreme interpolation
  If you're pulling up something like a 3-bit LUT to an 8-bit LUT, you should encounter it. With a 4-bit LUT (16x16x16)
  I noticed no significant banding in a fullscreen gradient, but it might be more pronounced with less steep ramps
? Note that changedepth does not correctly tag the resulting LUT (in fact it strips all parameters). I won't change this
  until the version 2 spec of .3dlut comes out, for now just use tag3dlut.
- Tags --full and --limited removed from tag3dlut
+ Tags --input-range and --output-range added to tag3dlut instead.

Version 0.7:
! Addition of LutScript, a scripting language for generating .3dluts from scratch. Documentation will follow.
? Transformation facility now also works in per-channel mode, as opposed to per-color (for the multiplexer)
* 3d LUTs now interpolate their values (linearly), this allows for full 64-bit floating point precision
  Note that I have not yet done any gamma detection and curve adjustment, so this will still generate minor aliasing
+ Added logic to transform .3dluts further (that is, operate on its /output/)
+ New program: lutscript.exe
+ New program: merge3dlut.exe
~ More programs updated to use the transformation factory instead of hard logic

Version 0.6:
! All .3dlut code is now multi-threaded and will run many times faster
   (do note that loading and saving a 100 MB file still takes around a second on modern hard drives)
! Transformations are now abstracted behind ITransformationProvider and can be assembled into TransformationFactories
  Most program logic has been rewritten to use this functionality, and a GUI version for .3dlut creation
  should be coming sometime soon
? dispclut can save and modify the CLUT at the same time now,
   eg. dispclut -d 1 --save old.txt --reset before playback and dispclut -d 1 --load old.txt afterwards
+ New program: imagecal.exe
+ Beta functionality for 3dlut color transformations using 3dLutLib added, note that does not yet properly
   interpolate the values. Once I've rewritten it to do so, I'll rewrite .ChangeRange to use this, thus
   eliminating its current banding as well - which will affect changedepth.exe (when using the -r flag)
* make3dlut's built-in --calibrate option now works properly
* All tools now work with limited-range floating point values as well (0.0625 as black and 0.918 as white, approximately)
* 3dLutLib correctly detects full-range LUTs created with the RGB_PC format flag (instead of Output_Range 0 255)
* gen3dlut no longer prints luma information to screen with --grayscale, was left over from debugging
* The parser for .ti3/.cal files is now culture-invariant, to solve an issue where it would fail to
  parse the values in some european locales which used a comma instead of a period to specify decimals
+ dispclut now has an “--reset-all” flag
* dispclut now uses the linear 0-65535 levels

Version 0.5:
! Re-worked entire .3dlut code base, much cleaner now (at the cost of a tiny speed loss)
* Detection for range (full or limited) is now handled by the 3dLUT loader itself and should be more consistent
   In addition, this information will be displayed by the inspect3dlut program
+ applycal can load .ti3 files as well, and...
+ ..make3dlut can automatically apply .cal data from a .ti3 if found
+ applycal now supports Calibration Tester's #GammaRamp format
+ 32-bit floating point LUTs are now supported in all applicable programs
+ New program: changedepth.exe
   Do note that while changedepth can pull values up/down to full or limited, this introduces a lot of banding
+ New program: dispclut.exe
+ New program: tag3dlut.exe

Version 0.4:
* Saving/loading to stdin or stdout now works properly
* Applycal now (attempts to) correctly detect the input value range, and also works for 8-bit LUTs (not just 16-bit)
* The TI3 parsing process now actually converts the entries to numbers,
   removing the need for ugly string detection hacks - should fix the duplicate key bug also
+ It's now possible to generate new .3dluts from scratch using the TI3ParserLib
+ New program: inspect3dlut.exe
+ New program: gen3dlut.exe
? Speed improvements here and there

Version 0.3:
+ New program: applycal.exe
+ New program: swapchan.exe
+ New program: version.exe

Version 0.2:
? Now requires the .NET version 4.0 to run properly
~ Updated UI, removed ugly buttons and added a menu
+ New program: genscript.exe
+ New program: make3dlut.exe
+ New program: parseti3.exe
+ New program: updateycms.exe
* Temporary files are properly removed now
* Updating yCMS now correctly downloads the 64-bit or 32-bit versions depending on your processor architecture

Version 0.1:
? Initial release

Legend:
! Major modification
+ New feature
- Removed feature
* Fixed bug
~ Sideways change
? Information

[alph@]

ok,what values you use in ambient light level,Have you tried the 'black point correction'. I usually made ​​a calibration (d6500-120cd/m-gama 2.35- black level native-calbration quality high-testchart 124)and i check with color hcfr.

[leeperry]

looks good! if you could allow merging LUT's like one for gamut mapping and one from Argyll off a .cal file, that'd be full of win

and if you could allow importing the primaries/secondaries saturations from Color HCFR, then it'd be



maybe none of this is technically possible though, and I need to document myself on the .ti3 file format

An option to output 8bit 3DLUT's could be useful for the happy few using rgb3dlut() within the Avisynth filter of ffdshow.

[nand chan]

Originally Posted by leeperry

looks good! if you could allow merging LUT's like one for gamut mapping and one from Argyll off a .cal file, that'd be full of win

and if you could allow importing the primaries/secondaries saturations from Color HCFR, then it'd be



maybe none of this is technically possible though, and I need to document myself on the .ti3 file format

An option to output 8bit 3DLUT's could be useful for the happy few using rgb3dlut() within the Avisynth filter of ffdshow.

Calibration is done using the TI1 files, I could do that though - just generate two .3dluts and link them together.

Also the 8 bit 3DLUTs is also possible.

[nand chan]

Originally Posted by leeperry

looks good! if you could allow merging LUT's like one for gamut mapping and one from Argyll off a .cal file, that'd be full of win

and if you could allow importing the primaries/secondaries saturations from Color HCFR, then it'd be



maybe none of this is technically possible though, and I need to document myself on the .ti3 file format

An option to output 8bit 3DLUT's could be useful for the happy few using rgb3dlut() within the Avisynth filter of ffdshow.

While thinking about this, I realized that it would be pretty hard without manual adjustment because ArgyllCMS works with 0-255 values, and madVR works with 16-235 levels. You can't simply generate a .3dlut off the .cal and combine it to the madVR .3dlut due to that reason.

I could possible morph the 3dlut by transforming it from 0-255 to 16-235, but that's significantly beyond the scope of this project and I don't intend to do any complex color management logic myself - just tie together existing tools (like yCMS).

Merging noncompatible 3dluts would be something yCMS would have to do, and you should request for that feature there.

Originally Posted by alph@

ok,what values you use in ambient light level,Have you tried the 'black point correction'. I usually made ​​a calibration (d6500-120cd/m-gama 2.35- black level native-calbration quality high-testchart 124)and i check with color hcfr.

I've never used black point correction, mainly because my black point is dark enough and doesn't show any noticeable color distortion.

I usually use 6504K (I enter the X/Y points manually) - 6500K is a bit redder than D65 - and 120 cd/m², sRGB tone curve, native black level, native black point and massive LUT testchart.

[yesgrey]

Congratulations for your tool!

[leeperry]

Originally Posted by nand chan

While thinking about this, I realized that it would be pretty hard without manual adjustment because ArgyllCMS works with 0-255 values, and madVR works with 16-235 levels. You can't simply generate a .3dlut off the .cal and combine it to the madVR .3dlut due to that reason.

I could possible morph the 3dlut by transforming it from 0-255 to 16-235, but that's significantly beyond the scope of this project and I don't intend to do any complex color management logic myself - just tie together existing tools (like yCMS).

Merging noncompatible 3dluts would be something yCMS would have to do, and you should request for that feature there.

TBH I use 0-255 LUT's using an old version of cr3dlut, but I'm quite sure yCMS could create them as well...the only problem would be mVR that might expect 16-235 LUT's indeed, but if you want automatic gamut mapping depending on the frame rate/resolution, the only solution atm is to use rgb3dlut(8bit) or t3dlut(16bit) within ffdshow automatic profiles.

Several ppl already requested the feature of embedding together within the same 3DLUT both the Argyll calibration and the gamut mapping, so mVR's dithering wouldn't be messed around by the graphic card's CLUT...this would require an option in mVR to disable the latter, though.

Anyway this was just some plain brainstorming, and you're the author...making you the final judge in the end

[nand chan]

Version 0.1.1 is out now that I have gained yesgrey's acceptance of the yCMS download method

Originally Posted by leeperry

TBH I use 0-255 LUT's using an old version of cr3dlut, but I'm quite sure yCMS could create them as well...the only problem would be mVR that might expect 16-235 LUT's indeed, but if you want automatic gamut mapping depending on the frame rate/resolution, the only solution atm is to use rgb3dlut(8bit) or t3dlut(16bit) within ffdshow automatic profiles.

True, but I dislike the usage of ffdshow because it does not yet have a native 10-bit pipeline, also that would require working with YUV LUTs instead of RGB luts unless you want to use ffdshow's piss poor RGB conversion. I'll still consider adding 3dLUT combining as a future feature but it's not high on the priority list because there are too many variables to be careful of when combining them, and it's not easily doable with madVR LUTs. (unless madVR allows working with 0-255 LUTs in the future)

It's fairly straightforward to /create/ the correct LUTs using yCMS, it's just that madVR will reject the .3dlut.

Several ppl already requested the feature of embedding together within the same 3DLUT both the Argyll calibration and the gamut mapping, so mVR's dithering wouldn't be messed around by the graphic card's CLUT...this would require an option in mVR to disable the latter, though.

This would indeed be very important since iirc the CLUTs are only one-dimensional. The main problem would be generating a proper 16-235 .3dlut from .cal data, which probably lies within the domain of a tool such as yCMS. I do not (yet) have the necessarily technical knowledge to interpolate and generate tables like that.

My first priority now though is getting a working, automatized front-end for Argyll and yCMS in place to bridge the process from profiling to .3dlut creation.

[leeperry]

Oops yeah, indeed I use YUY2 LUT's in ffdshow so I guess merging them w/ a .cal file would turn into headaches, if even possible.

And tbh I'm very happy w/ tritical's yv12toyuy2() because it allows you to choose what resizing algorithm to use for the YV12>YUY2 interpolation. I've finetuned all thoses settings to get the crispiest non-ringing PQ I've ever seen. I use SmoothL() to dither nicely and avoid banding, I'm not hung up on 10bit processing anymore...especially when dealing w/ 16-235 SMPTE-C content, because it might be a different story w/ xvYCC but we're not quite there yet.

[nand chan]

Originally Posted by nand chan

This would indeed be very important since iirc the CLUTs are only one-dimensional. The main problem would be generating a proper 16-235 .3dlut from .cal data, which probably lies within the domain of a tool such as yCMS. I do not (yet) have the necessarily technical knowledge to interpolate and generate tables like that.

My first priority now though is getting a working, automatized front-end for Argyll and yCMS in place to bridge the process from profiling to .3dlut creation.

Update: Just realized that Argyl''s .cal files are one-dimensional LUTs as well, so the practical gains of this are questionable. It would only serve to eliminate the video card's CLUTs, which are potentially sub-optimal.

Applying a .cal to a .3dlut seems like a simple process then, I'll write the code for it sooner or later.

Generating a .3dlut from a .ti1 however would be much harder since it requires significant interpolation.

Updated the UI slightly also: http://img805.imageshack.us/img805/7...parserdemo.png

[nand chan]

Bumping with a semi-big update, version 0.2 is now out. See the initial posts for a changelog.

Download link: http://www.mediafire.com/?9by785si95tu73b (source is included)

[madshi]

Thank you, nice work...

[ryrynz]

Thank you. just a minor issue, the about box for 0.2 shows version number 0.1.1.4186.

[Audionut]

Spent about an hour last night trying to get everything working again with madVR after not using it for awhile.

Then come across this one-click solution this morning

Nice work and thanks.

Originally Posted by ryrynz

the about box for 0.2 shows version number 0.1.1.4186.

I see version 0.2.0.36408

[ryrynz]

Different naming (no space) for the new version, must have deleted the wrong folder. All good, thanks.

[nand chan]

HUGE SUCCESS

I've finally finished writing all of the code for loading, editing (mapping a function over each entry) and saving .3dlut files, as well as the code for loading and properly interpolating a .cal file. All of this takes place in little under a second of processing time (and they said C# was slow).

As proof of concept, I've edited my existing .3dlut and swapped the red and blue channels just for fun. Here's the somewhat amusing result: http://img5.imageshack.us/img5/5215/chanswap.jpg

Expect to see a fully working version that will apply .cal calibration data to a .3dlut sometime tomorrow, as well as a few utilities to swap channels for fun.

For now, more fun stuff:
chanswap.exe -i test.3dlut --invert produces: http://img192.imageshack.us/img192/2923/chaninvert.jpg

Originally Posted by ryrynz

Thank you. just a minor issue, the about box for 0.2 shows version number 0.1.1.4186.

While this was not intentional, do note that the packaging version (0.2) is not necessarily the version of the GUI program (in this case, 0.2.0.36408) - I'll update it in the future to include both. The packaging version is merely a version number for the overall collection and will get increased with every release (similarly to madVR's current 0.74 version number). The versions of the individual programs (eg. the GUI) will be following a more traditional major.minor.iteration.build version number based on features on that specific program.

[Audionut]

Originally Posted by nand chan

Expect to see a fully working version that will apply .cal calibration data to a .3dlut sometime tomorrow

Excellent. Can't wait to try it out.

[nand chan]

Another small update, 0.3 is out.

With this release, it's now possible to directly alter .3dlut files by applying a .cal file or swapping the levels around (for fun).

Unfortunately, there is currently no GUI front-end for this functionality - I am waiting until I've come up with a better GUI design before I add this. However, it's easily done via the command line:

applycal.exe -i source.3dlut -o destination.3dlut calibrationfile.cal

Edit: I just realized this technique has another advantage. Since I'm interpolating the values to a 16-bit LUT, it transcends the precision of the graphics card's built-in 8-bit CLUTs. So if you have a 10-bit monitor (like I do), and an appropriate graphics card (like I don't), your calibration corrections will preserve that depth when merged onto a .3dlut.