Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
20th August 2011, 06:25 | #1 | Link |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
.ti3 Parser / .3dlut Generator for madVR + .3dlut toolset
TI3 Parser (v0.16)
Information I hereby present my TI3 Parser toolset. This collection of programs is intended to provide a somewhat easier front-end for color management using ArgyllCMS¹, yCMS and madVR, as well as a small collection of command line tools. The motivation for creating this was due to the steps involved in locating, opening and extracting the information from a .ti3 file, then properly formatting them so madVR or yCMS accept them was a bit too daunting to explain well in my madVR CMS guide. This program initially started out as a script to simply format the values properly, but I decided to refactor it and enlarge the scope to include a largely automatized 3dLUT generation process. This also makes it easy to generate multiple .3dLUTs for different environments and choose a certain one easily using madVR (instead of having to copy/paste over the new primaries + re-generate it every time). All logic is also abstracted into a managed .dll so that those who wish to use it in other projects can do so, and there is a collection of (somewhat) useful command line tools included. 1. At the moment, ArgyllCMS is still not automatized - you will have to calibrate+profile manually using dispcalGUI Features: This program will:
Usage guide: I intend for this section to be obsolete by version 1.0, but for now, a quick guide towards calibrating your monitor / profiling it properly can be found here or here. If you just want the GUI (shown in the screenshot), after extracting, run TI3Parser\bin\LinkICCGUI.exe. A list of current command line utilities:
Help for each program can be obtained by running <program name> --help. To-do (planned features list):
Download and license information All code is currently in public domain but I withhold the rights to change this license at any time (license changes will not apply retroactively).
All executables can be found in the TI3Parser\bin folder. The program requires the .NET library version 4.0 or above to run. Feedback Please post any bugs, issues, suggestions or other sorts of feedback in this thread. If you want to submit source code patches you can do that too. I'm also not quite satisfied with the current interface. What would you prefer - the current, a tab-based layout or a bunch of buttons like it was before? I'm also probably going to make the interface much larger, add more regions for everything, allow loading multiple files and generally expand the program into a larger, fullscreen management studio instead of a simple tool. I'm also taking suggestions for a better name, since it is no longer a simple TI3 Parser Changelog Moved to second post due to length issues
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 11th November 2011 at 16:41. |
20th August 2011, 13:27 | #3 | Link | |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Quote:
In fact, I've discovered that calibrating to a Rec.709 level makes madVR adjust more using its 3dlut than calibrating to an sRGB or even 2.35 tone curve, so I'm thinking of changing that. I have it calibrated to the D65 sRGB curve right now and the 3dlut doesn't make a huge difference, so I know it's only minor corrections. I'm thinking of having the program calibrate to a set gamma instead (allowing you to choose between “bright, medium and dark” room or something) Changelog Version numbers refer to the packaging version, not the version numbers of the individual components. Code:
*all new changes will be in the release post, vBulletin caps the length of posts unfortunately* Version 0.10: ! Added basic support for ICC profiles + Color space information is extracted from the ICC profiles and written into the resulting .3dl2 + New program: linkicc.exe + New LutScript function: IccTransform(String, String, Intent), intents: Perceptual, Relative, Saturation or Absolute + New LutScript function: !Filetype(Type), possible values: 3DL2 or 3DLUT + New LutScript function: !Parameters(String), overrides the parameters data (not recommended for 3DLUT filetype) + Added support for LZO compression (CompressionMethod=1). This is used by default. ? Note: Legacy LUTs do not support compression, it will always be disabled when using --legacy mode in tag3dlut + Added a --compression (-c) flag to tag3dlut, valid values are: None, LZO + New LutScript function: !Compression(Method), possible values: None, LZO + LutScript now computes the intermediate gamut after each step (this changes when using eg. IccTransform()) It is recommended to *NOT* set the !Input_Primaries or !Output_Primaries tags when performing gamut mapping! LutScript will auto-detect these correctly for you, eg. when mapping together multiple ICC profiles ~ LutScript now defaults to Limited range when not specified, since it just makes more sense for video content ? Changelog v0.9 was flawed, !Input_Primaries did not accept a string but an /enum/ (eg. “!Input_Primaries(BT709)” and not “!Input_Primaries("BT709")”) Version 0.9: ! Created and added support for the new .3dl2 format, changes include: + Added support for 64-bit floating point LUTs + Added support for color-space aware LUTs + Added support for RGB/YCbCr -> XYZ LUTs for profiling purposes, similar to how ICC device profiles work + Added support for file-encoded value ranges * Utilities such as gen3dlut and changedepth which create new LUTs now use the 3dl2 by default, so you no longer need to fix tagging using tag3dlut ? Note: Since programs such as madVR still only support .3dluts, you need to “convert” these files for those programs + --legacy (-l) tag added to tag3dlut in order to convert to a legacy .3dlut, this also overwrites the params in order to correctly tag everything for madVR etc. + --color-space (-c) flag added to inspect3dlut + !Input_Primaries(rx, ry, gx, gy, bx, by, wx, wy) tag added to LutScript + !Input_Primaries(ColorSpace) overload added to LutScript, acceptable values are currently: BT709 and None + Same as above for !Output_Primaries() added as well ? I aim to make the input_primaries and output_primaries tags obsolete sooner or later though, when I introduce gamut conversion functions + --ycbcr (-y) flag added to gen3dlut to generate a blank YCbCr LUT * ColorTriple<T> is now aware of its encoding, in order to ensure transformations are properly chained eg. the grayscale filter now simply sets Cr and Cb to 0 if the input is already YCbCr ~ Significantly increased performance here and there by operating on a single value in memory instead of copying it over and over, however the old style of doing it is still available for cases where it's needed - Removed the static methods .FromFile() and .FromStream(), moved logic into constructor instead * !Pixel() no longer has to be the last function in a .3ls, meta-tags can appear after it * make3dlut now automatically converts the yCMS output to a .3dl2, use tag3dlut to convert it back if desired * Disabled the “Update program” field of the GUI, this is not supposed to do anything yet Version 0.8: * 3dluts now detect input and output range independently, allowing for a TV -> PC or PC -> TV 3dlut. * Limited range floating point 3dluts work correctly now, the white level was set too high - Removed the level override flag from applycal, it was in the wrong place - if you want to override levels, use tag3dlut to modify the parameters data - Removed many functions from _3dLut since transformation factories deprecate them, changes affect: .Map(), .Assign(), .Modify(), .ChangeDepth(), .ChangeRange(), .Transform() - Removed all of the overloaded Pulldown / Pullup functions from ColorTriple (which has been renamed Color). Instead of using these functions, the proper method is to create a Pulldown or Pullup transformer (available in Triple variants) + Due to the above changes, level limits for pulldown/pullup are now precomputed on demand instead of at program start + LUTs are now theoretically unlimited, you could create a 32-bit LUT if you wanted to (and had 950737 YB of RAM) - Removed the CalFileTransformer and _3dLutTransformer classes, implemented ITransformationProvider instead To transform an arbitrary color, first get the transformation using .GetTransformer(), then .Invoke() that * Updated changedepth to use a transformer to handle range conversions which uses the internal transformer; due to this change, range conversions will *no* longer result in banding. ? Do note that since interpolation is still linear you will encounter macro-banding when performing extreme interpolation If you're pulling up something like a 3-bit LUT to an 8-bit LUT, you should encounter it. With a 4-bit LUT (16x16x16) I noticed no significant banding in a fullscreen gradient, but it might be more pronounced with less steep ramps ? Note that changedepth does not correctly tag the resulting LUT (in fact it strips all parameters). I won't change this until the version 2 spec of .3dlut comes out, for now just use tag3dlut. - Tags --full and --limited removed from tag3dlut + Tags --input-range and --output-range added to tag3dlut instead. Version 0.7: ! Addition of LutScript, a scripting language for generating .3dluts from scratch. Documentation will follow. ? Transformation facility now also works in per-channel mode, as opposed to per-color (for the multiplexer) * 3d LUTs now interpolate their values (linearly), this allows for full 64-bit floating point precision Note that I have not yet done any gamma detection and curve adjustment, so this will still generate minor aliasing + Added logic to transform .3dluts further (that is, operate on its /output/) + New program: lutscript.exe + New program: merge3dlut.exe ~ More programs updated to use the transformation factory instead of hard logic Version 0.6: ! All .3dlut code is now multi-threaded and will run many times faster (do note that loading and saving a 100 MB file still takes around a second on modern hard drives) ! Transformations are now abstracted behind ITransformationProvider and can be assembled into TransformationFactories Most program logic has been rewritten to use this functionality, and a GUI version for .3dlut creation should be coming sometime soon ? dispclut can save and modify the CLUT at the same time now, eg. dispclut -d 1 --save old.txt --reset before playback and dispclut -d 1 --load old.txt afterwards + New program: imagecal.exe + Beta functionality for 3dlut color transformations using 3dLutLib added, note that does not yet properly interpolate the values. Once I've rewritten it to do so, I'll rewrite .ChangeRange to use this, thus eliminating its current banding as well - which will affect changedepth.exe (when using the -r flag) * make3dlut's built-in --calibrate option now works properly * All tools now work with limited-range floating point values as well (0.0625 as black and 0.918 as white, approximately) * 3dLutLib correctly detects full-range LUTs created with the RGB_PC format flag (instead of Output_Range 0 255) * gen3dlut no longer prints luma information to screen with --grayscale, was left over from debugging * The parser for .ti3/.cal files is now culture-invariant, to solve an issue where it would fail to parse the values in some european locales which used a comma instead of a period to specify decimals + dispclut now has an “--reset-all” flag * dispclut now uses the linear 0-65535 levels Version 0.5: ! Re-worked entire .3dlut code base, much cleaner now (at the cost of a tiny speed loss) * Detection for range (full or limited) is now handled by the 3dLUT loader itself and should be more consistent In addition, this information will be displayed by the inspect3dlut program + applycal can load .ti3 files as well, and... + ..make3dlut can automatically apply .cal data from a .ti3 if found + applycal now supports Calibration Tester's #GammaRamp format + 32-bit floating point LUTs are now supported in all applicable programs + New program: changedepth.exe Do note that while changedepth can pull values up/down to full or limited, this introduces a lot of banding + New program: dispclut.exe + New program: tag3dlut.exe Version 0.4: * Saving/loading to stdin or stdout now works properly * Applycal now (attempts to) correctly detect the input value range, and also works for 8-bit LUTs (not just 16-bit) * The TI3 parsing process now actually converts the entries to numbers, removing the need for ugly string detection hacks - should fix the duplicate key bug also + It's now possible to generate new .3dluts from scratch using the TI3ParserLib + New program: inspect3dlut.exe + New program: gen3dlut.exe ? Speed improvements here and there Version 0.3: + New program: applycal.exe + New program: swapchan.exe + New program: version.exe Version 0.2: ? Now requires the .NET version 4.0 to run properly ~ Updated UI, removed ugly buttons and added a menu + New program: genscript.exe + New program: make3dlut.exe + New program: parseti3.exe + New program: updateycms.exe * Temporary files are properly removed now * Updating yCMS now correctly downloads the 64-bit or 32-bit versions depending on your processor architecture Version 0.1: ? Initial release Legend: ! Major modification + New feature - Removed feature * Fixed bug ~ Sideways change ? Information
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 18th September 2011 at 12:45. |
|
20th August 2011, 13:57 | #4 | Link |
Replicant
Join Date: Jan 2007
Location: strasbourg
Posts: 49
|
ok,what values you use in ambient light level,Have you tried the 'black point correction'. I usually made a calibration (d6500-120cd/m-gama 2.35- black level native-calbration quality high-testchart 124)and i check with color hcfr.
|
20th August 2011, 15:49 | #5 | Link |
Kid for Today
Join Date: Aug 2004
Posts: 3,492
|
looks good! if you could allow merging LUT's like one for gamut mapping and one from Argyll off a .cal file, that'd be full of win
and if you could allow importing the primaries/secondaries saturations from Color HCFR, then it'd be maybe none of this is technically possible though, and I need to document myself on the .ti3 file format An option to output 8bit 3DLUT's could be useful for the happy few using rgb3dlut() within the Avisynth filter of ffdshow. |
20th August 2011, 15:51 | #6 | Link |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Quote:
Also the 8 bit 3DLUTs is also possible.
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management |
21st August 2011, 11:28 | #7 | Link | |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Quote:
I could possible morph the 3dlut by transforming it from 0-255 to 16-235, but that's significantly beyond the scope of this project and I don't intend to do any complex color management logic myself - just tie together existing tools (like yCMS). Merging noncompatible 3dluts would be something yCMS would have to do, and you should request for that feature there. Quote:
I usually use 6504K (I enter the X/Y points manually) - 6500K is a bit redder than D65 - and 120 cd/m², sRGB tone curve, native black level, native black point and massive LUT testchart.
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 21st August 2011 at 11:36. |
|
21st August 2011, 13:58 | #9 | Link | |
Kid for Today
Join Date: Aug 2004
Posts: 3,492
|
Quote:
Several ppl already requested the feature of embedding together within the same 3DLUT both the Argyll calibration and the gamut mapping, so mVR's dithering wouldn't be messed around by the graphic card's CLUT...this would require an option in mVR to disable the latter, though. Anyway this was just some plain brainstorming, and you're the author...making you the final judge in the end |
|
22nd August 2011, 02:27 | #10 | Link | ||
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Version 0.1.1 is out now that I have gained yesgrey's acceptance of the yCMS download method
Quote:
It's fairly straightforward to /create/ the correct LUTs using yCMS, it's just that madVR will reject the .3dlut. Quote:
My first priority now though is getting a working, automatized front-end for Argyll and yCMS in place to bridge the process from profiling to .3dlut creation.
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 22nd August 2011 at 02:31. |
||
22nd August 2011, 14:35 | #11 | Link |
Kid for Today
Join Date: Aug 2004
Posts: 3,492
|
Oops yeah, indeed I use YUY2 LUT's in ffdshow so I guess merging them w/ a .cal file would turn into headaches, if even possible.
And tbh I'm very happy w/ tritical's yv12toyuy2() because it allows you to choose what resizing algorithm to use for the YV12>YUY2 interpolation. I've finetuned all thoses settings to get the crispiest non-ringing PQ I've ever seen. I use SmoothL() to dither nicely and avoid banding, I'm not hung up on 10bit processing anymore...especially when dealing w/ 16-235 SMPTE-C content, because it might be a different story w/ xvYCC but we're not quite there yet. Last edited by leeperry; 22nd August 2011 at 14:42. |
22nd August 2011, 16:55 | #12 | Link | |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Quote:
Applying a .cal to a .3dlut seems like a simple process then, I'll write the code for it sooner or later. Generating a .3dlut from a .ti1 however would be much harder since it requires significant interpolation. Updated the UI slightly also: http://img805.imageshack.us/img805/7...parserdemo.png
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 22nd August 2011 at 17:33. |
|
22nd August 2011, 20:33 | #13 | Link |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Bumping with a semi-big update, version 0.2 is now out. See the initial posts for a changelog.
Download link: http://www.mediafire.com/?9by785si95tu73b (source is included)
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management |
23rd August 2011, 00:53 | #16 | Link |
Registered User
Join Date: Nov 2003
Posts: 1,281
|
Spent about an hour last night trying to get everything working again with madVR after not using it for awhile.
Then come across this one-click solution this morning Nice work and thanks. I see version 0.2.0.36408
__________________
http://www.7-zip.org/ Last edited by Audionut; 23rd August 2011 at 01:42. |
23rd August 2011, 03:03 | #18 | Link |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
HUGE SUCCESS
I've finally finished writing all of the code for loading, editing (mapping a function over each entry) and saving .3dlut files, as well as the code for loading and properly interpolating a .cal file. All of this takes place in little under a second of processing time (and they said C# was slow). As proof of concept, I've edited my existing .3dlut and swapped the red and blue channels just for fun. Here's the somewhat amusing result: http://img5.imageshack.us/img5/5215/chanswap.jpg Expect to see a fully working version that will apply .cal calibration data to a .3dlut sometime tomorrow, as well as a few utilities to swap channels for fun. For now, more fun stuff: chanswap.exe -i test.3dlut --invert produces: http://img192.imageshack.us/img192/2923/chaninvert.jpg While this was not intentional, do note that the packaging version (0.2) is not necessarily the version of the GUI program (in this case, 0.2.0.36408) - I'll update it in the future to include both. The packaging version is merely a version number for the overall collection and will get increased with every release (similarly to madVR's current 0.74 version number). The versions of the individual programs (eg. the GUI) will be following a more traditional major.minor.iteration.build version number based on features on that specific program.
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 23rd August 2011 at 03:43. |
23rd August 2011, 03:49 | #19 | Link |
Registered User
Join Date: Nov 2003
Posts: 1,281
|
Excellent. Can't wait to try it out.
__________________
http://www.7-zip.org/ |
23rd August 2011, 05:35 | #20 | Link |
( ≖‿≖)
Join Date: Jul 2011
Location: BW, Germany
Posts: 380
|
Another small update, 0.3 is out.
With this release, it's now possible to directly alter .3dlut files by applying a .cal file or swapping the levels around (for fun). Unfortunately, there is currently no GUI front-end for this functionality - I am waiting until I've come up with a better GUI design before I add this. However, it's easily done via the command line: Code:
applycal.exe -i source.3dlut -o destination.3dlut calibrationfile.cal
__________________
Forget about my old .3dlut stuff, just use mpv if you want accurate color management Last edited by nand chan; 23rd August 2011 at 05:44. |
Tags |
3dlut, argyllcms, color management, icc, madvr, ti3, ycms |
Thread Tools | Search this Thread |
Display Modes | |
|
|