Well, I just tried the CCE default "Segment" setting and then transplanted the "FOX1" values from HCEnc. I would struggle to pick one result as being better than the other. If anything, the FOX1 matrix looks slightly worse, but this might as well just be a coincidence.
On the other hand, I find that the Quantizer Characteristics settings (Activity, Residual, Luma) are what make a bigger difference to the image (and, obviously bitrate most of all). I also re-ran the test with CCE SP3's Adaptive Quantization Matrix option disabled.
Note: images sharpened for the web to make differences more obvious.
Source
CCE SP3 Default Matrix. AAQM on.
Fox1 matrix. AAQM on.
CCE SP3 Default Matrix. AAQM off.
Fox1 matrix. AAQM off.
Just to reiterate my position: I have never seen changing the QM settings produce a "Wow, look at that!" result, so I choose to spend my time erasing defects in the master, pre-processing, and also correctly setting SP3's Activity/Residual/Luma settings. I would still like to understand why some people spend so much time with QM and I would like to see some cases where it has been worthwhile for them. Surely it can't all be theory?
Settings in all cases were 6mbps average (2/8 min/max), 9 passes, new VAF created each time, NTSC with 2:3 Pulldown. Picture settings were completely flat (no LPF) except for some slight roll-off for Horizontal chroma (my reasoning: this telecine transfer has fine coloured grain which can't be fully reproduced by stepping down 4:2:0, and filtering chroma results in less "coloured compression artefacts" in the difficult scenes). Quant characteristics were 20/40/70 (Activity/Residual/Luma). Motion estimation Normal, Picture Type Progressive.
Your thoughts? I'd just love to see a practical example, that's all.