Psy RDO: Official testing thread (version 0.6 out!)

[Dark Shikari]

Originally Posted by IgorC

Too much ringing and artifacts for my eyes

Well the non-psy-RDO version has far too much blurring for my eyes. The facial texture is completely flattened and there's blocking all over the place.

The slight ringing increase is because Psy-RD moved more bits to the background of the frame, forcing the quantizer up a little bit. If you don't like the bit distribution differential between the characters and the background, you could try lowering AQ strength, since both AQ and psy-RD are, in this case, moving around bits.

[Sagittaire]

Well I make test on Casino Royal (really grainy movie) and the old FGO produce really better result for the grain retention (--fgo 5 --trelli 2 vs --trelli 2). It's particuraly true for the very fine grain. If you want sample it's possible.

[Dark Shikari]

Originally Posted by Sagittaire

Well I make test on Casino Royal (really grainy movie) and the old FGO produce really better result for the grain retention (--fgo 5 --trelli 2 vs --trelli 2). It's particuraly true for the very fine grain. If you want sample it's possible.

Sure, toss a sample. There are two potential reasons why FGO could beat the current psy RD:

1. FGO uses an overlapped transform (2x2 SATD) while psy RD uses non-overlapped.

2. FGO uses a much smaller transform.

The former gets very slow for large transforms, so isn't very practical for this situation IMO. The latter issue is likely the same reason that FGO does worse than nothing on many sources; its aimed towards a very specific kind of "detail" rather than complexity in general.

Anyways, upload the source, I'm interested.

Edit: Oh, another possible issue is that FGO lowers B-frame quantizers and adjusts RD thresholds, which this doesn't (yet). Thus comparisons could potentially get misleading when trying to compare just the RD metric alone.

[ToS_Maverick]

how does Psy RDO work together with AQ, and especially with NO AQ?
how much is the grain retention depenend on the AQ with this patch?

i will definitely test this in the next days, when i got some time!

[Dark Shikari]

Originally Posted by ToS_Maverick

how does Psy RDO work together with AQ, and especially with NO AQ?
how much is the grain retention depenend on the AQ with this patch?

i will definitely test this in the next days, when i got some time!

I haven't tested much with regard to AQ; I've just been leaving it on default.

I assume its still probably a good idea to do so

[Sagittaire]

Originally Posted by Dark Shikari

Anyways, upload the source, I'm interested.

Edit: Oh, another possible issue is that FGO lowers B-frame quantizers and adjusts RD thresholds, which this doesn't (yet). Thus comparisons could potentially get misleading when trying to compare just the RD metric alone.

Source is uncompressed FFV1 file ...

Here the sample:
- New psy mode
http://jfl1974.free.fr/upload/Sample_psy_01.mkv

- Old FGO mode
http://jfl1974.free.fr/upload/Sample_FGO_01.mkv


Yes it's perhaps bframe problem because in low motion scene grain retention is really good for FGO and grain retention is really better for PSY RDO in high motion scene. Anyway for HVS I think that grain retention for low motion part is really more important than grain retention in high motion part.

Here my profil for FGO encoding ...

@REM -----------------------------------------------------------
@REM
@REM Profil BluRay 1080p23.976 extra high quality
@REM
@REM -----------------------------------------------------------


@REM Source file name (suffit de mettre la source ici)
set E_SRC=Lossless.avs

@REM Set of quality (ici la qualité 1-50)
set E_BR=22

@REM Set of max bitrate (ici le bitrate max)
set MAX_BR=20000

@REM Set of Buffer (ici le buffer)
set BUF_BR=30000

@REM Set credit (frame de début du générique)
set CRE_FR=201560

@REM Set end credit (frame de fin du générique)
set END_FR=207442



@REM Profil

x264.exe --threads auto --thread-input --keyint 24 --min-keyint 1 --crf %E_BR% --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --mvrange 511 --level 4.1 
--bframe 3 --b-pyramid --b-rdo --bime --weightb --ref 3 --mixed-refs --direct auto --deblock -2:-2 --ipratio 1.10 --pbratio 1.10 --partitions "all" --8x8dct 
--me "umh" --subme 7 --trellis 2 --no-fast-pskip --no-dct-decimate --aud --nal-hrd --sar 1:1 --cqmfile Sagittaire.cfg --aq-strength 1.00 --aq-mode 2 --fgo 5 
--zone %CRE_FR%,%END_FR%,b=0.33 --progress -o 1080p_Q1.264 %E_SRC%

pause

[sysKin]

+#define DC_COEFS_MB(block,src,stride)\
+ dc_coefs[block][0] = h->pixf.sad[PIXEL_16x16]( zero, 0, src[0], stride ) >> 1;\
+ dc_coefs[block][1] = h->pixf.sad[PIXEL_8x8] ( zero, 0, src[1], stride ) >> 1;\
+ dc_coefs[block][2] = h->pixf.sad[PIXEL_8x8] ( zero, 0, src[2], stride ) >> 1;

Shouldn't that be stride/2 for chroma planes?

[Dark Shikari]

Originally Posted by sysKin

Shouldn't that be stride/2 for chroma planes?

No, because x264 uses constant stride for macroblock analysis.

[Dark Shikari]

Originally Posted by Sagittaire

Source is uncompressed FFV1 file ...

Here the sample:
- New psy mode
http://jfl1974.free.fr/upload/Sample_psy_01.mkv

- Old FGO mode
http://jfl1974.free.fr/upload/Sample_FGO_01.mkv

I can hardly tell the difference between then, but psy looks slightly better to me in the flattest areas (textures on the characters, etc).

Your test is a bit screwed up because you didn't use the same video clip for both though, so I suspect ratecontrol allocated bits somewhat differently

I think its probably better to test psy RDO at low bitrates, because at high the differences are so slight that nobody is going to notice it anyways.

If there is a clear case that old FGO beats psy RDO, we want it to be very blatantly obvious so I can try to find the source of the issue.

[MfA]

Originally Posted by IgorC

Too much ringing and artifacts for my eyes

I don't particularly care for the overall effect either.

In theory the skin effect is nice ... but the problem with that is that apart from adding some texture it adds a spurious line to his neck. Structured artifacts are a complete no go IMO.

[Dark Shikari]

Originally Posted by MfA

In theory the skin effect is nice ... but the problem with that is that apart from adding some texture it adds a spurious line to his neck. Structured artifacts are a complete no go IMO.

Line? If you're looking at what I'm looking at, that's an artifact of AQ, not psy-RD, and exists on both...

[Yoshiyuki Blade]

The quality gain on Neo's black shirt stands out the most in the comparison. I'm really eager to see the results in anime (gonna try various bitrates), but since I'm in vietnam on vacation, I just have a lappy to test this out on ;_;. What can I expect on animated material where there's rarely any dark details on dark objects, or noise?

I knew I'd find myself wanting to tinker with encodes even while overseas lol.

[Adub]

Truthfully, I rather prefer the second picture, honestly. The jaggies introduced along all of the edges are making my eyes bleed. If this can be fixed by lowering aq, then great! But frankly, I will take something over the jaggies. Sorry DS.

[asdfsauce]

Dear God, that rapidshare capcha is the worst! Why do people use that site? It's horrible.. (no offense)

Sorry, so um, the Psy RDO screen for those matrix clips is the top one? To me that looks the best. There may be more artifacts but it doesn't have that artificial airbrushed look that I dislike most about h.264. It's all about seeing the forest through the trees.

[MfA]

Originally Posted by Dark Shikari

Line? If you're looking at what I'm looking at, that's an artifact of AQ, not psy-RD, and exists on both...

I'm talking about the line in the small light area at the bottom of his neck, I can't see it in the other picture ... he also seems to have grown a scar on his forehead.


PS. I have got to be honest though ... I have to either zoom in or reduce my resolution to see it clearly

[Dark Shikari]

It appears we're now getting down into the true danger of psy optimizations... not everyone agrees on the concept of "better"

[MfA]

How about a settable cut off for the frequencies considered in either dimension? Excluding lower frequencies would probably help prevent structured artifacts.

[Dark Shikari]

Originally Posted by MfA

How about a settable cut off for the frequencies considered in either dimension? Excluding lower frequencies would probably help prevent structured artifacts.

Excluding lower frequencies would completely ruin the effect though; in many of my test encodes there was an enormous benefit from the lower frequencies, especially in "300" at lower bitrates, but really in anything where the bitrate isn't high enough to flawlessly keep grain. I also suspect the dither-keeping effect depends strongly on lower frequencies.

I'd say the low frequencies are actually the primary reason that this beats FGO in many cases.

Structured artifacts aren't a large problem if they're small and don't stay between frames. Plus, trellis should usually handle getting rid of them anyways; at least it does in my experience.

[Adub]

New picture for people to compare based on IgorC's provided files.

Psy RDO is on top, "regular" is on bottom.
http://i34.photobucket.com/albums/d1...ovsnordo-1.png

The more noticeable part is around his figure.

[Dark Shikari]

The main issue just seems to be that psy RDO moves more bits to the background in order to keep the grain, losing bits elsewhere.

In almost all the cases I've tested, this results in a dramatic improvement (even in the areas that lose bits), but this is a particularly high contract situation, so it ends up ringing a bit more.

Of course, I prefer ringing to blurring, but that's just me.