Psy RDO: Official testing thread (version 0.6 out!)

[mahsah]

How do you think this would work on anime? Could it be used with gradfun2db to hide DCT blocks?

[Dark Shikari]

Originally Posted by mahsah

How do you think this would work on anime? Could it be used with gradfun2db to hide DCT blocks?

Yes, in my experience so far it is incredibly good at keeping the dither that gradbun2db introduces as long as you give it enough bitrate (while without, regular x264 will generally butcher banding even at absurdly high bitrates).

[desta]

I've tried it on quite a bit of animated material now, and have got to say I'm very impressed.

Would I be right in assuming that although psy rdo is activated by default at >subme 6, it's optimised to process an image based on how grainy/noisy it is, including fine texture/detail, etc (in other words good on clean, grainy, or mixed footage)... whereas fgo by nature assumes a source is grainy/noisy, and could therefore be detrimental to a source that has both clean and grainy scenes?

[Dark Shikari]

Originally Posted by desta

I've tried it on quite a bit of animated material now, and have got to say I'm very impressed.

Would I be right in assuming that although psy rdo is activated by default at >subme 6, it's optimised to process an image based on how grainy/noisy it is, including fine texture/detail, etc (in other words good on clean, grainy, or mixed footage)... whereas fgo by nature assumes a source is grainy/noisy, and could therefore be detrimental to a source that has both clean and grainy scenes?

Both psy RDO and FGO attempt to make the output have the same <metric> as the input. But you are sort of correct, the metric used by psy RDO will generally be more friendly towards all types of complexity, not merely grain.

[desta]

Brilliant, thanks. I know you said the object of psy rdo was to eventually replace fgo. Is that likely soon? Would you consider it to be more efficient than fgo already, or do you think they both still have their place?

[Dark Shikari]

Originally Posted by desta

Brilliant, thanks. I know you said the object of psy rdo was to eventually replace fgo. Is that likely soon? Would you consider it to be more efficient than fgo already, or do you think they both still have their place?

I think its more efficient, but I haven't done enough testing.

One thing I may see is if I can combine the two; Psy RD uses a 4x4 and 8x8 transform for comparison, while FGO uses a 2x2. Maybe I could combine all three transforms?

[gav1577]

Originally Posted by Dark Shikari

I think its more efficient, but I haven't done enough testing.

One thing I may see is if I can combine the two; Psy RD uses a 4x4 and 8x8 transform for comparison, while FGO uses a 2x2. Maybe I could combine all three transforms?

Wow that would be neat would love to test that one out

[Yoshiyuki Blade]

With these visual improvements (VAQ and Psy RDO), I wonder if CQMs still have a large impact on image quality. I always have Sharktooth's AVC CQM on, even with animated material, but I haven't tried without it yet. Encoding a full length animated episode is incredibly slow on my lappy. You think CQMs will still remain significant at this stage (given a reasonably high bitrate/resolution)?

[Dark Shikari]

Originally Posted by Yoshiyuki Blade

With these visual improvements (VAQ and Psy RDO), I wonder if CQMs still have a large impact on image quality. I always have Sharktooth's AVC CQM on, even with animated material, but I haven't tried without it yet. Encoding a full length animated episode is incredibly slow on my lappy. You think CQMs will still remain significant at this stage (given a reasonably high bitrate/resolution)?

I've never really personally liked CQMs, for many reasons.

If you want to make comparisons, short samples should probably be sufficient.

[shae]

Originally Posted by Dark Shikari

I've never really personally liked CQMs, for many reasons.

Like what?

[Dark Shikari]

Originally Posted by shae

Like what?

1) When you're actually trying to keep fine detail, they butcher it.
2) They skew x264's lambda values.
3) They confuse RDO.
4) They confuse trellis even more than they confuse RDO.
5) Prestige, especially, does some really bizarre things.

[ToS_Maverick]

sry for hijacking this thread, but why is x264 so not-suited for CQMs? do other encoders work better with them?

[Dark Shikari]

Originally Posted by ToS_Maverick

sry for hijacking this thread, but why is x264 so not-suited for CQMs? do other encoders work better with them?

It isn't that x264 isn't suited for use with CQMs; nothing of the sort.

First, on the topic of RDO, you can intentionally make CQMs so that RD-wise, the inter matrix will always be favorable (for example), and so inter blocks are nearly always used when in RD mode.

However, the creator of the CQM might not have intended this; for example, the CQM might have been intended for an encoder that didn't use RDO, and therefore would not have taken into account the CQM when making its decision. I strongly suspect Prestige is this sort of CQM.

With trellis, trellis simply does not take into account CQM weights when working. I've found doing so actually reduces visual quality, unless I made a mistake in my patch--go figure.

[techouse]

Originally Posted by Razorholt

Is it REALLY worth it? Maybe Wizboy or Bobor can try and compile an x264 version with the fixed me-prepass patch for us to test?

Thanks,
- Dan

Just Check out my site http://x264.tk

x264_x86_r871_techouse

Source: x264 r871 GIT (git://git.videolan.org/x264.git)

Applied patches (current versions):

x264_hrd_pulldown.04_interlace.diff

x264_me-prepass_DeathTheSheep_techouse_fix.diff

x264_progress.diff

x264_psy_rdo_0.22.diff


Please check http://forum.doom9.org/showthread.php?t=130364 and http://git.videolan.org/gitweb.cgi?p...git;a=shortlog for more info

Compiled by techouse on June 6th 2008, 03:13:38 CEST with GCC-4.3.0 on Windows Vista Ultimate SP-1 32-bit.

Commandline used: ./configure&&make fprofiled

Platform: X86
System: MINGW
avis input: yes
mp4 output: yes
pthread: yes
gtk: no
debug: no
gprof: no
PIC: no
shared: no
visualize: no

[desta]

Originally Posted by techouse

Just Check out my site http://x264.tk

x264_x86_r871_techouse

http://forum.doom9.org/showthread.ph...86#post1146886

[shae]

Originally Posted by Dark Shikari

1) When you're actually trying to keep fine detail, they butcher it.

Well, doesn't that depend on the QM and is exactly what CQM are for, to allow you to fine tune what's kept and what's not?

2) They skew x264's lambda values.
3) They confuse RDO.
4) They confuse trellis even more than they confuse RDO.

Wouldn't that be implementation specific? It may conflict with other features currently, but in an ideal AVC encoder it should just add flexibility.

5) Prestige, especially, does some really bizarre things.

Just extreme values leading to extreme results, no?

[LoRd_MuldeR]

Originally Posted by shae

Wouldn't that be implementation specific? It may conflict with other features currently, but in an ideal AVC encoder it should just add flexibility.

I'd say an ideal AVC encoder wouldn't need any special CQM to workaround it's visual problems

Also I think features like VAQ, FGO and Psy RDO do exactly what people tried to achieve with CQM's before, only in a more sophisticated and more general way.
IMO it's always better to resolve the cause of the problem itself instead of fighting the symptom of the problem...

[foxyshadis]

An ideal AVC encoder would vary the quant matrix per frame to completely maximize RD. That's one of those exponentially hard problems though, unless someone comes up with a good way to quickly test whether a CQM benefit outweighs its size every frame.

[Blue_MiSfit]

I had an idea for something similar back in the Xvid days, but it would have been very time consuming, and required a lot of user interaction. Here's how I saw it:

1) Take a movie, and split it into comp-test style chunks
2) Encode this multiple times with different CQMs
3) Show the compressionist the individual chunks, with the ability to flip between versions easily, stackhorizontal, interleave, whatever.
4) Allow the compressionist to pick a "top 3" or something for each chunk
5) Tally the votes, and pick a CQM for the movie.

I wish I could code, or I would have cobbled this together just out of curiosity.

But back on topic - I'm a huge fan of PsyRDO. It's improved almost every single thing I've thrown at it.

~MiSfit

[Razorholt]

Originally Posted by Blue_MiSfit

I'm a huge fan of PsyRDO. It's improved almost every single thing I've thrown at it.

Are you using Trellis 2 ?