Psy RDO: Official testing thread (version 0.6 out!)

[Ranguvar]

Originally Posted by Razorholt

Are you using Trellis 2 ?

Why? Does it help, or hurt PsyRDO?

[Razorholt]

Originally Posted by Dark Shikari

Trellis 1 probably is not the best idea with this because that means the quantization that x264 chooses is not the quantization that psy RDO looked at in mode decision. Therefore, unless its demonstrated that this isn't a real issue, deadzones or trellis 2 are probably a better idea.

Originally Posted by Dark Shikari

Trellis 2 is not a bad thing at all with psy RD; the problem before was that it encouraged blurring because RD would often prefer blurrier modes, and trellis would generate blurrier modes. But now, RD prefers *less* blurry modes.

Trellis 2 sharpen the picture. What settings you had? Trellis 1 or were you using deadzones?

[gav1577]

I find low deadzones usually inter 6 intra 6 work great for me

[Blue_MiSfit]

Yes I am using Trellis 2, and SubME 7, when I can afford the speed hit.

I did Princess Mononoke the other day, a nice high quality soft-telecined DVD9 source.

I did a very light script with light fft3dgpu(sigma=1), Toon() and gradfun2db(1.8). Then I encoded into Psy-RDO'd CRF19 with Multi-Hex SubME 7, Trellis 2, 16 refs, all the usual bells and whistles. It came out to 1.2 GB. That's really impressive to me

Muxing the original english and japanese 5.1ch AC3s, plus an english SRT netted a file size of just under 2gb. 9gb -> 2gb with utterly transparent quality (I actually preferred my version due to the denoise / warpsharp / line darkening / dithering). Now THAT's impressive.

Heck, the encoding only took a few hours. I think I was getting ~15fps on my Q6600 @ 3 GHz

~MiSfit

[TheRyuu]

Originally Posted by Ranguvar

Why? Does it help, or hurt PsyRDO?

Trellis 2 generally provides the best result of the different possible options you can use. I consider using trellis 2 to just be 'smarter' then using deadzones with it which does make sense.

I think people need to break out of the mentality that trellis is bad for keeping grain when isn't anymore with psy RDO. (no idea of anyone has this mentality to begin with... just making comments)

[LoRd_MuldeR]

Question: I see a lot of people talking about "using deadzones"

Aren't deazones always there? If you don't set them explicitly, they will simply default to "deadzone=21,11", right?

[Dark Shikari]

Originally Posted by LoRd_MuldeR

Question: I see a lot of people talking about "using deadzones"

Aren't deazones always there? If you don't set them explicitly, they will simply default to "deadzone=21,11", right?

"Using deadzones" means not using trellis.

[LoRd_MuldeR]

Originally Posted by Dark Shikari

"Using deadzones" means not using trellis.

I see. thx.

[MasterNobody]

Probably deadzones are used even with trellis=2 because trellis is used only for luma plane compression, and not used for chroma planes.

/* no trellis; it doesn't seem to help chroma noticeably */

[Dark Shikari]

Originally Posted by MasterNobody

Probably deadzones are used even with trellis=2 because trellis is used only for luma plane compression, and not used for chroma planes.

This is correct.

Also note that deadzones are used for RDO when trellis=1, and even in trellis=2, deadzones are used for B-frames unless b-rdo is enabled. Finally, even if RDO is on all frametypes, deadzones are still used for fast P-skip decision.

[gav1577]

Are there any advantages in using trelis 2 over low deadzones Qualitywise with bitrates up to and above 4000kbps ?

[TheRyuu]

Originally Posted by gav1577

Are there any advantages in using trelis 2 over low deadzones Qualitywise with bitrates up to and above 4000kbps ?

In my opinion, using trellis 2 will always provide a better looking result then using deadzones simply because it is smarter then deadzones. More so at lower bitrates, but I've seen it seems to consistantly look better any a number of bitrates I have tried.

Although you should always do the testing and decide for yourself as I am no expert on the matter.

[bokonon]

Originally Posted by Dark Shikari

This is correct.

Also note that deadzones are used for RDO when trellis=1, and even in trellis=2, deadzones are used for B-frames unless b-rdo is enabled. Finally, even if RDO is on all frametypes, deadzones are still used for fast P-skip decision.

Would a combination of --trellis 2 and low deadzones (6, 4) work favorably? OR would it work better with default deadzones (21, 11)

[Irakli]

Originally Posted by bokonon

Would a combination of --trellis 2 and low deadzones (6, 4) work favorably? OR would it work better with default deadzones (21, 11)

No, modify deadzones if and only if trellis is off. So, either use trellis 2 with default deadzones or modify deadzones if trellis is off.

[gav1577]

Originally Posted by Dragon5152

In my opinion, using trellis 2 will always provide a better looking result then using deadzones simply because it is smarter then deadzones. More so at lower bitrates, but I've seen it seems to consistantly look better any a number of bitrates I have tried.

Although you should always do the testing and decide for yourself as I am no expert on the matter.

Thanks for the reply will try some clips trellis 2 and see if there is any increase in quality the only thing about trellis 2 is i find it slows things down quite a bit

[Dark Shikari]

Here's an encode of part of the original clip that inspired psy RDO in the first place. The flashing lights, the flat but gradient-filled background areas--all of them contrive to make encoding a nightmare for x264.

[yesgrey]

I want to try this patch. I am thinking in using Megui with the DXVA-HD-HQ profile with trellis 2 and subme 7. The bitrate used will be 16000 or 22000 kbps. Will it be worth it? or due to the high bitrate the effect will be negligible... even using trellis 2 and subme 7 the encoded file will be DXVA compatible?
For using this is enough substituting the x264.exe by the patched one in megui directory, right?

[Blue_MiSfit]

Originally Posted by yesgrey3

I want to try this patch. I am thinking in using Megui with the DXVA-HD-HQ profile with trellis 2 and subme 7. The bitrate used will be 16000 or 22000 kbps. Will it be worth it? or due to the high bitrate the effect will be negligible... even using trellis 2 and subme 7 the encoded file will be DXVA compatible?
For using this is enough substituting the x264.exe by the patched one in megui directory, right?

Yep All you have to do is enable Sub ME >=6, and use MeGUI's preferences to select the patched .exe.

At that bitrate, subme7 will likely be totally unnecessary (guessing). It boosts the effectiveness of Psy-RDO, but is a lot slower as well.

You might try plain old FGO too, instead of Psy-RDO.

Just out of curiousity, why target a specific bitrate? CRF mode is wonderful

~MiSfit

[yesgrey]

Originally Posted by Blue_MiSfit

Just out of curiousity, why target a specific bitrate? CRF mode is wonderful

Because I want a specific target file size to put in dvds.

[bkman]

Hi Dark Shikari,

A couple of observations from testing the psyRDO.

1.) It works noticeably better if employed in the first pass also.
2.) The picture seems to degrade much quicker between I-frames, making for a much more noticeable transition on a mid-scene keyframe.

Any ideas?