Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
12th December 2024, 06:17 | #9641 | Link | ||
Registered User
Join Date: Aug 2024
Posts: 321
|
Quote:
What it does is actually removing frames and using SEI to tell decoder to duplicate existing frames, which actually should be called de-duplication I think. Maybe you are not looking at the right place (i.e. you think it's duplicating frames while it actually deletes frames), but assuming you are not able to get frames to trigger the de-duplication during encoding: There's PSNR thresholding, can be configured via --dup-threshold parameter. If you use low enough threshold, eventually some frames will be de-duped, but of course this should only be used in experiments, low threshold will just destroy the video. The second case: the decoding side... You need a decoder that's able to recognize and utilize the picture timing SEI (which only tells decoder to double or triple the frame, no actual timestamp is stored) and I guess things will... "just work"... yeah, who knows, the most common decoder (avcodec) doesn't support it so I can't test. Quote:
But I agree, it (FGS) has potential... just it still has a long way to go. Last edited by Z2697; 12th December 2024 at 06:23. |
||
15th December 2024, 19:43 | #9642 | Link | |
Lunarian
Join Date: Dec 2024
Posts: 15
|
Hello Doom9! It's been nearly 20 years since I last posted here although I've been lurking on and off all these years. I recently got back into this old hobby and have greatly appreciated the information ITT. Since I'd been away since the early x264 days and it took several weeks to get up to speed with x265. If it wasn't for the information ITT I would have spent many more weeks aimlessly testing stuff I'm sure. I have a bunch of questions. I hope you don't mind if I ask them.
1) --frame-rc / https://bitbucket.org/multicoreware/...396de62dc618c1 Can someone please explain how to work with this recent commit? Can I just define this stuff in my qpfile? If so, what syntax should I use? There is no documentation about this that I can find and it isn't really explained on the git repo from what I've seen. I've been slowly trying to work my way through ratecontrol.cpp to understand it but I haven't had a chance to throw random stuff in a qpfile and check yet (I will soon when I'm back at home). Has anyone used this feature yet? I am very interested in this since I already manually create a qpfile anyway to set I-frames at particular frames for various reasons (seeking speed and ensuring frames with overlays get an I-frame when needed). I'm already creating qpfiles by hand for each source I work with. So the ability to manually set CRF/QP/Bitrate would be a much welcomed addition. Up until now I've been having to manually make zones and multiply the bitrate up/down which is less than ideal. 2) Misc. cutree stuff and mods I did a lot of testing with cutree on/off last year and visually I frankly couldn't see much difference but turning off cutree would sometimes greatly increase bitrates for little gain. The extra bits didn't seem worth what I was getting. I currently "reign in" cutree by tweaking --qcomp typically at something like 0.7 or 0.8 at most. Boulder posted a diff file ITT several pages ago and the repo linked in this post is supposed to allow tweaking cutree directly instead of in the round about way with qcomp. But I've also heard from many people over the last few years that I shouldn't use cutree at all because it's a Quote:
I would greatly appreciate hearing your opinions on cutree and if using the above modifications (along with the hours of testing involved) would be worth my time. I've been building x265 from source for the last year or so. But I only use a handful of modifications and mainly for the AQ modes that aren't in mainline. Which brings me to my next and last question for now; 3) Auto-aq Earlier this year I built a modded version of x265 from I believe version 3.6 or thereabouts. It included the auto-aq mode which I quickly found to be a huge benefit for the sources I work with. In the version I had it was invoked as so; Code:
--aq-auto 10 Code:
--auto-aq I do have more questions about how x265 works and some of the settings I've been using for the last 2 years or so. I've gotten really good results at acceptable bitrates so I'm happy with them. But improvements are always welcomed of course. My workstation is getting a bit long in the tooth these days and I'm due for an upgrade. So I'm not able to do as much testing as I'd like. My usual config is already lucky to obtain 1fps. Which means a typical project for me takes several days and sometimes weeks to finish. Although the whole manually creating qpfiles and the type of editing/filtering I do takes up far more time. As I'm usually stepping through things frame-by-frame and splicing together sometimes 4 different sources to restore the content I work with. This is getting verbose so I'll hold off on asking more questions even though I'm full of them. It's good to be back. I'm happy to see this community is still very active. I prefer using forums to these modern locked down social media networks or God forbid Discord. Also, thanks again to the kind admin that allowed me to post this today instead of a week from now. Last edited by LunaRabbit; 15th December 2024 at 20:00. |
|
15th December 2024, 21:10 | #9643 | Link |
Registered User
Join Date: Aug 2024
Posts: 321
|
Some (paranoid) people have been disabling mbtree since x264, no wonder they'll disable cutree as well.
The real problem is not even whether cutree is a bad mbtree mimic or not, they don't even use mbtree to begin with. |
15th December 2024, 21:36 | #9644 | Link | |
Lunarian
Join Date: Dec 2024
Posts: 15
|
Quote:
My limited understanding of cutree is that it isn't the same as mbtree at all. From what I understand cutree is more about looking ahead and attempting to either insert more I-frames or shift data in a GOP towards I-frames. With the hope that the following P/B frames will benefit and thus be smaller. I might be wrong but I've always understood it as being more like lookahead than what mbtree was. But I don't pretend to understand the inner workings of mbtree either. It's just frustrating that the only discussion I see about cutree is either "turn it off it'll destroy quality" or "leave it on but I don't really know what it's doing. It just saves bits". I want to have a better understanding of how it works. I do know for my uses/config it does save bitrate without affecting quality much that I can see. I suppose if I pixel peep I could spot very mild drop in quality here and there. But over the course of an entire episode or movie it seems to save a lot of bitrate and produce something of equal quality if nothing else is changed. But perhaps that has more to do with the fact that I used closed GOPs and usually VBV. I do understand GOPs in x265 fine. I think I understand VBV as well and I don't think it really hurts quality that much the way I use it. I use it to impose an upper limit on the buffer so my encodes play nicer with some set-top devices and over the network. As I want to stream my stuff over my LAN and sometimes over WAN when I'm away from home. I was going to ask about VBV another time. But I only bring it up because maybe that's why I'm not seeing any improvement with cutree off. The bitrate savings for me with cutree are very good. In a clip of say 100 frames with cutree on and closed_GOP set to 24 frames I'll produce a video stream that's about 1.5-2.0MB. The --no-cutree version would be 3.5-4MB with no gain in quality that I can see. Although I do "reel it in" with qcomp. But perhaps that's placebo. I'm mainly doing that in an attempt to prevent much variation between keyframes and non-keyframes. Some guidance concerning cutree, how it relates to qcomp and if these modifications to tweak cu-tree are worth exploring would be very helpful and appreciated. Since I was surprised to find so little discussion about it outside of; "turn it off" or "if you're leaving it on make sure to raise qcomp a bit". I feel like there are a lot of things like this in x265 that is just advice being repeated for the last 10+ years from people assuming it works just like early x264. Some time way back when everyone just decided to do things a certain way and never bothered to do tests as the underlying code and the codec itself changed. I do know that people claim cutree can hurt backgrounds and scenes with a lot of movement. But in my experience it doesn't. But again. Perhaps I'm not seeing these issues because 1) I use a low CRF for 720p-1080p content anyway and 2) I regularly boost bitrate through the use of zones. Do you use cutree? Last edited by LunaRabbit; 15th December 2024 at 21:44. |
|
16th December 2024, 07:11 | #9645 | Link |
Registered User
Join Date: Jun 2024
Location: South Africa
Posts: 227
|
As you point out, cutree saves a lot of bitrate, sometimes close to -50% in anime, and the same goes for mbtree. Those savings can go to smaller CRFs for example. So, I do not understand what people gain by disabling these tools. There is a small increase in quality on certain parts of the frame, but you've got to look for it, and I think one would better spend bits by lowering the CRF.
Recently, encoding a set of anime at reference quality, I settled on leaving cutree on and using a low CRF (14). I also set the rc-lookahead to 240 because a longer lookahead is said to be more beneficial for cutree. I think if one is encoding anime in particular, disabling mbtree or cutree is a massive waste of bits, and a 2-pass encode would likely suffer more for it. As for how it works, my limited understanding is that it increases the quantiser for parts of a frame that are less referenced temporally. Dark Shikari wrote a paper about mbtree back in the day but it was a bit technical. Last edited by GeoffreyA; 16th December 2024 at 07:23. |
16th December 2024, 07:32 | #9646 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
I recently requested jpsdr to include the cutree strength mod (in addition to those couple of other interesting AQ related tweaks) and it might happen if he just has the time for that. Then it would be easy to control cutree if it seems too strong, I also don't believe that it should be disabled completely. Raising qcomp by 0.1 makes cutree much less effective so that's probably why it has been mentioned that qcomp should be raised a bit.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
16th December 2024, 08:47 | #9647 | Link | |
Registered User
Join Date: Aug 2024
Posts: 321
|
Quote:
Note that I used 2-pass bitrate control in the testing because the "cutree-strength" strongly impacted the CRF rate control (iirc). Last edited by Z2697; 16th December 2024 at 08:54. |
|
16th December 2024, 08:51 | #9648 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
Do you mean increasing qcomp and increasing CRF as well to match bitrate?
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
16th December 2024, 09:37 | #9649 | Link | |
Registered User
Join Date: Aug 2024
Posts: 321
|
Quote:
Because without mod, the "cutree-strength" is internally calculated from qcomp, changing the "cutree-strength" without touching qcomp seems to just confuses the CRF rate control so much that iirc, makes them off-chart. Edit: I just did a quick encode with that mod, obviously I didn't remember correctly... not off-chart. Last edited by Z2697; 16th December 2024 at 09:49. |
|
16th December 2024, 10:19 | #9650 | Link | ||
Lunarian
Join Date: Dec 2024
Posts: 15
|
Quote:
I was getting very good results with --aq-auto 10 for SDR content over the last several months. But obviously now my copy/pasted general settings no longer work. I did some testing with --auto-aq but I'm not sure how I'm supposed to be using it. Is it just the same thing with a different flag? Quote:
Thanks for the explanation everyone. Edit: My hope was I could leave qcomp near the default of 0.6 and tweak cutree directly with the goal of maybe shaving off some file size. My CRF is already set so low that I don't want to reduce it any further. Since I try to keep half hour content under 1GB for HD content if possible (lower is obviously better and it varies from episode to episode). Last edited by LunaRabbit; 16th December 2024 at 10:25. |
||
16th December 2024, 10:21 | #9651 | Link | |
Registered User
Join Date: Nov 2004
Posts: 241
|
Quote:
I'm not a developer but I can partly understand and navigate the source code. So I headed to: https://bitbucket.org/multicoreware/...it/src/master/ I found all portions of code related to --aom-film-grain and found that the encoder actually expects .bin files as input for grain parameters. I also found specifications as to what parameters that SEI payload should include. So I grabbed all of that code and fed it to o1-preview and Claude Sonnet 3.5 and asked them to provide me with compatible .bin files. Both of them, based on those source code snippets, were able to give me a python script to produce a (supposedly) compatible grain.bin file and that's what I did, eventually. I got those .bin files. However, no matter what I did for about 3 hours, I still couldn't get this feature to work. x265 always throws error that it cannot read characteristics of the grain file. I'm baffled. I tried to register on bitbucket, but it requires a corporate account to do so, so mere mortals cannot register so they're effectively blocked out from asking questions to developers on that platform. And nobody provided any examples of those .bin files and in what way they can actually be acquired. The 4.1 release notes are vague, the new commandline parameter is vague. There are no examples anywhere and there's no one that I can ask. I have no idea how to solve this and make it work. Apparently, a simple function of a modern video encoder requires brains of Einstein to even get started with it. Very frustrating to be honest. If anyone can provide any insights, that would be awesome. Last edited by Leo 69; 16th December 2024 at 10:43. |
|
16th December 2024, 11:51 | #9652 | Link | |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
Quote:
It's plain silly that they didn't choose the simple grain table file method which is already used by at least aomenc and SVT-AV1.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
|
16th December 2024, 11:57 | #9653 | Link | |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
Quote:
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
|
16th December 2024, 12:17 | #9655 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
I think someone mentioned that they committed a wrong set of patches under SBRC at first
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
16th December 2024, 13:17 | #9656 | Link | |
Lunarian
Join Date: Dec 2024
Posts: 15
|
Quote:
Do you happen to know anything about the --frame-rc stuff I asked about and if it can be controlled from a qpfile? I'm having the same issue as the person above. The release notes are very sparse about how these new features work. I'll see if I can find an archive of the mailing list in the mean time. |
|
16th December 2024, 13:24 | #9657 | Link | |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
Quote:
Unfortunately I don't know anything about the --frame-rc parameter. Nothing in the online CLI docs either, which is not unexpected as documentation seems to be irrelevant
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
|
16th December 2024, 13:47 | #9658 | Link | |
Lunarian
Join Date: Dec 2024
Posts: 15
|
Quote:
If anyone is curious I tried patching the .diff you posting a few pages back against v4.1 from master. As expected, so much changed it can't apply automatically anymore. But I plan to go through the source code later today to see if I can get it working. Here is the output from patch; Code:
patching file README.md patching file build/MSYS_jpsdr/Build_Win32.sh patching file build/MSYS_jpsdr/Build_Win32_AVX.sh patching file build/MSYS_jpsdr/Build_Win32_AVX2.sh patching file build/MSYS_jpsdr/Build_Win32_Broadwell.sh patching file build/MSYS_jpsdr/Build_Win32_x86.sh patching file build/MSYS_jpsdr/Build_mcf_10_Broadwell.sh patching file build/VS2019_LLVM/Command_Line.txt patching file build/VS2019_LLVM/Create_Solutions_x64.bat patching file build/VS2019_LLVM/Create_Solutions_x86.bat patching file doc/reST/cli.rst patching file source/CMakeLists.txt patching file source/abrEncApp.cpp Hunk #2 NOT MERGED at 1173-1178. patching file source/avisynth/avisynth.h patching file source/avisynth/avisynth_c.h patching file source/avisynth/avs/alignment.h patching file source/avisynth/avs/arch.h patching file source/avisynth/avs/capi.h patching file source/avisynth/avs/config.h patching file source/avisynth/avs/cpuid.h patching file source/avisynth/avs/filesystem.h patching file source/avisynth/avs/minmax.h patching file source/avisynth/avs/posix.h patching file source/avisynth/avs/types.h patching file source/avisynth/avs/version.h patching file source/avisynth/avs/win.h patching file source/cmake/Version.cmake patching file source/common/CMakeLists.txt patching file source/common/common.cpp patching file source/common/common.h patching file source/common/event.cpp patching file source/common/event.h patching file source/common/frame.cpp patching file source/common/frame.h patching file source/common/lowres.cpp patching file source/common/lowres.h patching file source/common/param.cpp Hunk #1 merged at 154-155. Hunk #18 NOT MERGED at 1378-1392. Hunk #27 NOT MERGED at 2449-2455. Hunk #28 NOT MERGED at 2478-2488. Hunk #29 NOT MERGED at 2530-2544,2546-2553. Hunk #30 merged at 2607. Hunk #31 NOT MERGED at 2638-2646. Hunk #35 merged at 2907-2908. Hunk #37 merged at 3003-3005. patching file source/common/param.h patching file source/common/threading.h patching file source/common/x86/asm-primitives.cpp patching file source/common/x86/loopfilter.asm Hunk #1 already applied at 3870. patching file source/encoder/api.cpp Hunk #1 merged at 120. patching file source/encoder/encoder.cpp Hunk #1 merged at 292, NOT MERGED at 298-302. Hunk #4 NOT MERGED at 2169-2184,2186-2197, merged at 2199-2202, merged at 2205, merged at 2210. Hunk #5 NOT MERGED at 2536-2542. Hunk #7 merged at 3748,3750-3751. Hunk #11 NOT MERGED at 4618-4625, merged at 4627. patching file source/encoder/frameencoder.cpp Hunk #1 merged at 467. patching file source/encoder/ratecontrol.cpp patching file source/encoder/ratecontrol.h patching file source/encoder/rdcost.h patching file source/encoder/search.cpp patching file source/encoder/slicetype.cpp patching file source/encoder/slicetype.h patching file source/input/avs.cpp patching file source/input/avs.h patching file source/input/input.cpp Hunk #2 NOT MERGED at 41-53. patching file source/input/input.h patching file source/input/vpy.cpp patching file source/input/vpy.h patching file source/input/y4m.cpp Hunk #1 NOT MERGED at 43-48. patching file source/input/y4m.h Hunk #1 NOT MERGED at 58-62. patching file source/input/yuv.h patching file source/output/gop.h patching file source/output/gop_engine.hpp patching file source/output/output.cpp patching file source/vapoursynth/VSConstants4.h patching file source/vapoursynth/VSHelper4.h patching file source/vapoursynth/VSScript4.h patching file source/vapoursynth/VapourSynth4.h patching file source/x265.h patching file source/x265cli.cpp Hunk #4 NOT MERGED at 106-122. Hunk #6 merged at 163-165. Hunk #7 NOT MERGED at 306-328, merged at 330, merged at 332-336. Hunk #9 merged at 565-572, NOT MERGED at 574-601, NOT MERGED at 604-611. Hunk #11 NOT MERGED at 978-989. Hunk #12 merged at 1112. Hunk #13 NOT MERGED at 1201-1207. patching file source/x265cli.h Hunk #5 merged at 450. Hunk #6 NOT MERGED at 505-511. patching file source/x265res.manifest.in |
|
16th December 2024, 13:57 | #9659 | Link | |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,812
|
Quote:
Fixing the patch file can be a tedious job but just takes time to sit down and work through the errors one by one. The original changes are from so long time ago that the codebase has changed quite a lot since then.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
|
16th December 2024, 13:58 | #9660 | Link |
Registered User
Join Date: Aug 2024
Posts: 321
|
Here're 2 test results of "cutree-strength" I just ran.
I leave them as links because they are long images. Please ignore the speed, I ran them in VM. https://files.catbox.moe/4qugbj.png https://files.catbox.moe/1tqlou.png The curves from "corresponding" qcomp and "cutree-strength" values are almost completely aligned with each other, maybe not easy to see in CRF results since the final bitrate differs quite a lot, so I did a 2-pass test. (qcomp 0.65 corresponds to cutree-strength 1.75 and qocmp 0.7 corresponds to cutree-strength 1.5) Also keep in mind this is not a valid quality comparison across different "qcomp and cutree-strength pairs", the matrics usually "performs" poorly when it comes to "bits re-distribution". (my layman's understanding is: aq is bits re-distribution within frame, cutree/mbtree is bits re-distribution across frames) Update: I ran a CRF test with more data points but only 2 sets of data. Make the interpolated curve more accurate and the "alignment" easier to see. And switched to a test build I did back in the day. The points are from CRF 12 to 32. Last edited by Z2697; 17th December 2024 at 11:12. |
Thread Tools | Search this Thread |
Display Modes | |
|
|