Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > VP9 and AV1

Reply
 
Thread Tools Search this Thread Display Modes
Old 16th April 2019, 12:45   #1141  |  Link
Beelzebubu
Registered User
 
Join Date: Feb 2003
Location: New York, NY (USA)
Posts: 109
Quote:
Originally Posted by user1085 View Post
What is the proper way to multithread in 2019? Is it possible to achieve 100% cpu usage on 8 cores?
Using aomenc, --tile-columns=3 --threads=8 should give you 8 tile col threads. Also try --row-mt and non-zero values for --tile-rows.

Quote:
Originally Posted by user1085 View Post
Also, I read somewhere that frame parallel is not needed anymore. Is that correct?
See last paragraph in https://forum.doom9.org/showthread.p...91#post1859991
Beelzebubu is offline   Reply With Quote
Old 16th April 2019, 20:28   #1142  |  Link
singhkays
Registered User
 
Join Date: Aug 2018
Posts: 18
Quote:
Originally Posted by poisondeathray View Post
Still buggy . More frame drops now, gaps in motion, and the framecount does not match when setting timebase and pts

You can index it with avisynth/vapoursynth, but frames still don't match

I suspect he had issues with the immediate source
Quote:
Originally Posted by Beelzebubu View Post
Sometimes the timebase is different. You can ignore timestamps by using "settb=1/30,setpts=N" as extra lavfilters in your commandline for each of the two video streams.
I'm seeing another issue with my VP9 encode. I'm getting the following message after 1st pass -"output file is empty". Is this expected for first pass?

Code:
[libvpx-vp9 @ 0x5f5d480] v1.8.0-366-gc46694c1d
Output #0, mp4, to '/dev/null':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.27.101
    Stream #0:0(eng): Video: vp9 (libvpx-vp9) (vp09 / 0x39307076), yuv420p, 720x300 [SAR 1:1 DAR 12:5], q=-1--1, 500 kb/s, 24 fps, 12288 tbn, 24 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.49.100 libvpx-vp9
    Side data:
      cpb: bitrate max/min/avg: 500000/500000/500000 buffer size: 1000000 vbv_delay: -1
frame=  158 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.00 bitrate=N/A speed=   0x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded
Here's the CLI for trying to get a 500K file
Code:
./ffmpeg -i scene6.mp4 -c:v libvpx-vp9 -pix_fmt yuv420p -auto-alt-ref 1 -lag-in-frames 25 -b:v 500k -filter:v scale=720:-1 -row-mt 1 -quality best -cpu-used 1 -tile-rows 2 -tile-columns 2 -threads 16  -enable-tpl 1 -pass 1 -minrate 500k -maxrate 500k -bufsize 1000k -g 240 -f mp4 -hide_banner -y /dev/null && \
./ffmpeg -i scene6.mp4 -c:v libvpx-vp9 -pix_fmt yuv420p -auto-alt-ref 1 -lag-in-frames 25 -b:v 500k -filter:v scale=720:-1 -row-mt 1 -quality best -cpu-used 1 -tile-rows 2 -tile-columns 2 -threads 16 -enable-tpl 1 -pass 2 -minrate 500k -maxrate 500k -bufsize 1000k -g 240 -hide_banner -movflags faststart ./vp9/scene6/vp9_500k.mp4
singhkays is offline   Reply With Quote
Old 17th April 2019, 06:54   #1143  |  Link
Asilurr
Registered User
 
Join Date: Jan 2019
Posts: 9
These suggestions will probably help you:
1. With contemporary versions of libvpx there's no real benefit to using tile-rows>0. The combination of tile-rows=0/tile-columns>0/row-mt>0 is advisable.
2. Do not expect multithreading miracles when working with sources which have a paltry 720 pixels horizontal resolution. There are "thresholds" which allow specific values of tile-columns (the primary MT mechanism of libvpx), and then there's row-mt which basically "doubles" the thread count (it's not exactly a 2x count, it depends on the source). Practical example: with your 720 pixels horizontal resolution source, you input tile-columns=2 and threads=16 to the encoder. It accepts them without throwing errors back at you, but it will default to what it can actually use on your source: tile-columns=1 and threads~4 (assuming row-mt=1); the encoder simply can't "fit in" 16 threads working with a 720 width source.
Code:
horizontal resolution | superblock columns | horizontal tiles | tile-columns | rough thread count with row-mt=1
0001-0448 | 001-007 | 01 | 0 | ~02
0449-0960 | 008-015 | 02 | 1 | ~04
0961-1984 | 016-031 | 04 | 2 | ~08
1985-4032 | 032-063 | 08 | 3 | ~16
4033-8128 | 064-127 | 16 | 4 | ~32
3. Considering your current quality best, you don't really care about the encoding speed; however do keep in mind that quality good usually provides superior results (yes, really). As such, switching to quality good/cpu-used=0/auto-alt-ref=6 is advisable.
4. FFmpeg's "output file is empty" notification is entirely expected, you are outputting the first pass to /dev/null.
Asilurr is offline   Reply With Quote
Old 17th April 2019, 07:39   #1144  |  Link
singhkays
Registered User
 
Join Date: Aug 2018
Posts: 18
Quote:
Originally Posted by Asilurr View Post
These suggestions will probably help you:
1. With contemporary versions of libvpx there's no real benefit to using tile-rows>0. The combination of tile-rows=0/tile-columns>0/row-mt>0 is advisable.
2. Do not expect multithreading miracles when working with sources which have a paltry 720 pixels horizontal resolution. There are "thresholds" which allow specific values of tile-columns (the primary MT mechanism of libvpx), and then there's row-mt which basically "doubles" the thread count (it's not exactly a 2x count, it depends on the source). Practical example: with your 720 pixels horizontal resolution source, you input tile-columns=2 and threads=16 to the encoder. It accepts them without throwing errors back at you, but it will default to what it can actually use on your source: tile-columns=1 and threads~4 (assuming row-mt=1); the encoder simply can't "fit in" 16 threads working with a 720 width source.
Code:
horizontal resolution | superblock columns | horizontal tiles | tile-columns | rough thread count with row-mt=1
0001-0448 | 001-007 | 01 | 0 | ~02
0449-0960 | 008-015 | 02 | 1 | ~04
0961-1984 | 016-031 | 04 | 2 | ~08
1985-4032 | 032-063 | 08 | 3 | ~16
4033-8128 | 064-127 | 16 | 4 | ~32
3. Considering your current quality best, you don't really care about the encoding speed; however do keep in mind that quality good usually provides superior results (yes, really). As such, switching to quality good/cpu-used=0/auto-alt-ref=6 is advisable.
4. FFmpeg's "output file is empty" notification is entirely expected, you are outputting the first pass to /dev/null.
Thank you for the recommendations, will update my scripts!

Re: quality = good - wow! the more I delve into VP9, the more I'm surprised who came up with all these quirks! Is there more info on why quality = good is better?

Re: auto-alt-ref - Is there more info on what values 1 to 6 mean?

Re: output file - The reason I ask is that ffmpeg VP9 wiki states - "In pass 1, output to a null file descriptor, not an actual file. (This will generate a logfile that ffmpeg needs for the second pass.)" https://trac.ffmpeg.org/wiki/Encode/VP9

Few other questions
  • Do you have recommendations for aq-mode?
  • Any other recommendations on preventing the VP9 encoder from undershooting the target bitrate?

Last edited by singhkays; 17th April 2019 at 07:45.
singhkays is offline   Reply With Quote
Old 17th April 2019, 12:22   #1145  |  Link
Leeloo Minaļ
Registered User
 
Join Date: Nov 2007
Posts: 50
Quote:
Originally Posted by singhkays View Post
Re: output file - The reason I ask is that ffmpeg VP9 wiki states - "In pass 1, output to a null file descriptor, not an actual file. (This will generate a logfile that ffmpeg needs for the second pass.)" https://trac.ffmpeg.org/wiki/Encode/VP9
If you are under Windows, you may have not noticed the following note on https://trac.ffmpeg.org/wiki/Encode/VP9 :
Quote:
Note: Windows users should use NUL instead of /dev/null
Leeloo Minaļ is offline   Reply With Quote
Old 17th April 2019, 18:32   #1146  |  Link
singhkays
Registered User
 
Join Date: Aug 2018
Posts: 18
Quote:
Originally Posted by Leeloo Minaļ View Post
If you are under Windows, you may have not noticed the following note on https://trac.ffmpeg.org/wiki/Encode/VP9 :
I did but I'm on Linux
singhkays is offline   Reply With Quote
Old 18th July 2019, 08:20   #1147  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
New upload (MSYS2; MinGW32 / MinGW64: GCC 9.1.0):

VPx v1.8.1-34-g53dc2d9d9
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 21st August 2019, 06:58   #1148  |  Link
utack
Registered User
 
Join Date: Apr 2018
Posts: 63
Is there a way to get libvpx not to "crush" flat or dark parts of the image so much? It is one of the major weak points it still has that can ruin certain parts of a stream and that x264 solved ages ago.
Both tune-content=film and aq-mode=1/2 do not seem to suffice to control this behaviour.
Is there another method like limited the quantizer range or such?
Thank you
utack is offline   Reply With Quote
Old 21st August 2019, 08:31   #1149  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Apart from raising the bitrate? ... Reducing "noise" is one of the most important ways to encode efficiently; maybe you can change the amount with the VP9 specific parameter
Code:
            --noise-sensitivity=<arg>  	Noise sensitivity (frames to blur)
but its default is already 0, so this may make it only worse...

Or it might be possible to move bitrate distribution away from usually preferred frames (keyframes, "golden frames"), which may hurt the quality in other scenes.

Unfortunately, not all encoders offer control over the same kind of algorithms. They may not even contain the same set of algorithms. If VPx has no exposed parameters to control rate distribution in relation to specific content dynamics, "enough bitrate" (or a forced limit for the maximum quantization) may be your only hope.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid

Last edited by LigH; 21st August 2019 at 08:34.
LigH is offline   Reply With Quote
Old 22nd August 2019, 18:44   #1150  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by utack View Post
Is there a way to get libvpx not to "crush" flat or dark parts of the image so much? It is one of the major weak points it still has that can ruin certain parts of a stream and that x264 solved ages ago.
Both tune-content=film and aq-mode=1/2 do not seem to suffice to control this behaviour.
Is there another method like limited the quantizer range or such?
Thank you
This is where adaptive quantization algorithms shine, and where PNSR-tuned encoders can fall flat.

I've yet to see a VPx series encoder that had a good adaptive quant algorithm. I don't know that this is a limitation of the bitstream itself; it's more likely just a reflection of the psychovisual maturity of the encoders.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 23rd August 2019, 20:09   #1151  |  Link
mandarinka
Registered User
 
mandarinka's Avatar
 
Join Date: Jan 2007
Posts: 729
I recall there being an actual limitation in the bitstream. The AQ has to be done via segmentation maps where you pick quants you can use and then you can assign one of those to blocks through assigning them to one or another segment. Roughly speaking. I recall that it was believed to be a hinder to AQ being useful.
mandarinka is offline   Reply With Quote
Old 23rd August 2019, 22:50   #1152  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by mandarinka View Post
I recall there being an actual limitation in the bitstream. The AQ has to be done via segmentation maps where you pick quants you can use and then you can assign one of those to blocks through assigning them to one or another segment. Roughly speaking. I recall that it was believed to be a hinder to AQ being useful.
Oh, yeah. That would require weird loops where you'd encode the frame once, figure out the segments, and then reencode again. MPEG-4 part 2 had a similar issue in its adaptive quant which also was a huge hassle to tune.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 25th August 2019, 19:49   #1153  |  Link
IgorC
Registered User
 
Join Date: Apr 2004
Posts: 1,315
Quote:
Originally Posted by utack View Post
Is there a way to get libvpx not to "crush" flat or dark parts of the image so much? It is one of the major weak points it still has that can ruin certain parts of a stream and that x264 solved ages ago.
Both tune-content=film and aq-mode=1/2 do not seem to suffice to control this behaviour.
Is there another method like limited the quantizer range or such?
Thank you
If 10 bits is an option that would fix a major part of described issue.
IgorC is offline   Reply With Quote
Old 26th August 2019, 16:00   #1154  |  Link
Beelzebubu
Registered User
 
Join Date: Feb 2003
Location: New York, NY (USA)
Posts: 109
Quote:
Originally Posted by benwaggoner View Post
Oh, yeah. That would require weird loops where you'd encode the frame once, figure out the segments, and then reencode again.
Why? For example, x264 heuristically assigns quant deltas before the frame encode. To fit this in a segment-ID design, all you need is a clustering algorithm (see e.g. webp).
Beelzebubu is offline   Reply With Quote
Old 26th August 2019, 20:52   #1155  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,988
Super interesting stuff. I totally agree that these types of issues are what disqualify VP9 for my use cases. It's better than AVC at low bitrates for sure, but when we go for high quality / transparency as required for premium OTT delivery of Hollywood content it's a bit lacking in exactly this way.

It's useful for UHD, but there we usually can use HEVC which just ends up looking better.

If we had good AQ I could totally see VP9 being incredibly useful, especially in cases where HEVC is off the table!

Last edited by Blue_MiSfit; 26th August 2019 at 20:55.
Blue_MiSfit is offline   Reply With Quote
Old 27th August 2019, 20:57   #1156  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by Beelzebubu View Post
Why? For example, x264 heuristically assigns quant deltas before the frame encode. To fit this in a segment-ID design, all you need is a clustering algorithm (see e.g. webp).
Oh, it would be possible, just a lot more finicky to do so optimally. x264 devs figured out how to do adaptive quant in xvid eventually, although it was slower and less optimal than in x264.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 10th September 2019, 09:15   #1157  |  Link
Spyros
Registered User
 
Join Date: Jun 2019
Posts: 16
SVT-VP9 performance boost on AVX2

Phoronix - Intel's Open-Source VP9 Video Encoder Just Scored A Massive ~3x Performance Boost

Quote:
SVT-VP9 is now a lot faster on AVX2 CPUs from both Intel and AMD.

[...]

The i7-7740X went from 30 FPS to 120 FPS, the E5-1680 v3 from 38 to 113 FPS, and the E5-2687W v3 from 46 to 150 FPS. Damn!
Spyros is offline   Reply With Quote
Old 10th September 2019, 17:32   #1158  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Impressive, but not fundamentally surprising. Libvpx never received the sort of sustained high-touch optimization or tuning that the x24? series of encoders got. A really good encoder matters more than a bitstream with really good potential without encoders well tuned to use that potential.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 15th September 2019, 09:21   #1159  |  Link
dapperdan
Registered User
 
Join Date: Aug 2009
Posts: 201
That particular speed up isn't impressive, nor does it say much about libvpx.

Effectively they forgot to flick a switch to send the fast code to all platforms that could use it. Then they noticed and fixed it. The headline could just as easily been "vp9-svt pointlessly 3x slower on some platforms until now".

That all said, I'm intrigued to see how the SVT family of encoders turns out. They seem to be throwing a fair bit of engineering at it, though initially it seemed the VP9 team were redirected to work on the AV1 encoder, so good to see some work on it again, even if it's just basic stuff like that. Part of the SVT idea is that optimisations can be shared between codec families I believe, so VP9 should be benefitting even when not the focus.

Their effective use of all a available cores seems like their secret weapon. In many real world situations that might give them an edge that other encoders would have to work hard to match if coming from the libvpx codebase.
dapperdan is offline   Reply With Quote
Old 22nd November 2019, 20:11   #1160  |  Link
Adonisds
Registered User
 
Join Date: Sep 2018
Posts: 14
Has anyone done a comparison of the quality of youtube VP9 vs Stadia VP9?
Adonisds is offline   Reply With Quote
Reply

Tags
google, ngov, vp8, vp9, vpx, webm

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:01.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.