Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > New and alternative video codecs

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th May 2018, 14:09   #61  |  Link
pie
Registered User
 
pie's Avatar
 
Join Date: Sep 2012
Posts: 17
Quote:
Originally Posted by LigH View Post
Building the googletest sources fails
iirc with mingw you need to pass -Dgtest_disable_pthreads=ON or something to that effect.
pie is offline   Reply With Quote
Old 18th May 2018, 14:26   #62  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
@pie: That was the old "master" version.
_

@jonatans:

This dev version crashes reading from a Y4M:

Code:
Assertion failed!

Program: E:\Programme\xvc\Win64\xvcenc.exe
File: H:/development/media-autobuild_suite-master/build/xvc/app/xvc_enc_app/y4m_reader.cc, Line 124

Expression: pic_fmt.input_bitdepth != 0
Source is the usual "foreman" clip from Derf's collection on xiph.org, only patched to PAL frame rate:

Code:
YUV4MPEG2 W352 H288 F25:1 Ip A128:117
FRAME
Maybe a "C420" should be added to the Y4M header so that your Y4M reader knows exactly which YUV variant it is?
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 18th May 2018, 14:47   #63  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
Yes, our y4m parser assumed that the C parameter would be present in the y4m header.

I have just changed so that "C420" is assumed if there is no C parameter present.

I also added support for "Cmono".
jonatans is offline   Reply With Quote
Old 18th May 2018, 17:45   #64  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
A good reason to build anew: xvc-dev 2018-05-18 bbf1325
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 22nd May 2018, 19:59   #65  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by augman000 View Post
By the way: I am really liking xvc. It is beating AV1, x265, and VP9 in my tests with PSNR and SSIM scores. I'd like to measure VMAF as well but I have been unsuccessful in all my attempts to compile a working win64/win32 binary.
SSIM and especially PSNR are not very good metrics, particularly with advanced new codecs. Or anything that is psychovisually tuned.

VMAF is probably the least-bad objective metric we have, but still has some pretty big limitations and blind spots. It wasn't tested with anything >1080p or below something like 300 Kbps. SDR 8-bit only. A quite weak temporal comparison module. And it was trained on just x264, and so doesn't know what to do with new types of artifacts AV1 and HEVC can have.

Plus there's the whole question of how you aggragate individual frame scores into a clip score. Just the mean of the metrics can't discriminate between content that is consistently mediocre versus oscillating between terrible and pristine.

With a new codec, actually looking at it is really the only thing that can give a better than a rough ballpark. Certainly a difference of less than 5 VMAF, 4 PSDR dB, and 3 SSIM dB should be verified visually.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 23rd May 2018, 05:13   #66  |  Link
maxcoolzero
Registered User
 
maxcoolzero's Avatar
 
Join Date: Apr 2018
Location: Sydney, Australia
Posts: 4
Wow. Looks cool
maxcoolzero is offline   Reply With Quote
Old 23rd May 2018, 09:12   #67  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
Quote:
Originally Posted by benwaggoner View Post
SSIM and especially PSNR are not very good metrics, particularly with advanced new codecs. Or anything that is psychovisually tuned.

VMAF is probably the least-bad objective metric we have, but still has some pretty big limitations and blind spots. It wasn't tested with anything >1080p or below something like 300 Kbps. SDR 8-bit only. A quite weak temporal comparison module. And it was trained on just x264, and so doesn't know what to do with new types of artifacts AV1 and HEVC can have.

Plus there's the whole question of how you aggragate individual frame scores into a clip score. Just the mean of the metrics can't discriminate between content that is consistently mediocre versus oscillating between terrible and pristine.

With a new codec, actually looking at it is really the only thing that can give a better than a rough ballpark. Certainly a difference of less than 5 VMAF, 4 PSDR dB, and 3 SSIM dB should be verified visually.
Thanks Ben, those are all very valid points.

I agree that PSNR is not a very good metric for determining visual quality. It is good for determining how close the compressed pictures are to the original pictures (in a mathematical sense - after all that is what PSNR measures, on a sample by sample basis). So if two different implementations are tuned to minimize MSE ("tune PSNR") then PSNR gives a good indication of how good the implementations are at doing just that. And even though this generally doesn't correlate well with visual quality it is good in the sense that it does not give any "false positives" (i.e. if one compressed picture A is tuned towards PSNR you cannot easily create another compressed picture B with higher PSNR by making it look worse than A).

We have put some results for the NETVC test conditions and test sequences at awcy.divideon.com, comparing xvc to AV1, both codecs with PSNR tuning (https://awcy.divideon.com/?job=1pass...3A34%3A38.388Z) including results for PSNR (16.5% savings) MS-SSIM (23.3% savings) and VMAF (18.5% savings). It can be seen that the gains are quite sequence dependent but the overall trend is very clear and quite consistent among the metrics.

There is also a comparison between xvc and HM (https://awcy.divideon.com/?job=1pass...3A34%3A38.388Z) which shows slightly larger bitrate savings.

But, as you correctly point out, what matters in the end is the visual quality which can only be determined by actually looking at compressed video (encoded under fair conditions when it comes to complexity etc.)

Please share your impressions if you have had a chance to look at any xvc encoded sequences or made any visual comparisons.
__________________
Jonatan Samuelsson
Co-founder and CEO at Divideon

www.divideon.com | xvc.io
jonatans is offline   Reply With Quote
Old 1st June 2018, 16:35   #68  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
xvc-dev 2018-05-24 9722ca7 (dev branch commits)
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 4th July 2018, 09:37   #69  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
New version of xvc just released!

The source code is available in the master branch at https://github.com/divideon/xvc.

More info about the release in the official press release: http://www.releasewire.com/press-rel...ne-1005793.htm

A few highlights:
- Better compression performance
- A royalty-free baseline profile
- Multi-threaded encoding
- Dual-licensing scheme: LGPL and commercial license

Enjoy! / Jonatan
__________________
Jonatan Samuelsson
Co-founder and CEO at Divideon

www.divideon.com | xvc.io
jonatans is offline   Reply With Quote
Old 4th July 2018, 10:47   #70  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Fresh build of xvc v2.0 2018-07-04 (commit 283009f) in my archive. No obvious code additions since last dev build in May.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 4th July 2018, 13:26   #71  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
^ Thanks!
colinhunt is offline   Reply With Quote
Old 4th July 2018, 14:02   #72  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Hmm, I can't make xvc 2.0 do multi-threading. Tried with "-threads 10" and "-threads -1" but it insists on running on a single core. Cmdline:

xvcenc.exe -input-file moto.yuv -input-width 1920 -input-height 1080 -input-bitdepth 8 -framerate 24 -verbose 1 -threads -1 -speed-mode 2 -qp 30 -explicit-encoder-setti
ngs "aqp_strength 16" -output-file output.xvc
colinhunt is offline   Reply With Quote
Old 4th July 2018, 14:46   #73  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
Thanks for making it available LigH. And thanks for testing colinhunt!

Did you try to run it long enough for the multi-threading to kick in?

Our multi-threading implementation uses picture level parallelism and due to the prediction structure it is only after the first two pictures have been completely encoded that multiple pictures are encoded in parallel. (Before that everything is coded in a single thread)

I just tried both my own compiled xvcenc and the one provided by LigH and both are able to do multi-threading.
__________________
Jonatan Samuelsson
Co-founder and CEO at Divideon

www.divideon.com | xvc.io
jonatans is offline   Reply With Quote
Old 4th July 2018, 14:56   #74  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Well, shoot. My impatience got the best of me yet again! I'll re-run the cmdline, and this time I'll wait for longer than 30 seconds. Thanks Jonatan!

update: it has now processed 19 pictures, based on the list it outputs on screen, but I still don't see multi-threading. Average CPU usage hovers at 5-6% and it looks like only a single core/thread is running at 100%.

...okay, I noticed now there is *some* life in the graphs for 7-8 threads but those peak at 5% before quickly dying back down to 0%. Not quite what I expected tbh.

update #2: I changed "-threads -1" to "threads 20" and now re-running the cmdline. A moment ago xvc's CPU usage hit 47%, while 10-ish threads were running at 100%. Encoder spat a number of lines of text on screen, followed by CPU usage dropping to 15% ... It jumped to 25% while I was typing that previous sentence ... And now to 45%. Here comes another set of text on screen, CPU use goes down to 10% and the cycle starts again.

Last edited by colinhunt; 4th July 2018 at 15:33.
colinhunt is offline   Reply With Quote
Old 4th July 2018, 17:07   #75  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Encoding finished in 94 minutes. Some months ago the same source file with same encoding parameters (except multi-threading, of course) took 183 minutes to encode. So almost 50% faster now with multi-threading.
colinhunt is offline   Reply With Quote
Old 5th July 2018, 07:59   #76  |  Link
jonatans
Registered User
 
Join Date: Oct 2017
Posts: 56
Hmm, that does not seem to be right. You should get substantially more than 2x speed-up (unless you are actually running it on a 2 core CPU ).

Perhaps you can try using this binary instead:

https://drive.google.com/open?id=1sP...5Mdixkvo5TqsSI

The reason for why you don't see higher than 50% CPU usage could potentially be related to memory-usage. With 20 threads and 1080p resolution, xvc might try to allocate more memory than what is available on the system. Are you running on a 20 core system? Perhaps you can try using something in the order of 8 or 10 threads instead to get more stable performance. Or try with lower resolution video just to see if you can get all cores fully utilized.

As you have correctly noted there is currently no way of letting xvcenc figure out the optimal number of threads to use (as is done in xvcdec). This is also related to the memory-usage. With additional testing on a wide variety of combinations of CPU-cores and memory-availability, it should be possible to figure out a good scheme for doing this automatically, but we haven't done that yet.
__________________
Jonatan Samuelsson
Co-founder and CEO at Divideon

www.divideon.com | xvc.io
jonatans is offline   Reply With Quote
Old 5th July 2018, 18:20   #77  |  Link
iwod
Registered User
 
Join Date: Apr 2002
Posts: 756
Quote:
The first version of xvc was released in September 2017 and was able to clearly outperform all existing video codecs including HEVC. The second version of xvc delivers even better performance and makes it possible to reduce the bitrate compared to the recently released AV1 codec by between 10% and 20% for the same visual quality.
I really rather like the PR. The humble, tone down figures. Pretty sure the first version already had similar to AV1 performance or better. At least according to other members clips and testing.
iwod is offline   Reply With Quote
Old 5th July 2018, 19:28   #78  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
Quote:
Originally Posted by jonatans View Post
Perhaps you can try using this binary instead
I'll give it a shot.

Quote:
The reason for why you don't see higher than 50% CPU usage could potentially be related to memory-usage. With 20 threads and 1080p resolution, xvc might try to allocate more memory than what is available on the system.
It was hogging a bit over 5GB, on a 32GB system.

Quote:
Are you running on a 20 core system?
No, the CPU's a 10-core / 20-thread Xeon.

Quote:
Perhaps you can try using something in the order of 8 or 10 threads instead to get more stable performance.
I'll try with 10 threads next time. Thanks Jonatan.
colinhunt is offline   Reply With Quote
Old 5th July 2018, 20:21   #79  |  Link
Leeloo Minaļ
Registered User
 
Join Date: Nov 2007
Posts: 50
I tried binary from jonatans and multi-threading seems to work well, at least for me.

Is it possible to show encoding progress in the next release ?
Leeloo Minaļ is offline   Reply With Quote
Old 5th July 2018, 22:51   #80  |  Link
colinhunt
Registered User
 
Join Date: Dec 2002
Posts: 1,022
xvcenc.exe -input-file moto.yuv -input-width 1920 -input-height 1080 -input-bitdepth 8 -framerate 24 -verbose 1 -threads 20 -speed-mode 2 -qp 30 -explicit-encoder-settings "aqp_strength 16" -output-file output.xvc

v1 binary: 183 minutes, no multi-threading.
v2 binary (LigH's): 94 minutes. 20 threads, CPU usage peaked at 47%.
v2 binary (Jonatan's): 51,3 minutes. 20 threads, RAM usage 5.9GB, CPU usage peaked at 56%.

v2 binary (Jonatan's): 51 minutes. 10 threads, CPU usage peaked at 55%.

v2 binary (Jonatan's): 51 minutes. 40 threads, RAM usage 8.5GB, CPU usage peaked at 59%.
v2 binary (Jonatan's): 51 minutes. 8 threads, RAM usage 2.1GB.

Last edited by colinhunt; 6th July 2018 at 18:34.
colinhunt is offline   Reply With Quote
Reply

Tags
codec, compression, video codec, video encoding, xvc

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:18.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.