Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd April 2021, 00:49   #1  |  Link
Dyomich
Codec Analysis Expert
 
Join Date: Feb 2006
Location: Moscow
Posts: 37
MSU released benchmark of online video encoding services

Hi! MSU Graphics & Media Lab Video Group has released a new Codecs Comparison Report and we are happy to present it to you.

This test included encoding performance comparison using services' default settings. This comparison is based on these main points:
  • 7 cloud-based video-encoding services were compared: 3 with their own storages (Alibaba ApsaraVideo for Media Processing, Kingsoft Cloud and Tencent Media Processing Service) and 4 with storages at Amazon S3 (AWS Elemental MediaConvert, Coconut, Qencode, and Zencoder)
  • Encoding time measurement. We analyzed how much time a job waits in the services’ queue.
  • Default encoding settings efficiency. We launched encoding out-of-the-box without changing services options and measured objective video quality.
  • Automatic sequence-based encoding options (per-title encoding) for services that support this feature.
  • Test setup: 15 FullHD videos of different types and content (including UGC). Two types of measurements: the main case involved encoding into FullHD, and the additional case included transcoding into several resolutions. And finally, we tried to use anonymous accounts for testing for fair comparison
In our previous comparison (2019 Cloud Comparison) the results showed that for the same quality, the file-size difference can reach 100% and the cost per minute difference is 700%. This year’s comparison showed about 80% file-size difference and 150% cost per minute difference. Below we show the winners by different categories.

Best services in speed/quality trade-off
We measured encoded video quality and encoding time using services’ callbacks. This nomination shows how frequently a service was Pareto-optimal for encoding speed and quality scores for HEVC and H.264 averaged for all test videos (SSIM metric). Results of AWS Elemental MediaConvert were used as a reference to get quality scores.

The most frequent Pareto-optimal services in terms of encoding speed and quality were:
  • Tencent Media Processing Service
  • Zencoder
  • Kingsoft Cloud
Also Alibaba ApsaraVideo for Media Processing showed good quality for higher encoding speed.

The plot below shows an average encoding speed and file size among all videos. This statistics is a bit different from a number of videos, where a service showed optimal scores.




Best services in cost/quality trade-off
This nomination shows how frequently a service was Pareto-optimal for video encoding cost and quality among all test videos. The following chart shows relative bitrate and encoding cost for the same quality (SSIM) averaged for all videos. The best quality for the least prices has:
  • Tencent Media Processing Service showed best encoding efficiency
  • Kingsoft Cloud, Qencode (for H.264 and HEVC encoding) and Coconut (for HEVC encoding) are the cheapest and keep a good bitrate at the same time



Per-title encoding test
In addition to default encoding settings, per-sequence encoding optimization was tested for services, which supported this feature during testing. Unfortunately, we didn't test AWS Elemental MediaConvert sequence-based encoding, because it was released after the comparison was started, and Qencode sequence-based encoding, because it produces only one output without any bitrate or resolution variation.

Per-title encoding option optimizes encoding settings for each input video to get better encoding performance. The first and second places in nomination for the best quality per-title encoding:
  • Tencent Media Processing Service
  • Alibaba ApsaraVideo for Media Processing



Here we show the winners in different categories in one table.


We should say that using only default settings may not provide a fully comprehensive comparison. Some services can set base profiles by default for better streams compatibility. On the other hand, when we tried to set the same settings for all services, we found out that sometimes we can not vary encoding options in service interfaces.

There may be different ways on how to compare services, and we would really love to hear which scenarios are interesting for you. Please, tell us what information in our comparison was the most useful for you? And, more importantly, what would you like to see in future comparisons? Or what would you like to change in our comparisons?
You can help to make codecs comparisons better!

Stay tuned!
Dyomich is offline   Reply With Quote
Old 2nd April 2021, 01:50   #2  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,627
The mean YUV-SSIM value of all frames in a piece of content really isn't a very useful metric. It's better than Y-only PSNR (which we see a lot). But the mean of a bunch of frames doesn't account for variability in the quality throughout an encode. A file that oscillates between -5 dB and -15 dB has the same mean as a consistent -10 dB, but will look much worse.

A single value rating of the quality of a file of >>10 second is really an unsolved problem for the industry, and an essential one if we want to make broad comparisons like this.

Even just publishing the variance of the metrics would help a lot.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Tags
cloud, codec comparison, comparison, transcoding, video encoding

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.