Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 11th July 2019, 19:11   #1  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 581
FFAStrans - FFMpeg Avisynth Transcoder

Hi,
I've been using Avisynth for years, but ever since 2016 we needed to automatize our workflows a lot as we had to encode too many files and we didn't have enough encoders, so manually encoding each and every raw file wasn't an option anymore.
Since then, I've started creating a few batch scripts and Avisynth scripts to automatize some common things based on the characteristics of every file, but then I came across a project that Steinar Apalnes was working on called FFAStrans and I switched to it.
Today I'm gonna talk about it and the power of open source, as I think that many broadcasters and production houses are paying a lot of money for softwares like Telestream Vantage, Harmonics ProCoder, Adobe Media Encoder, Selenio, AWS Transcoder and similar while Avisynth, ffmpeg and x264 can deliver better results and are also free.


User Interface:




This is how the user interface of one of my main workflows looks like. It might seem a bit messy, but it's actually pretty clear.
Let's take a look at the Avisynth_in workflow which is meant to encode files in MPEG-2 XDCAM with the following specifications:

Video Codec: MPEG-2
Resolution: 1920x1080
Bitrate: 50 Mbit/s
Sampling: 4:2:2 yv16 planar
Framerate: 25i
Type: Interlaced TFF
Bit depth: 8bit

Audio Codec: PCM
Bit Depth: 24bit
Audio Channels: 8ch mono

The workflow starts with a watchfolder, then it starts indexing audio and video using FFMpegSource2, then it checks for the number of audio channels and basically always makes 8 channels (i.e if the original file has 1 ch, it duplicates it to all the 8 channels, if it has 2 audio channels, it duplicates left and right to the 8 channels and so on). Then, it applies loudness correction by making sure that it's 48'000Hz, 24bit and that the loudness is -24LUFS, then it checks for interlacing and it decides whether it has to deinterlace or not and eventually use bob-deinterlacing using yadif and so on. After that, it cheks the resulting framerate and it decides whether it has to apply a speed-up + pitch adjustment (23.976fps -> 25), or if it has to leave it as it is 'cause it's 25fps progressive or if it has to blend everything to 50fps progressive and then divide in fields or just divide everything in fields 'cause it's already 50fps progressive. Last but not least, it converts everything to yv16 and it applies clipping in order to make sure that Luma and Chroma are in limited tv range (16-235) and then Avisynth gives the resulting uncompressed A/V stream to ffmpeg which encodes it in XDCAM and then remuxes it using BBC BMX Transwrap. Finally, it's gonna deliver the file to a shared storage and it's gonna delete the original file which has already been encoded (if it has been encoded successfully).


By default there are quite a few blocks, but it's very easy to create new blocks and entirely customize them. For instance, this is what my block called "Limiter Tv Range" looks like:



Let's zoom in:



As you can see, the user interface is just a visual representation of what Avisynth is doing, therefore it's possible to do pretty much everything you generally do with Avisynth but in automatic and since automatize workflows is something companies like... a lot, I thought that this was the right way forward.
Here's another example; this time, the block is the one in charge of the Loudness Correction:



See? Each and every block is just like a normal Avisynth Script that you generally write down on a daily basis whenever you have to encode something.
For instance, Limiter Tv Range is basically just clipping:

Code:
Limiter(min_luma=16, max_luma=235, min_chroma=16, max_chroma=240)
Loudness Correction is basically just:

Code:
ResampleAudio(48000)
Normalize(0.22)
ConvertAudioTo24bit()
Blending and interlacing to 25i is basically just:

Code:
ConvertFPS(50)
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()
The channels duplicator for stereo files is basically just:

Code:
video=FFVideoSource("file.mov")
audiook=FFAudioSource("file.mov")
audio=MergeChannels(audiook, audiook, audiook, audiook)
AudioDub(video, audio)
and so on...
Oh, just to be precise, I said that it's based on Avisynth, but it's actually based on Avisynth+ (both x86 and x64), so it can use all the additional features of Avisynth+ like regular high bit depth, dealing with different color curves, HDR and so on; for instance, this is what converting a C-Log3 to BT709 linear looks like:



Let's zoom in once again:



As you can see, it basically takes the input as it is (either 8bit, 10bit, 12bit etc), it brings everything to 16bit planar, then it converts it to RGB, applies one of my LUTs and then it dithers everything down to 8bit with the Floyd-Steinberg error diffusion.

You can basically create many different workflows according to your needs, 'cause the potential of Avisynth is huge; those are some of the watchfolders I made:



As you can basically see, it pretty much encodes files for either archival or as mezzanine files to be used later or as files to be delivered to users via whatever service and so on.

Alright, let's crack on and see how encoding looks like, shall we?



The monitor looks essential and perfectly productive; you can basically see which workflow is doing what and how long it's gonna take before the encode finishes based on how long a file is and the fps of the encode. It can deliver the resulting file to a folder or to an FTP, according to your needs. In my case, I just set it to deliver the file to a shared folder in a shared storage.
As to the CPU efficiency, it's as efficient as Avisynth and ffmpeg/x262/x264 etc would be on your CPU.
In my case, since I still do many encodes in MPEG-2 and the encoder is definitely not optimized for multithreading, the power of Avisynth comes in parallel encodes rather than single encodes, which can saturate all the cores/threads of my Dual Xeon:





If you work in a company and need to encode many different files on a daily basis, I strongly recommend you to choose FFAStrans as it's free to use, based on open source programmes like Avisynth+, ffmpeg and so on and it's completely customizable.

This is supposed to be an introduction to this awesome software which I've been using ever since 2016-2017 and that I still use on a daily basis whenever I don't have time to manually write a script and manually encode a masterfile.
Of course, not everything can be done automatically and a manual encode should always be preferred to an automatic one, but let's face it: sometimes having something implemented for you that helps you finishing your work is actually very useful. If you want a complete guide on how to use it with many more screenshots, let me know. In the meantime, here's my personal copy of FFAStrans which comes pre-loaded with scripts, plugins, an updated and compiled copy of plugins, Avisynth+ and ffmpeg and of course my LUT collection:

FFAStrans FranceBB pre-loaded version: Link

FFAStrans Default Normal (Steinar Apalnes) version: Link

FFAStrans Legacy 32bit Normal Version with XP compatibility mod: Link

Please note that the legacy 32bit is legacy 'cause it's a very old version with some components modified with XP compatible ones, but you shouldn't really be using it.

Password: Itadakimasu0


Useful Links:

Official FFAStrans website by Steinar Apalnes: http://www.ffastrans.com/
__________________
Broadcast Encoder
LinkedIn

Last edited by FranceBB; 11th July 2019 at 19:48.
FranceBB is offline   Reply With Quote
Old 11th July 2019, 20:55   #2  |  Link
Gser
Registered User
 
Join Date: Apr 2008
Posts: 411
Huh so people really use x262?
Gser is offline   Reply With Quote
Old 11th July 2019, 21:55   #3  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 874
The images with the "graphs" are not really readable, could you post a higher res or a version with a zoomed in area?
ChaosKing is offline   Reply With Quote
Old 12th July 2019, 01:19   #4  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 581
Quote:
Originally Posted by Gser View Post
Huh so people really use x262?
MPEG-2 in 2019? Yep. Either x262 or Lavc58.19.102 mpeg2video encoder. Anyway, according to PSNR and SSIM values I've got in many tests their performances are not very good. Still, they are free and open source, so this is what I use. Besides, old playout ports force me to encode in MPEG-2 still today so... this is pretty much it.

Quote:
Originally Posted by ChaosKing View Post
The images with the "graphs" are not really readable, could you post a higher res or a version with a zoomed in area?
Sure, I'm gonna zoom them tomorrow and upload a few workflows zoomed in so that they are readable.
__________________
Broadcast Encoder
LinkedIn
FranceBB is offline   Reply With Quote
Old 13th July 2019, 17:58   #5  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 581
Alright, as requested, here are the zoomed in images.

Let's start with Avisynth_in:



Let's zoom in:



As you can see, once it indexes the audio and video file, HuffYUV is called and it creates a mezzanine lossless video file.
Then, such a video is indexed again and channels are duplicated/ordered accordingly.
Then, Loudness correction to 48'000Hz, 24bit and -24LUFS with Normalize(0.22) is applied.
The next step is yadif which decides whether it's gonna deinterlace it, bob-deinterlace it or leave it as it is if it's progressive.
Of course, this doesn't work with telecined material, which has to be handled differently but once again, as I said before, a manual encode should be preferred over an automatic one and in case of telecined material, you could just make the effort to write

Code:
tfm(mode=1,pp=5,slow=2,micmatching=2,clip2=tdeint(mode=2,type=3)) 
tdecimate()
Anyway, let's keep going with the second zoomed part of the Avisynth_in workflow.



As you can see, blending is applied according to the framerate. The idea is to blend everything to 50fps progressive and then divide into fields upon encoding. If it's over 70fps, then it just drops frames 'till it gets to 50fps. This workflow is mainly used for our news channel, therefore blending is more than enough. For the other channels, I created another conditional which applies speed-up with pitch adjustment if the framerate is 23.976fps, or leave it as it is if it's 25fps progressive and makes a progressive flagged as interlaced.
Then clipping is applied in the "Limiter Tv Range" block and the resulting uncompressed A/V stream is encoded in MPEG-2.
Lastly, the resulting file is remuxed by BBC BMX Transwrap, delivered to a folder and the original file is deleted.
"Pulizia Cache" means "Cache deletion", which basically removes the mezzanine file, the Avisynth Script created and the original file from the temp folder.


Let's take a look at the "Repair_in" workflow:



As you can see "Repair_in" is just a fancy name for "Linear Interpolation" performed by mvtools. It basically consists in Indexing, deinterlacing eventually interlaced materials, Resize using Spline64Resize, Linear Interpolation with mvtools, clipping 16-235 limited tv range, encoding and delivering the file, then deleting the temp files/cache.
Sure, linear interpolation ain't perfect and it might introduce artifacts, however it can be useful in some cases like whenever a normal static pan has been recorded by a camera and we want it to be as smooth as possible. MVTools in this case always tries to apply linear interpolation and create a real 50fps progressive file which is then divided into fields to make a real 25i file.

Speaking of MVTools, let's take a look at the workflow "Slowmo_any":



Let's zoom in:



I gotta say that whenever we need to make a slowmotion, we record it with a very high framerate, like at 100, 200, 300, 500, and even 700fps, however we might not have cameras capable of recording such an high framerate at an high enough resolution; for instance, 700fps can be recorded by our cameras in HD only (1280x720). As a matter of fact, we generally upscale them to FULL HD and 4K. So, whenever we need to slow down those files, we simply use "AssumeFPS(50)".



After AssumeFPS(50), we apply clipping in order to have luma and chroma in Limited Tv Range and then we divide in fields to get a 25i and encode it. If and only if we need to make a slowmotion of something that we didn't record ourselves at high framerate, then we use MVTools and we cross our fingers.

Let's now take a look at this x264 workflow:



Let's zoom in:



As you can see, this workflow is made to produce low-res, low-bitrate proxy files to be viewed by journalists, reporters and so on out of our MPEG-2 50Mbit/s yv16 mezzanine files. In the x264_proxy workflow, the duration of the file is analysed and bitrate is calculated accordingly in order to be as large as 2GB maximum. If it's only few seconds, it would get a too high bitrate, so it's capped at 2500 kbit/s. x264 What'sapp has a similar constrain.



x264 and x264 "Archivio Esterne" means "Archive for External Productions" and in this case, x264 can go up to 25 Mbit/s level 4.1, which is a good enough bitrate for a video to be used for productions that are sent on site.

All these workflows make use of custom avisynth scripts and you can create them as well by right-clicking on the UI and either write your own script from scratch, use some of my personalized blocks (Scripts) or use some of the built-in blocks/scripts that Steinar wrote:










I hope it makes things a bit more clear.
I'll dive into the creation of workflows in a more accurate way if you want.

Cheers,
Frank.
__________________
Broadcast Encoder
LinkedIn

Last edited by FranceBB; 13th July 2019 at 18:06.
FranceBB is offline   Reply With Quote
Old 16th July 2019, 12:19   #6  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 707
Now that's awesome !!!
Deep respect and many thanks !
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're working on that issue. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.