Doom9's Forum - View Single Post - FFAStrans

FranceBB · 11th July 2019, 19:11

Hi,
I've been using Avisynth for years, but ever since 2016 we needed to automatize our workflows a lot as we had to encode too many files and we didn't have enough encoders, so manually encoding each and every raw file wasn't an option anymore.
Since then, I've started creating a few batch scripts and Avisynth scripts to automatize some common things based on the characteristics of every file, but then I came across a project that Steinar Apalnes was working on called FFAStrans and I switched to it.
Today I'm gonna talk about it and the power of open source, as I think that many broadcasters and production houses are paying a lot of money for softwares like Telestream Vantage, Harmonics ProCoder, Adobe Media Encoder, Selenio, AWS Transcoder and similar while Avisynth, ffmpeg and x264 can deliver better results and are also free.

User Interface:

This is how the user interface of one of my main workflows looks like. It might seem a bit messy, but it's actually pretty clear.
Let's take a look at the Avisynth_in workflow which is meant to encode files in MPEG-2 XDCAM with the following specifications:

Video Codec: MPEG-2
Resolution: 1920x1080
Bitrate: 50 Mbit/s
Sampling: 4:2:2 yv16 planar
Framerate: 25i
Type: Interlaced TFF
Bit depth: 8bit

Audio Codec: PCM
Bit Depth: 24bit
Audio Channels: 8ch mono

The workflow starts with a watchfolder, then it starts indexing audio and video using FFMpegSource2, then it checks for the number of audio channels and basically always makes 8 channels (i.e if the original file has 1 ch, it duplicates it to all the 8 channels, if it has 2 audio channels, it duplicates left and right to the 8 channels and so on). Then, it applies loudness correction by making sure that it's 48'000Hz, 24bit and that the loudness is -24LUFS, then it checks for interlacing and it decides whether it has to deinterlace or not and eventually use bob-deinterlacing using yadif and so on. After that, it cheks the resulting framerate and it decides whether it has to apply a speed-up + pitch adjustment (23.976fps -> 25), or if it has to leave it as it is 'cause it's 25fps progressive or if it has to blend everything to 50fps progressive and then divide in fields or just divide everything in fields 'cause it's already 50fps progressive. Last but not least, it converts everything to yv16 and it applies clipping in order to make sure that Luma and Chroma are in limited tv range (16-235) and then Avisynth gives the resulting uncompressed A/V stream to ffmpeg which encodes it in XDCAM and then remuxes it using BBC BMX Transwrap. Finally, it's gonna deliver the file to a shared storage and it's gonna delete the original file which has already been encoded (if it has been encoded successfully).

By default there are quite a few blocks, but it's very easy to create new blocks and entirely customize them. For instance, this is what my block called "Limiter Tv Range" looks like:

Let's zoom in:

As you can see, the user interface is just a visual representation of what Avisynth is doing, therefore it's possible to do pretty much everything you generally do with Avisynth but in automatic and since automatize workflows is something companies like... a lot, I thought that this was the right way forward.
Here's another example; this time, the block is the one in charge of the Loudness Correction:

See? Each and every block is just like a normal Avisynth Script that you generally write down on a daily basis whenever you have to encode something.
For instance, Limiter Tv Range is basically just clipping:

Code:

Limiter(min_luma=16, max_luma=235, min_chroma=16, max_chroma=240)

Loudness Correction is basically just:

Code:

ResampleAudio(48000)
Normalize(0.22)
ConvertAudioTo24bit()

Blending and interlacing to 25i is basically just:

Code:

ConvertFPS(50)
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()

The channels duplicator for stereo files is basically just:

Code:

video=FFVideoSource("file.mov")
audiook=FFAudioSource("file.mov")
audio=MergeChannels(audiook, audiook, audiook, audiook)
AudioDub(video, audio)

and so on...
Oh, just to be precise, I said that it's based on Avisynth, but it's actually based on Avisynth+ (both x86 and x64), so it can use all the additional features of Avisynth+ like regular high bit depth, dealing with different color curves, HDR and so on; for instance, this is what converting a C-Log3 to BT709 linear looks like:

Let's zoom in once again:

As you can see, it basically takes the input as it is (either 8bit, 10bit, 12bit etc), it brings everything to 16bit planar, then it converts it to RGB, applies one of my LUTs and then it dithers everything down to 8bit with the Floyd-Steinberg error diffusion.

You can basically create many different workflows according to your needs, 'cause the potential of Avisynth is huge; those are some of the watchfolders I made:

As you can basically see, it pretty much encodes files for either archival or as mezzanine files to be used later or as files to be delivered to users via whatever service and so on.

Alright, let's crack on and see how encoding looks like, shall we?

The monitor looks essential and perfectly productive; you can basically see which workflow is doing what and how long it's gonna take before the encode finishes based on how long a file is and the fps of the encode. It can deliver the resulting file to a folder or to an FTP, according to your needs. In my case, I just set it to deliver the file to a shared folder in a shared storage.
As to the CPU efficiency, it's as efficient as Avisynth and ffmpeg/x262/x264 etc would be on your CPU.
In my case, since I still do many encodes in MPEG-2 and the encoder is definitely not optimized for multithreading, the power of Avisynth comes in parallel encodes rather than single encodes, which can saturate all the cores/threads of my Dual Xeon:

If you work in a company and need to encode many different files on a daily basis, I strongly recommend you to choose FFAStrans as it's free to use, based on open source programmes like Avisynth+, ffmpeg and so on and it's completely customizable.

This is supposed to be an introduction to this awesome software which I've been using ever since 2016-2017 and that I still use on a daily basis whenever I don't have time to manually write a script and manually encode a masterfile.
Of course, not everything can be done automatically and a manual encode should always be preferred to an automatic one, but let's face it: sometimes having something implemented for you that helps you finishing your work is actually very useful. If you want a complete guide on how to use it with many more screenshots, let me know. In the meantime, here's my personal workflows package which comes pre-loaded with scripts, plugins, and makes use of my LUT collection:

FranceBB Custom Workflows Package: Link

FFAStrans Stable version: Link

Please note that the legacy 32bit is legacy 'cause it's a very old version with some components modified with XP compatible ones, but you shouldn't really be using it.

Useful Links:

Official FFAStrans website by Steinar Apalnes: https://ffastrans.com/wp/
FFASTrans Wiki: http://www.ffastrans.com/wiki/doku.php

Developer: Steinar Apalnes
Maintainers: emcodem, momocampo, FranceBB

11th July 2019, 19:11	#1 \| Link
FranceBB Broadcast Encoder Join Date: Nov 2013 Location: Royal Borough of Kensington & Chelsea, UK Posts: 2,904	FFAStrans - FFMpeg Avisynth Transcoder Hi, I've been using Avisynth for years, but ever since 2016 we needed to automatize our workflows a lot as we had to encode too many files and we didn't have enough encoders, so manually encoding each and every raw file wasn't an option anymore. Since then, I've started creating a few batch scripts and Avisynth scripts to automatize some common things based on the characteristics of every file, but then I came across a project that Steinar Apalnes was working on called FFAStrans and I switched to it. Today I'm gonna talk about it and the power of open source, as I think that many broadcasters and production houses are paying a lot of money for softwares like Telestream Vantage, Harmonics ProCoder, Adobe Media Encoder, Selenio, AWS Transcoder and similar while Avisynth, ffmpeg and x264 can deliver better results and are also free. User Interface: This is how the user interface of one of my main workflows looks like. It might seem a bit messy, but it's actually pretty clear. Let's take a look at the Avisynth_in workflow which is meant to encode files in MPEG-2 XDCAM with the following specifications: Video Codec: MPEG-2 Resolution: 1920x1080 Bitrate: 50 Mbit/s Sampling: 4:2:2 yv16 planar Framerate: 25i Type: Interlaced TFF Bit depth: 8bit Audio Codec: PCM Bit Depth: 24bit Audio Channels: 8ch mono The workflow starts with a watchfolder, then it starts indexing audio and video using FFMpegSource2, then it checks for the number of audio channels and basically always makes 8 channels (i.e if the original file has 1 ch, it duplicates it to all the 8 channels, if it has 2 audio channels, it duplicates left and right to the 8 channels and so on). Then, it applies loudness correction by making sure that it's 48'000Hz, 24bit and that the loudness is -24LUFS, then it checks for interlacing and it decides whether it has to deinterlace or not and eventually use bob-deinterlacing using yadif and so on. After that, it cheks the resulting framerate and it decides whether it has to apply a speed-up + pitch adjustment (23.976fps -> 25), or if it has to leave it as it is 'cause it's 25fps progressive or if it has to blend everything to 50fps progressive and then divide in fields or just divide everything in fields 'cause it's already 50fps progressive. Last but not least, it converts everything to yv16 and it applies clipping in order to make sure that Luma and Chroma are in limited tv range (16-235) and then Avisynth gives the resulting uncompressed A/V stream to ffmpeg which encodes it in XDCAM and then remuxes it using BBC BMX Transwrap. Finally, it's gonna deliver the file to a shared storage and it's gonna delete the original file which has already been encoded (if it has been encoded successfully). By default there are quite a few blocks, but it's very easy to create new blocks and entirely customize them. For instance, this is what my block called "Limiter Tv Range" looks like: Let's zoom in: As you can see, the user interface is just a visual representation of what Avisynth is doing, therefore it's possible to do pretty much everything you generally do with Avisynth but in automatic and since automatize workflows is something companies like... a lot, I thought that this was the right way forward. Here's another example; this time, the block is the one in charge of the Loudness Correction: See? Each and every block is just like a normal Avisynth Script that you generally write down on a daily basis whenever you have to encode something. For instance, Limiter Tv Range is basically just clipping: Code: Limiter(min_luma=16, max_luma=235, min_chroma=16, max_chroma=240) Loudness Correction is basically just: Code: ResampleAudio(48000) Normalize(0.22) ConvertAudioTo24bit() Blending and interlacing to 25i is basically just: Code: ConvertFPS(50) assumeTFF() separatefields() selectevery(4,0,3) weave() The channels duplicator for stereo files is basically just: Code: video=FFVideoSource("file.mov") audiook=FFAudioSource("file.mov") audio=MergeChannels(audiook, audiook, audiook, audiook) AudioDub(video, audio) and so on... Oh, just to be precise, I said that it's based on Avisynth, but it's actually based on Avisynth+ (both x86 and x64), so it can use all the additional features of Avisynth+ like regular high bit depth, dealing with different color curves, HDR and so on; for instance, this is what converting a C-Log3 to BT709 linear looks like: Let's zoom in once again: As you can see, it basically takes the input as it is (either 8bit, 10bit, 12bit etc), it brings everything to 16bit planar, then it converts it to RGB, applies one of my LUTs and then it dithers everything down to 8bit with the Floyd-Steinberg error diffusion. You can basically create many different workflows according to your needs, 'cause the potential of Avisynth is huge; those are some of the watchfolders I made: As you can basically see, it pretty much encodes files for either archival or as mezzanine files to be used later or as files to be delivered to users via whatever service and so on. Alright, let's crack on and see how encoding looks like, shall we? The monitor looks essential and perfectly productive; you can basically see which workflow is doing what and how long it's gonna take before the encode finishes based on how long a file is and the fps of the encode. It can deliver the resulting file to a folder or to an FTP, according to your needs. In my case, I just set it to deliver the file to a shared folder in a shared storage. As to the CPU efficiency, it's as efficient as Avisynth and ffmpeg/x262/x264 etc would be on your CPU. In my case, since I still do many encodes in MPEG-2 and the encoder is definitely not optimized for multithreading, the power of Avisynth comes in parallel encodes rather than single encodes, which can saturate all the cores/threads of my Dual Xeon: If you work in a company and need to encode many different files on a daily basis, I strongly recommend you to choose FFAStrans as it's free to use, based on open source programmes like Avisynth+, ffmpeg and so on and it's completely customizable. This is supposed to be an introduction to this awesome software which I've been using ever since 2016-2017 and that I still use on a daily basis whenever I don't have time to manually write a script and manually encode a masterfile. Of course, not everything can be done automatically and a manual encode should always be preferred to an automatic one, but let's face it: sometimes having something implemented for you that helps you finishing your work is actually very useful. If you want a complete guide on how to use it with many more screenshots, let me know. In the meantime, here's my personal workflows package which comes pre-loaded with scripts, plugins, and makes use of my LUT collection: FranceBB Custom Workflows Package: Link FFAStrans Stable version: Link Please note that the legacy 32bit is legacy 'cause it's a very old version with some components modified with XP compatible ones, but you shouldn't really be using it. Useful Links: Official FFAStrans website by Steinar Apalnes: https://ffastrans.com/wp/ FFASTrans Wiki: http://www.ffastrans.com/wiki/doku.php Developer: Steinar Apalnes Maintainers: emcodem, momocampo, FranceBB __________________ LUT Collection FFAStrans Videotek - AAA - SafeColorLimiter Last edited by FranceBB; 20th April 2021 at 22:00.