x264 Video Filters

[parent5446]

Quick question about the inner workings of x264's filter chain. I'm experimenting around and writing some new filters so that hopefully I can avoid Avisynth and other similar programs for basic pre-processing (primarily out of platform independence concerns). I've managed to figure out the basic interaction and have written filters for some of Avisynth's built-in filters (Reverse, FreezeFrame, SelectRangeEvery, and Trim), and now I'm looking to mess around with maybe a deinterlacing filter.

The only problem is that I have no idea how interlaced video is stored within x264. I've deduced that in progressive video each frame is stored in a cli_image_t (within a cli_pic_t) with an array of values for each plane in the image, but how is it done with interlaced video? My first guess would be that two fields are combined and stored as one frame, but it could be just as likely that each field is stored in a separate cli_image_t.

My secondary question is what the functionality of x264's cache filter. I mean the name makes it sound obvious, but I'm a little confused about how it would be used (or when it should be used) in the filter chain. The select_every filter uses a cache the size of the step_size, but I'm not entirely sure why because the filter doesn't technically need to rewind to other frames in the pattern in order to provide the next frame. And in the case of a deinterlacing filter, the filter would need access to the adjacent fields of the video, so would it be necessary to set up a cache for just two-three frame storage?

Thanks for the help.

[Dark Shikari]

Interlaced is just weaved, it's two fields per frame.

[kemuri-_9]

the select every filter can rewind depending on the pattern supplied,
e.g. step=5,4,3,2,1,0 is a full reverse of each 5 frame set, this is a rewind operation and the frames need to be cached to allow this.

as such, if you're ever rewinding then you will particularly need to use the cache_filter to cache the frames that arrive.

This is primarily because of x264's ability to read in piped input where naturally rewinding is impossible, but you may want to filter it in certain ways anyway.

most of x264's input system currently assumes rewinding is impossible so even for inputs where it could rewind, it doesn't allow for it.

[parent5446]

Thanks for the help! This will definitely help with the implementation of yadif as an x264 filter. In that case there's definitely going to need to be a small cache of one or two frames for the temporal prediction calculations. The interesting part is going to be implementing modes 1 and 3, which double the framerate and thus require new frames to be produced.

Also, kemuri-_9, I noticed your site has a patch that adds audio filters. Does this still work? I'm interested in what exactly is does.

[J_Darnley]

Should I point out that a patch for that already exists? Okay, I haven't updated it in a long while. It wasn't very hard to make it double the frame rate.

[kemuri-_9]

Originally Posted by parent5446

Also, kemuri-_9, I noticed your site has a patch that adds audio filters. Does this still work? I'm interested in what exactly is does.

there was once a failed GSOC project that was to add audio input, filtering, and encoding support to x264 cli.
What you found is one such patch from when it was actually being developed.
I was the one primarily reviewing the student's work on it, as I wrote the existing video filter system.
Don't use that patch - there has been work done since then, so you can find a newer patch elsewhere if you are still inclined to look at it.

and J_Darnley is correct, he already wrote a patch to add yadif functionality to the video filter system.

inserting new frames is not particularly hard, simply indicate that you're adding frames during the filter initialization (if there is a non-zero framecount to begin with) then make those frames when they're requested.

[parent5446]

Well if a patch already exists then that saves me a hell of a lot of time. XD Where is the patch now?

[J_Darnley]

There's probably an old one linked to around here somewhere. There will definitely be an old one on doom10, the filter thread in the h264 sub-forum. Regardless, it probably needs some changing due to the (not so) recent changes to the input and filter system. I'll see to it when I get back in front of the machine with it on.

[EDIT] Sorry, that will have to be in the morning. I did go to find this for you though: http://doom10.org/index.php?topic=177.0

[roozhou]

Originally Posted by kemuri-_9

inserting new frames is not particularly hard, simply indicate that you're adding frames during the filter initialization (if there is a non-zero framecount to begin with) then make those frames when they're requested.

Is it impossible to add a filter that adaptively add or remove frames? Since x264 always pull frame by its frame No. instead of timestamps.

[parent5446]

Originally Posted by roozhou

Is it impossible to add a filter that adaptively add or remove frames? Since x264 always pull frame by its frame No. instead of timestamps.

You can adaptively add/remove frames. All you need to do is make sure to adjust the framerate in order to compensate for difference in frame count (or if you'r removing different amounts of frames in different sections, adjust the cli_pic_t.duration for VFR).

[kemuri-_9]

Originally Posted by roozhou

Is it impossible to add a filter that adaptively add or remove frames? Since x264 always pull frame by its frame No. instead of timestamps.

Indeed x264cli does pull frames by number, and not by pts.
It standardly reads up until the number of known frames from the result of the filter chain, or the user specified count.

As such, the only way I can see to do this as freely as you seem to be desiring it, is to have your filter set the frame count to 0.
With this, the filter system will keep processing until something returns a failure code, as it has no clue how many frames there will be.
You can then add new frames or discard frames from previous filters in the chain fairly arbitrarily upon the requests to your specific filter.

If you don't specify the frame count as 0 then removing frames arbitrarily should still be applicable,
but adding them arbitrarily can prove to be limited as x264 will only read up to a certain number of frames before stopping.