Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 6th November 2013, 18:32   #241  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by Lenchik View Post
"nah, this will break my scripts because I found a way to rely on this behavior" or something like that. I got used to current behaviour. And moreover i am often creating chains of scripts by importing them into one biggest. I think that new modification can make more difficulties than making life easier.
actually, i'm pretty confident that the change wouldn't break any existing scripts. from that point of view it should be totally harmless. i have other concerns, see my previous post.
ultim is offline  
Old 6th November 2013, 19:31   #242  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by SEt View Post
I doubt anyone it going to implement it, but as you requested, general outline:


In one word I can describe it as "superscalar" architecture. The same ideas as used in modern superscalar CPUs:

1) Work is separated from data flow. (-> denotes function call)
How it goes now: Render->FilterB->FilterA->Source.
How it should be: Scheduler->Source, Scheduler->FilterA, Scheduler->FilterB.
2) Filter has to be able to tell the scheduler which frame numbers from which its sources it needs to produce each destination frame number. Fast, without actual frames, but maybe imprecise (can fail its processing call and request more).
3) There are several kinds of filters: one instance multicall (like MTMode 1), one instance singlecall (MTMode 3), multi instance one call per instance (MTMode 2). Type is provided by the filter itself, always.
4) Based on 1-3, number of available hardware threads and data flow graph Scheduler builds filter call plan in superscalar way: several frames in flight on different stages.
5) Refcounting and no cache. Based on call plan each frame has its own reference count and will be kept exactly until all references are used. All produced frames are read-only, but could be forwarded (for filters like Trim).

As could be seen, such architecture not only would allow efficient and scalable utilization of hardware threads but will be able to incorporate even extremely threading-unfriendly filters with sequential frame processing without any correctness or speed drawbacks given enough other work in script.

Also it would be very memory-efficient: only actually needed data is kept and memory is consumed as much as needed for no redundant work in the scheduling window.

Cons: hard to implement, scheduling is active and will consume some resources, how filters work needs to be modified: source frames are provided to filter, not requested from inside.


Legacy Avisynth 2.5 filters work through separate filter "wrapper". Threading information is provided to it in textual user-editable file config.
1) As you said, Avisynth right now is "Render->FilterB->FilterA->Source". Multithreading in Avisynth+ is being implemented by letting multiple such (temporally different) chains be evaluated in parallel. I can't say yet how good it works though, as it needs more work I don't have results yet. Can you describe in more detail what advantage your scheduler would provide compared to a traditional threadpool?
2) That is not possible with current Avisynth. To make this work, all existing plugins would need to be modified (though rather trivially). Anyway, if you do that, you can throw out all exisitng plugins until they are changed, or you can use a wrapper layer which throws away the advantages of your system. Btw, VapourSynth does exactly lthis. It lets plugins specify precisely which frames they need for the current job without the thread having to wait for their availability. This is what allows VapourSynth to implement pipelining, which has its advantages, but also drawbacks IMHO.
3) This is not really a new thing. MT-mode is allowed to be provided by filter itself in both VapourSynth and Aisynth+ (I'll write about that soon in another post), and the modes are already established in all projects.
4) This is similar to what Avisynth+ will allow: to have several frames being processed in parallel, all potentially from different stages of the filterchain. And I think this is possible in VapourSynth too, though I don't know enough of it in this respect to be sure.
5) No cache and strict refcounting: The first is bad for performance, the second is not possible without breaking plugins. I'm in the middle of adapting the cache and memory management of Avisynth+ for multithreading right now actually. Out of curiosity I have made various tests and measurements along the way here, like seeing how the filter chain performs without caching. Absolutely miserably. Caching is so important, that without it performance drops to only a fraction of what it was. Strict refcounting (throwing out frames the moment their refcount drops to zero) is not posisble without breaking all 2.5 plugins, because they have the relevant piece of code baked in. So you can modify the refcounting code to drop frames when the count reaches zero, but that part will not execute in 2.5 plugins. The result is that you will leak every single video frame (possibly multiple times, once for each stage), unless you only use 2.6 plugins, otherwise you'll run out of memory in 100-200 frames using a 720p video. I have found that out the hard way, trying to debug said memory leak for more than a day. FYI, this is exactly why I need to rethink my concept of the new caches in Avs+, which leads to MT being delayed somewhat. A possible "solution" to this refcount problem is to recompile all plugins for the 2.6 interface. But there are a lot of plugins out there, and just because you publish recompiles does not mean the users will not keep using old 2.5 plugins. And we don't even have the source for all plugins (even though we do for the most). All in all, I'm not gonna pursue this, and will instead just implement a different architecture that will work with existing plugins.

Last edited by ultim; 6th November 2013 at 19:41.
ultim is offline  
Old 6th November 2013, 21:44   #243  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
1) Scheduler is "advanced" threadpool. Instead of throwing many threads with random caching on hardware and hoping it would work faster, scheduler do exactly the needed work.
2) Like I said, "wrapper" plugin can provide the required additional information, no problem for most filters. In worst case it'll do fail-rerequest route.
5) Cache is absolutely meaningless waste of space if you already can provide all required source frames for the filter call. General refcounting is done by scheduler based on 2, wrapper can do additional refs in response to 2.5 frame refs, but I guess it won't be used much.
SEt is offline  
Old 6th November 2013, 22:14   #244  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Is my understanding correct that you'd let each single filter work on multiple frames in parallel in its isngle thread? And multiple of such superscalar filters would be executing on each HW thread.
ultim is offline  
Old 6th November 2013, 22:21   #245  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by SEt View Post
I doubt anyone it going to implement it, but as you requested, general outline:
...
Since there may be some interest in comparing approaches and ideas...

VapourSynth works kinda 80% like what you describe.

1. There's scheduling, all frame requests are basically put in a queue and processed in order using a thread pool. If a request can't be processed at the moment (the filter instance is already busy) then it simply looks at the next request in line.

2. VapourSynth filters basically work like this. On the first call they request all (or at least most) frames necessary. Then the filter gets called again when they are ready. The procedure can be repeated if necessary.

3. Exactly

4. You get kinda an implicit, cheap version of this in VapourSynth. It combines requests for the same frame (even before looking in caches) so in many cases it ends up working very well. Even temporal filters normally have a very low number of requests to the actual caches.

5. If filters can do unexpected things you need some kind of cache. Of course 80% of the ones automatically inserted will just be wasting their time doing nothing much at all. And remember that users can also do unexpected things such as seek around in vdub. I do however think avisynth wastes obscene amounts of ram for its caching. Frames have strict reference counting.


Avisynth compatibility works like you say, only difference is that the magic "prefetch list" is compiled in. The method of threading for avisynth filters is a single instance running in its own avisynth environment wrapper. So there are always as many "avisynth environments" as there are filters. In combination with the prefetching each filter instance can at most fully use one cpu core (which usually turns out to be enough since stuff like mvtools is a gazillion medium complexity filters stuck together).
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline  
Old 6th November 2013, 23:46   #246  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
ultim
No, superscalar is scheduler. Its execution pipelines are hardware threads, executed "instructions" are filter instances (according to their threading capability) that work on "registers" – frames.
The key points are breaking the call chain and refcounting instead of cache.

Myrsloik
I'm not interested in VapourSynth.
SEt is offline  
Old 7th November 2013, 02:07   #247  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
So in the end it all comes to the "we need to modify all existing plugins" problem. "Nice".
I guess we'll have to set up a huge repo for all the (useful) plugins ever created and start making them less bad some time soon. I just really want to hope I won't be the only one working on this.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 7th November 2013, 12:44   #248  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by TurboPascal7 View Post
So in the end it all comes to the "we need to modify all existing plugins" problem. "Nice".
I guess we'll have to set up a huge repo for all the (useful) plugins ever created and start making them less bad some time soon. I just really want to hope I won't be the only one working on this.
Well this wouldn't be the first thing we could use such a repository for. Making plugins MT-compatible, recompiling for 64-bit, and checking ASM for calling convention violations would also all require a plugin source repository anyway. Not to mention, it would allow us to get rid of plugins with baked-in code, legacy code in Avisynth's core could be removed, and I could probably also implement a more efficient (and less leaky?) caching.

So there's a real bunch of reasons for setting up a repo for plugins, but there are some basic questions to be answered:
  • How would plugin authors react? If they are not interested in commiting to the repos in the future, will we have to port every single change they make?
  • How do we assign plugin maintainership? Obviously we must give authors "VIP" access, and divide the rest of the work between us.
  • What's the destiny of closed-source plugins?
  • Who decides which plugins are "useful"? "Useful" depends on who you ask, and what the plugin is used for. And just because some use cases are rare, it doesn't mean that the corresponding plugins aren't useful.
  • Is it realistic to work on most of the plugins and make all the changes? The changes necessary per plugin are small (unless you have to rewrite asm), but multiplied by the number of plugins...
  • How do we make sure that a user doesn't use legacy plugins mistakenly after that in Avisynth?
  • What happens if we fail for whatever reason? Can we depend on plugins being updated in a reasonable amount of time? Should development directions depending on the outcome be halted or not?
  • Do we have the manpower/time/resolve?

IMHO these are important questions that we should try to answer appropriately before doing anything rash. On the other hand, if we do can handle these question correctly, I'm all for it.

Last edited by ultim; 7th November 2013 at 12:48.
ultim is offline  
Old 7th November 2013, 13:07   #249  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Wow, stop ruining my dreams so fast...

Quote:
Originally Posted by ultim View Post
How would plugin authors react? If they are not interested in commiting to the repos in the future, will we have to port every single change they make?
How do we assign plugin maintainership? Obviously we must give authors "VIP" access, and divide the rest of the work between us.
We do not touch actively maintained plugins. If the author wants to continue using zip packages attached to doom9 posts as his version control system - it's "fine". Our problem is making him "want" to update his plugin for better integration with avs+. The only way we could do it is making sure any updates do not break compatibility with other versions of avisynth. For example asking the author to provide another registration function like AvisynthPluginInit is okay. Asking him to use internal avs+ threadpool is not unless he's really fond of the idea or prepared to maintain a few versions of the plugin.
Quote:
Originally Posted by ultim View Post
What's the destiny of closed-source plugins?
Are there many useful of those? I never actually bothered to check what's closed and what's not.
But well, reverse-engineering is pretty much the only thing we could do about these. Not that you don't have to do the same with some open "source" plugins... Of course anything large and complicated is out of questions.
Quote:
Originally Posted by ultim View Post
Who decides which plugins are "useful"? "Useful" depends on who you ask, and what the plugin is used for. And just because some use cases are rare, it doesn't mean that the corresponding plugins aren't useful.
What users request. I have maybe 15-20 I use the most, other people use other etc. Yes we'll probably end up with basically all plugins ever created. Can't be helped.

Quote:
Originally Posted by ultim View Post
Is it realistic to work on most of the plugins and make all the changes? The changes necessary per plugin are small, but multiplied by the number of plugins...
Simple stuff like recompilation and maybe updating to a newer version of avisynth interface - yes, sure. Rewriting the asm, handling multithreading and making stuff cross-platform - basically impossible unless you find a large dedicated team for that. Hint: you won't.
Quote:
Originally Posted by ultim View Post
How do we make sure that a user doesn't use legacy plugins mistakenly after that in Avisynth?
We can't and we don't. Centralizing plugin distribution is pretty much the only thing we can do, but even then no one can stops the user from installing broken plugins from other sources.
Quote:
Originally Posted by ultim View Post
What happens if we fail for whatever reason? Can we depend on plugins being updated in a reasonable amount of time? Should development directions depending on the outcome be halted or not?
No, most plugins will never be updated or ported, especially in reasonable amount of time. No, development directions should not depend on this and we should find a way to make things work with what we get right now, at the same time trying to make the most use of plugins that do get updated. Yes it won't be efficient "for now" but it's the only choice we have. I might be able to update some useful plugins which I already worked with and a few more, but I'm a human too.

What we can and should do is making the most popular plugins used in literally 100% of scrips better, so most users get the most benefit from our work. But we cannot save the whole avisynth ecosystem.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth

Last edited by TurboPascal7; 7th November 2013 at 13:10.
TurboPascal7 is offline  
Old 7th November 2013, 14:52   #250  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Quote:
Originally Posted by TurboPascal7 View Post
So in the end it all comes to the "we need to modify all existing plugins" problem. "Nice".
Of course interfaces must be changed to support new things. It's absolutely normal considering the state of Avisynth interface today.
Binary compatibility with old 2.5 plugins would still be maintained – wrapper is exactly the thing for that. It would cost some efficiency and will require handwriting threading information, but nothing like recompilation required.

Even with other design plans I still suggest the "wrapper" idea: separate from the core plugin that provides compatibility layer. No need to integrate legacy into the new core.
SEt is offline  
Old 7th November 2013, 19:37   #251  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
So basically, I can't count on any of the interface improvements made in 2.6. "Great." I can always use wrappers for any kind of 3rd, 4th, 5th interface, but that would still mean keeping around old code, except maybe in its own module. Not to mention now I'd even have to make sure the "glue" logic also works. So thanks, but no. In this case I'll rather continue what I was doing sticking to 2.5-style, and devise *one* new interface for Avisynth+ in the future, when the more important stuff are out of the way, and I can make sure that it will finally be alright. I am not in the mood of maintaining 4+ (redundant!) interfaces, and I don't think I ever will be. But to avoid misunderstandings, plugins should of course still use the 2.6 header. There is currently nothing better than that, as it is most feature complete, and does not suffer from the "baked code syndrome".

Quote:
Originally Posted by SEt View Post
... General refcounting is done by scheduler based on 2, wrapper can do additional refs in response to 2.5 frame refs, but I guess it won't be used much.
Refcounting is nice and I would want to use it myself, but I really think I can't because of 2.5 baked code. You can supply the cleanup logic in the users of the frame classes instead of in the frame class, sure, so you could think that might work as long as the core always has the last reference to any frame, instead of the filters. The problem is, how will you know when the refcount reaches zero? The baked-in code won't tell you, so you are left with the choice of either periodic scanning, or with the choice of scanning only when needed, for example on out of memory, which is what current Avisynth does.

You say that we can do refcounting our own way however we want, and provide a wrapper for 2.5 plugins. That means we still have to duplicate the extremely complex old-style Avisynth caching, except in a wrapper, but in addition we must also develop and maintain a second caching system, the primary one?! This is just not worth it, because the old cache basically works. Yes it is way too complex for what it does. Yes it eats memory. But it works and it is not a performance bottleneck. So what reason is there to develop a completely new one if we still have to continue living with all the drawbacks of the current cache, except in a wrapper?

Last edited by ultim; 7th November 2013 at 19:40.
ultim is offline  
Old 8th November 2013, 17:41   #252  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
2.6 interfaces are useless – they solved nothing while breaking binary compatibility, usage is very low. I'd say implement only 2.5 compatibility.
New interface obviously needs to be the only one and extensible. Btw, this implies that it must be C style and not the C++ hell like now. For exact design I'd say COM-like: great scalability and linking compatibility.

As for wrapper complexity – it greatly depends on which and how efficient you want it to support quirks of 2.5. Sure, maybe 2.5 plugin that formatted your hard drive would require total emulation to work correctly, but do we care? I suspect that quite simple wrapper + some special cases would be ok for most interesting plugins.

And FTY, caching is the main problem of Avisynth MT. Not threading hacks, not 32-bitness how quite a few people think. And it is performance bottleneck.
SEt is offline  
Old 8th November 2013, 19:07   #253  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by SEt View Post
2.6 interfaces are useless – they solved nothing while breaking binary compatibility, usage is very low. I'd say implement only 2.5 compatibility.
New interface obviously needs to be the only one and extensible. Btw, this implies that it must be C style and not the C++ hell like now. For exact design I'd say COM-like: great scalability and linking compatibility.

As for wrapper complexity – it greatly depends on which and how efficient you want it to support quirks of 2.5. Sure, maybe 2.5 plugin that formatted your hard drive would require total emulation to work correctly, but do we care? I suspect that quite simple wrapper + some special cases would be ok for most interesting plugins.

And FTY, caching is the main problem of Avisynth MT. Not threading hacks, not 32-bitness how quite a few people think. And it is performance bottleneck.
Well I'm rewriting caching and memory management from scratch anyway. Coz I agree that current cache doesn't lend itselft to MT naturally. In single-threaded mode though it didn't seem like a bottleneck, as it didn't even show up in profiling results.

I also agree that new interface needs to be C-style. My plans are to make it object oriented (as far as C allows), but keep objects totally opaque. An extremely thin C++ wrapper shipped as public include files will provide C++ binding, but it will only provide syntax sugar and some help in memory management thanks to destructors.
ultim is offline  
Old 9th November 2013, 06:03   #254  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,646
Quote:
Originally Posted by SEt View Post
That have been said, I'm busy with other areas atm. (And you will likely see something useful for Avisynth in several days. )
What is it?
ryrynz is offline  
Old 11th November 2013, 14:32   #255  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Status update

You have probably noticed that this release of Avisynth+ is taking longer than previous ones. The good news is that not because we are stalling, but because the features we are working on are a lot larger, and also a lot more complex.

Other good news is that people have started joining in on the project. TurboPascal7 and innocenat have been working on bringing you full-fledged 64-bit functionality, qyot27 has already sent a pull request to give you a nice automated installer, and I myself am introducing multi-threaded processing.

So, look forward to the next release!

Last edited by ultim; 11th November 2013 at 14:46.
ultim is offline  
Old 12th November 2013, 06:37   #256  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
Looking forward to 64 bit and multithreading. A few questions:
1. Will the installer be able to extract the dll's with 7zip?
2. Will the dll package still be available?
3. At some point in the future, would JincResize (from madvr) be considered?
__________________
PC: FX-8320 GTS250 HTPC: G1610 GTX650
PotPlayer/MPC-BE LAVFilters MadVR-Bicubic75AR/Lanczos4AR/Lanczos4AR LumaSharpen -Strength0.9-Pattern3-Clamp0.1-OffsetBias2.0
turbojet is offline  
Old 12th November 2013, 06:52   #257  |  Link
innocenat
Registered User
 
innocenat's Avatar
 
Join Date: Dec 2011
Posts: 77
Quote:
Originally Posted by turbojet View Post
3. At some point in the future, would JincResize (from madvr) be considered?
I have been considering adding Jinc support (as I also want them), but I have trouble finding reference to this filter. I'd be grateful if someone could link a nice explanation of how this filter works.
innocenat is offline  
Old 12th November 2013, 07:16   #258  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
There was a lot of talk about it in the madvr thread last year. I believe it's based on: http://www.imagemagick.org/discourse...p?f=22&t=21415 there's some code on page 2 and maybe more in later pages. This might be part of it: http://forum.doom9.org/showthread.ph...id#post1595043

But only madshi and maybe NicolasRobidoux would really know.
__________________
PC: FX-8320 GTS250 HTPC: G1610 GTX650
PotPlayer/MPC-BE LAVFilters MadVR-Bicubic75AR/Lanczos4AR/Lanczos4AR LumaSharpen -Strength0.9-Pattern3-Clamp0.1-OffsetBias2.0
turbojet is offline  
Old 12th November 2013, 08:36   #259  |  Link
Keiyakusha
契約者
 
Keiyakusha's Avatar
 
Join Date: Jun 2008
Posts: 1,576
For Jinc, you also need madshi's anti-ringing, it sux without it.
Also if someone is going to implement it, it will be cool if needed calculations can be done in GPU. Unless it'll be comparable to spline36 by speed. Cause if it'll be slow, one could just go with nnedi/eedi+subpixel shift(if desired) and get better results. And Jinc is also not something you're going to use for downscale.

It could be way simpler if madshi could just turn that part of his renderer into a plugin (even if closed-source).
Keiyakusha is offline  
Old 12th November 2013, 08:40   #260  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Moving stuff to GPU and back is not worth it if resizing is the only thing you're going to do. So no, this is not going to work.
Also, fmtconv might get ported to avisynth some time in the future. We might as well wait for it before spending our time on something that might end up replaced by fmtconv.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.