Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd November 2016, 01:18   #2541  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,057
Quote:
Originally Posted by pinterf View Post
Does Avisynth have problems with strings?
What kind of problem(s) exactly? tp7 mentioned the probable need to add something better in the core to deal with string formatting and concatenation a couple years ago, but that may not be the same thing you're referring to.
qyot27 is offline   Reply With Quote
Old 2nd November 2016, 02:52   #2542  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,010
I was referring to memory usage, ie not freeing of temporary strings, strings used in eg any single instance of a Scriptclip frame
are not released until Avisynth closure, despite them never being used again. We should be able to automatically release local strings that are
created in any script function and are not returned as the function result. I dont know if memory used for other non string local variables
additionally are not released until closure.
EDIT: Plugin writers can rely upon strings returned via SaveString() to be still available later on, so to be able to release strings
allocated via plugins, there must be another function akin to SaveString() that hands over total ownership to Avisynth and such
memory cannot be later accessed by the plugin. The original SaveString(), copies the string into Avisnth allocated memory.
How to return Strings (I'm guessin that you dont need this, perhaps some do):- http://forum.doom9.org/showthread.ph...ng#post1633936


Creating an expanded system environment string + file is not I think such a big problem, (does not need to be implemented in core,
although GetSystemEnv() could be a good idea).

Code:
AVSValue __cdecl RT_GetSystemEnv(AVSValue args, void* user_data, IScriptEnvironment* env) {
	// char * myName="RT_GetSystemEnv: ";
	const char *s=args[0].AsString();
	char * sysenv = getenv(s);
	if(sysenv==NULL)
		sysenv="";
	return env->SaveString(sysenv);
}
fn=RT_GetSystemEnv("HOMEPATH") + "\MyFile.xyz" although that aint exactly bullet proof.


this is string formatting as implemented in RT_Stats RT_string

Code:
AVSValue __cdecl RT_String(AVSValue args, void* user_data, IScriptEnvironment* env) {
    char *myName="RT_String: ";
    const char *s       =   args[0].AsString();             // text
    int arrsz           =   args[1].ArraySize();
    int esc             =   args[2].AsInt(1);

    if(esc < 0 || esc > 2)
        env->ThrowError("%sEsc range 0 -> 2 only",myName);

    enum {
        CHICKEN=64
    };

    // what size buffer we need ?
    int i,mem=strlen(s) + 1 + CHICKEN;
    for(i=0;i<arrsz;++i) {
        if(args[1][i].IsString()) {
            const char *st=args[1][i].AsString();
            mem += strlen(st) + 1 + CHICKEN;
        } else {
            mem += 8 + CHICKEN;     // no particular reason why so big, just chicken factor.
        }
    }

    char *pbuf = new char[(mem+1)*2];
    if(pbuf==NULL)
        env->ThrowError("%sCannot allocate memory",myName);

    char *ptem=pbuf+(mem+1);            // temp buffer

    const unsigned char * r= (const unsigned char *)s;
    char *p=pbuf;
    int  c,ix=0;
    int t=0;

    // Parse text and insert variables
    while(c=*r) {
        if(c=='%') {
            ++r;
            if(*r=='\0') {
                *p++ ='%';
            } else if(*r=='%') {
                *p++=*r++;              // replace escaped double % with single
            } else {
                if(ix>=arrsz) {
                    delete [] pbuf;
                    env->ThrowError("%sExpecting data arg (%d)",myName,ix+1);
                }
                char *tp=ptem;
                *tp++='%';
                if(*r=='-' || *r=='+' || *r=='0' || *r==' ' || *r=='#')     // flags
                    *tp++=*r++;
                if(*r=='*') {                                       // int holds length
                    t=args[1][ix].IsBool()          ?1: \
                            args[1][ix].IsString()  ?2: \
                            args[1][ix].IsInt()     ?3: \
                            args[1][ix].IsFloat()   ?4: \
                            0;
                    if(t!=3)    {
                        delete [] pbuf;
                        env->ThrowError("%sUnsupported data type, Expecting Width as Int (%d)",myName,ix+1);
                    }
                    tp+=sprintf(tp,"%d",args[1][ix].IsInt());
                    ++r;                                            // skip '*'
                    ++ix;                                           // next data
                } else {
                    while(*r>='0' && *r<='9') {
                        *tp++ = *r++;
                    }
                }
                if(*r=='.') {
                    *tp++ =  *r++;                                      // precision prefix
                    if(*r=='*') {                                       // int holds length
                        t=args[1][ix].IsBool()          ?1: \
                                args[1][ix].IsString()  ?2: \
                                args[1][ix].IsInt()     ?3: \
                                args[1][ix].IsFloat()   ?4: \
                                0;
                        if(t!=3) {
                            delete [] pbuf;
                            env->ThrowError("%sUnsupported data type, Expecting Precision as Int (%d)",myName,ix+1);
                        }
                        tp+=sprintf(tp,"%d",args[1][ix].AsInt());
                        ++r;                                            // skip '*'
                        ++ix;                                           // next data
                    } else {
                        while(*r>='0' && *r<='9') {
                            *tp++ = *r++;
                        }
                    }
                }
                t=args[1][ix].IsBool()          ?1: \
                        args[1][ix].IsString()  ?2: \
                        args[1][ix].IsInt()     ?3: \
                        args[1][ix].IsFloat()   ?4: \
                        0;
                // type
                if( (*r=='c' ) || (*r=='C' ) ||                                         // char as int
                    (*r=='d' || *r=='i') ||                                             // int
                    (*r=='o' || *r=='u' || *r=='x' || *r=='X'))  {                      // unsigned int
                    if(t!=3)    {
                        delete [] pbuf;
                        env->ThrowError("%sType='%c', Expecting Int data (%d)",myName,*r,ix+1);
                    }
                    *tp++=*r++;
                    *tp='\0';
                    p+=sprintf(p,ptem,args[1][ix].AsInt());
                    ++ix;                                                               // next data
                } else if(*r=='e' || *r=='E' || *r=='f' || *r=='g' || *r=='G') {        // double
                    if(t!=4&&t!=3)  {
                        delete [] pbuf;
                        env->ThrowError("%sType='%c', Expecting Float (%d)",myName,*r,ix+1);
                    }
                    *tp++=*r++;
                    *tp='\0';
                    p+=sprintf(p,ptem,args[1][ix].AsFloat());
                    ++ix;                                                               // next data
                } else if((*r=='s')||(*r=='S')) {                                                   // string
                    if(t!=2&&t!=1)  {
                        delete [] pbuf;
                        env->ThrowError("%sType='s', Expecting String (%d)",myName,ix+1);
                    }
                    *tp++=*r++;
                    *tp='\0';
                    if(t==1) {  // Bool
                        p+=sprintf(p,ptem,args[1][ix].AsBool()?"True":"False");
                    } else {    // String
                        p+=sprintf(p,ptem,args[1][ix].AsString());
                    }
                    ++ix;                                                               // next data
                } else {
                    delete [] pbuf;
                    env->ThrowError("%sUnknown format type '%c' (%d)",myName,*r,ix+1);
                }
            }
        } else if(c == '\\' && esc == 1) {
            ++r;
            c=*r;
            // abfnrtv
            switch (c) {
            case '\0' : *p++='\\';              break;      // copy single backslash at end of string
            case '\\' : *p++=*r++;              break;      // replace double backslash with single backslash
            case 'n' :  ++r;        *p++='\n';  break;
            case 'r' :  ++r;        *p++='\r';  break;
            case 't' :  ++r;        *p++='\t';  break;
            case 'v' :  ++r;        *p++='\v';  break;
            case 'f' :  ++r;        *p++='\f';  break;
            case 'b' :  ++r;        *p++='\b';  break;
            case 'a' :  ++r;        *p++='\a';  break;
            default :   *p++='\\';  *p++=*r++;  break;      // anything else we copy backslash and whatever follows
            }
        } else {
            *p++=*r++;
        }
    }
    *p=0;                                           // nul term

    if(ix<arrsz) {
        delete [] pbuf;
        env->ThrowError("%sUnexpected data arg (%d)",myName,ix+1);
    }

//  dprintf("RT_String: MEM Estimate=%d : Actual=%d\n",mem+1,strlen(pbuf));

    if(esc==2) {
        p=pbuf;
        r=(const unsigned char*)p;
        while(c=*r) {
            if(c=='\\') {
                ++r;
                c=*r;
                // abfnrtv
                switch (c) {
                case '\0' : *p++='\\';              break;      // copy single backslash at end of string
                case '\\' : *p++=*r++;              break;      // replace double backslash with single backslash
                case 'n' :  ++r;        *p++='\n';  break;
                case 'r' :  ++r;        *p++='\r';  break;
                case 't' :  ++r;        *p++='\t';  break;
                case 'v' :  ++r;        *p++='\v';  break;
                case 'f' :  ++r;        *p++='\f';  break;
                case 'b' :  ++r;        *p++='\b';  break;
                case 'a' :  ++r;        *p++='\a';  break;
                default :   *p++='\\';  *p++=*r++;  break;      // anything else we copy backslash and whatever follows
                }
            } else {
                *p++=*r++;
            }
        }
        *p=0;                                           // nul term
    }
    AVSValue ret = env->SaveString(pbuf,p-pbuf);
    delete [] pbuf;
    return  ret;
Much better than the basic Avisynth String() function.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 2nd November 2016 at 03:37.
StainlessS is online now   Reply With Quote
Old 2nd November 2016, 11:23   #2543  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
Quote:
Originally Posted by Gavino View Post
What about arrays as parameters to user-written functions?
This would require introduction of a new type-identifier (eg 'array') to rank alongside 'clip', 'int', 'float', etc in the function definition.

Overall, your idea seems like a good one, but careful thought (which I haven't yet had time for) is required to ensure that it doesn't break anything or introduce language inconsistencies of some kind.
Yes, there were heavy side-effects.

A couple of hours I went on a bad direction.

First I was thinking about using an existing feature: unnamed function parameters already accepted array-like parameter values, by the '+' specifier, e.g. in StackHorizontal, Spline or Select.

Code:
  { "StackHorizontal", BUILTIN_FUNC_PREFIX, "cc+", StackHorizontal::Create },
  { "Spline", BUILTIN_FUNC_PREFIX, "[x]ff+[cubic]b", Spline },
  { "Select",   BUILTIN_FUNC_PREFIX, "i.+", Select },
But it is only for one level.
And the broader support in AVSValue mechanism (constructor, etc.) failed. When I wanted to return such an array value, e.g. store it in a variable, the content of the array disappeared outside the internal method, because after instancing a function or method the array member AVSValues were freed up.

Another problem was that the existing array parameter type has special handling internally, the array content is "flattened" at some point.

So I had to introduce a "real" script array type, now I call it "A" in order not to mix it with the existing 'a' internal array specifier.
It is a vector<AVSValue>

Because of the working mechanism of '+' parameter specifier, which is putting together a one-level array AVSValue (type=='a') for the functions, I introduced a # (since it resembles the + sign) specifier that results a new script-array (type=='A') instead of that.

In my sample, BlankClip can have a real array, that can be used for a named parameter. The definition is like this (A= the new script array type)
Code:
  { "BlankClip", BUILTIN_FUNC_PREFIX, "[]c*[length]i[width]i[height]i[pixel_type]s[fps]f[fps_denominator]i[audio_rate]i[channels]i[sample_type]s[color]i[color_yuv]i[clip]c[colors]A", Create_BlankClip },
and the other array specific functions are defined as
Code:
  { "Array", BUILTIN_FUNC_PREFIX, ".#", ArrayCreate },  // # instead of +: creates script array
  { "IsArray",   BUILTIN_FUNC_PREFIX, ".", IsArray2 },
  { "ArrayGet",  BUILTIN_FUNC_PREFIX, "Ai", ArrayGet },
  { "ArrayGet",  BUILTIN_FUNC_PREFIX, "As", ArrayGet },
  { "ArraySize", BUILTIN_FUNC_PREFIX, "A", ArraySize },
It works as-is, parameter type is nicely checked against type 'A'.
Script function can check the script-array type with IsArray2 at the moment.
At least I had a little insight to the parser, too. Anyhow, this part was nicely written, I hope I can easily change to [] for definition and indexing arrays.

A little off-topic, but I relaxed with this problem after my weekend 85K trail running race Fun after fun.

BTW did you know that there is built-in for and while?
pinterf is offline   Reply With Quote
Old 2nd November 2016, 12:20   #2544  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,380
Quote:
Originally Posted by pinterf View Post
Code:
  { "Array", BUILTIN_FUNC_PREFIX, ".#", ArrayCreate },  // # instead of +: creates script array
Perhaps # should be allowed to also match zero or more values (like the existing * rather than +), so that you can create empty arrays. Or maybe you want another new symbol in addition to #, to be the equivalent of * for arrays.

Actually, on further thought, I'm not sure you really need # anyway.
Can't the built-in Array() function be defined as ".*", and simply copy its arguments into an new type 'A' AVSValue?

Quote:
BTW did you know that there is built-in for and while?
Yes (in Avisynth+).
In the early stages of development, I gave ultim permission to incorporate my GScript code into Avisynth+.
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 3rd November 2016, 07:52   #2545  |  Link
may24
Registered User
 
Join Date: Jan 2008
Posts: 41
Hi all,

I recently swapped avisynth for avisynth+

Yesterday I modified one of my avs Scripts. Just added some new pics - basically three more ImageSource(...)

But now I got: "Could not map shared memory into local space" in VirtualDub.

Any ideas ?

EDIT: I fall back to std. Avisynth 2.6.1 ... no problems anymore ...
may24 is offline   Reply With Quote
Old 3rd November 2016, 11:49   #2546  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
Quote:
Originally Posted by may24 View Post
Hi all,

I recently swapped avisynth for avisynth+

Yesterday I modified one of my avs Scripts. Just added some new pics - basically three more ImageSource(...)

But now I got: "Could not map shared memory into local space" in VirtualDub.

Any ideas ?

EDIT: I fall back to std. Avisynth 2.6.1 ... no problems anymore ...
Anyway, needed the
- avs script
- whether the script works alone w/o vdub (to rule out avs+ install problems, such as missing VS2015 update 3 redistributables)
- type and size of the images
- virtualdub version
- avisynth+ version
pinterf is offline   Reply With Quote
Old 3rd November 2016, 12:07   #2547  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,895
Quote:
Originally Posted by pinterf View Post
- whether the script works alone w/o vdub (to rule out avs+ install problems, such as missing VS2015 update 3 redistributables)
AviSynth scripts (no matter which fork of AviSynth) will always need a host application. The minimum is probably AVSMeter (which also may print more useful error messages). In case of AviSynth+ as engine, may24 can (and should!) test both the 32 bit and 64 bit version of AVSMeter as host.

If I understood may24 correctly, the script worked well with a few images less loaded, but he crossed some threshold by adding some more sources?
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 3rd November 2016, 12:16   #2548  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
Quote:
Originally Posted by LigH View Post
AviSynth scripts (no matter which fork of AviSynth) will always need a host application. The minimum is probably AVSMeter (which also may print more useful error messages). In case of AviSynth+ as engine, may24 can (and should!) test both the 32 bit and 64 bit version of AVSMeter as host.

If I understood may24 correctly, the script worked well with a few images less loaded, but he crossed some threshold by adding some more sources?
AVSMeter or x264 was in my mind (other than virtualdub).
I found that error message and all I could see that a MapViewOfFile after a successful CreateFileMapping has failed with a specific buffer size.
pinterf is offline   Reply With Quote
Old 3rd November 2016, 13:24   #2549  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,745
@pinterf :
Where is _MM_PACKUS_EPI32 defined ?
When i tried to build, the compiler complained that it can't find it.
I've not been able to find it either, at leat in the resample source files.
jpsdr is offline   Reply With Quote
Old 3rd November 2016, 15:09   #2550  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
Quote:
Originally Posted by jpsdr View Post
@pinterf :
Where is _MM_PACKUS_EPI32 defined ?
When i tried to build, the compiler complained that it can't find it.
I've not been able to find it either, at leat in the resample source files.
internal.h, it was used at several places
pinterf is offline   Reply With Quote
Old 3rd November 2016, 15:18   #2551  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,745
Just when i finaly found it...
Thanks anyway.
Now, i have to figure out why... why... why... the compiler don't want to compile minmax.h anymore, throwing me a lot of errors about the template T !
And i have to implement a test for detecting VS version, because i still want to be able to compile with VS2010, which the avx intrinsic don't allow anymore...
But first, i have to be able to build with VS2015. I can workaround minmax with #define macros, but i would better understand why sudenly it don't want to build anymore...

Last edited by jpsdr; 3rd November 2016 at 15:22.
jpsdr is offline   Reply With Quote
Old 3rd November 2016, 15:25   #2552  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
Quote:
Originally Posted by jpsdr View Post
Just when i finaly found it...
Thanks anyway. Now, i have to figure out why... why... why... the compiler don't want to compile minmax.h anymore, throwing me a lot of errors about the template T !
The types inside min, max or std::clamp are not similar. All parameters have to be the same int, short or float (or pixel_t in templates).

PS, I have cleaned up the resizers a bit (after you found unused avx path test code) and added a real avx2 path, I bet it is still not as fast as zimg, but at least it got cleaner. Just have commited, check my MT repo.
pinterf is offline   Reply With Quote
Old 3rd November 2016, 15:55   #2553  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,745
Quote:
Originally Posted by pinterf View Post
The types inside min, max or std::clamp are not similar. All parameters have to be the same int, short or float (or pixel_t in templates).
The problem is that i haven't touched the code where minmax is used... So for now, i have absolutely no idea why there is suddenly this error.
Quote:
Originally Posted by pinterf View Post
PS, I have cleaned up the resizers a bit (after you found unused avx path test code) and added a real avx2 path, I bet it is still not as fast as zimg, but at least it got cleaner. Just have commited, check my MT repo.
Argh....... After all that hard work... Nooooooooooooooooooooooooo...

zimg... If it works with direct full square ... (argh... don't remember the word now), it will be impossible to multi-thread by splitting.

And a more general question.
In image/video processing, the 1rst idea (at least, for me) which comes to mind when talking of multi-threading is splitting the picture, not multiply it.
So, my question is : Why when introducing multi-threading in avisynth, nobody seems to have thought at the 1rst obvious idea, internaly MT the core filters by splitting the picture,
instead of trying to run several GetFrame at the same time...?
Filters that are pure memory transfert (like doubleweave, or TurnL/R) may not benefit of MT (there is more chances of memory bandwith ... again don't remember the correct word ... issue,struggle), but the others may.
So... Why...?

Last edited by jpsdr; 3rd November 2016 at 16:08.
jpsdr is offline   Reply With Quote
Old 3rd November 2016, 16:38   #2554  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
8 bit was not affected, and the changes are not huge. Avx and avx2 is basically the same, only their templates are different you dont have to look for other differences
pinterf is offline   Reply With Quote
Old 3rd November 2016, 16:40   #2555  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,745
Ok, thanks.
jpsdr is offline   Reply With Quote
Old 3rd November 2016, 18:39   #2556  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,745
The minmax.h issue is solved : I must remove the #include within resample.cpp otherwise compiler throws me errors.
jpsdr is offline   Reply With Quote
Old 3rd November 2016, 21:02   #2557  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,071
No, jpsdr, you didn't suddenly come up with the obvious solution that nobody else has considered. There are reasons nobody uses slice based threading. The most obvious one is that if you're working with video you already have your input split up in neat discrete chunks for you: they're called frames. Splitting these chunks further just adds complexity for no real gain at all. I'm pretty sure it's also worse performance wise (for many reasons, especially for spatial filters where you might have to handle overlapping slices) but I haven't benchmarked it.
TheFluff is offline   Reply With Quote
Old 3rd November 2016, 22:55   #2558  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,895
Even worse, slices will limit the motion vector search for redundancies the codec could have used to increase efficiency.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 4th November 2016, 00:07   #2559  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 782
Quote:
Originally Posted by pinterf View Post
Code:
ArrayGet(test_array,0)
Opinions?
Why not
Code:
test_array(0)
?
ajp_anton is offline   Reply With Quote
Old 4th November 2016, 01:27   #2560  |  Link
Stephen R. Savage
Registered User
 
Stephen R. Savage's Avatar
 
Join Date: Nov 2009
Posts: 341
Quote:
Originally Posted by ajp_anton View Post
Why not
Code:
test_array(0)
?
Wouldn't that conflict with a function named test_array? AVS does not prevent creating variables with the same name as functions.
Stephen R. Savage is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:14.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.