Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 1st November 2017, 08:52   #3701  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,853
Quote:
Originally Posted by TheFluff View Post
Continuing the discussion from here, let's talk about those nasty memory offsets again.


In Avs+ all these size_t's are (32-bit signed) int instead. In Avisynth it doesn't really matter in practice because a frame is always allocated as one big chunk of memory, and nobody really has any use for more than 2^31 bytes of vfb. If you ever wanted to stop doing this though and use one individual pointer per plane like VS does, this won't fly because you can't reliably stuff a 64-bit pointer in a 32-bit integer, and it's generally bad coding practice to not use size_t (or ptrdiff_t) for these things.

Way back in the day ultim claimed that he didn't want to follow IanB's lead and switch to size_t because it'd break a number of completely irrelevant plugins (that would have been trivial to recompile). I didn't realize it at the time, but I'm pretty sure that in practice, he was wrong even on the technical aspect. If you look at these functions above, the only one that conceivably might end up getting called by plugins in reality is VideoFrame::GetOffset(), and I'm 99% sure that its return value just gets passed in a register and zero-extended, so as long as you keep passing the same old less-than-2^31 offsets, everything will keep working just fine (but most plugins just use GetRead/WritePtr). New VideoFrames and VFB's are only constructed by env->NewVideoFrame, so changing the signature of the VideoFrame constructor shouldn't be an issue.
Thanks, then it's worth a try.
pinterf is offline  
Old 1st November 2017, 11:06   #3702  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,293
Since we're on the subject of changes. Why was the return type of GetCPUFlags() changed from long to int? That one makes absolutely no sense to me.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline  
Old 6th November 2017, 12:38   #3703  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,853
Quote:
Originally Posted by Myrsloik View Post
Since we're on the subject of changes. Why was the return type of GetCPUFlags() changed from long to int? That one makes absolutely no sense to me.
Another piece of history:
"Standardize some type usage to prevent confusion of devs with GCC background."
https://github.com/AviSynth/AviSynth...dd60d203ea9504
pinterf is offline  
Old 6th November 2017, 12:42   #3704  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,853
Quote:
Originally Posted by enctac View Post
r2508 RGB->Y8 [incorrect]
r2508 RGB->YV24->Y8 [correct]
2.6MT RGB->Y8 [correct]
Thanks for the report, fixed. Rec601, Rec709 (limited range) RGB->Y conversion was affected. Regression since r2266 (a lot of high bit depth work at that time)
pinterf is offline  
Old 6th November 2017, 13:26   #3705  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,853
Quote:
Originally Posted by pinterf View Post
Quote:
Originally Posted by TheFluff View Post
Continuing the discussion from here, let's talk about those nasty memory offsets again.


In Avs+ all these size_t's are (32-bit signed) int instead. In Avisynth it doesn't really matter in practice because a frame is always allocated as one big chunk of memory, and nobody really has any use for more than 2^31 bytes of vfb. If you ever wanted to stop doing this though and use one individual pointer per plane like VS does, this won't fly because you can't reliably stuff a 64-bit pointer in a 32-bit integer, and it's generally bad coding practice to not use size_t (or ptrdiff_t) for these things.

Way back in the day ultim claimed that he didn't want to follow IanB's lead and switch to size_t because it'd break a number of completely irrelevant plugins (that would have been trivial to recompile). I didn't realize it at the time, but I'm pretty sure that in practice, he was wrong even on the technical aspect. If you look at these functions above, the only one that conceivably might end up getting called by plugins in reality is VideoFrame::GetOffset(), and I'm 99% sure that its return value just gets passed in a register and zero-extended, so as long as you keep passing the same old less-than-2^31 offsets, everything will keep working just fine (but most plugins just use GetRead/WritePtr). New VideoFrames and VFB's are only constructed by env->NewVideoFrame, so changing the signature of the VideoFrame constructor shouldn't be an issue.
Thanks, then it's worth a try.
Turned out that programs using C interface could be broken.

Current x264 is one of such programs, it fails, because it reads zero pitches (strides).
Code:
x264 [error]: Input picture width (640) is greater than stride (0)
x264 [error]: x264_encoder_encode failed
Having a look at the x264 source code, it contains a hybride avisynth_c.h (I guess that it is a mix of some previous headers from the "classic" avs line, inserted new high bit depth stuff from current avs+) in which the function avs_get_read_ptr_p is inlined and directly accesses the fields of AVS_VideoFrame struct
It fails because when we change the type of offset-like variables from int (32 bits) to size_t (64 bits on x64), the positions of fields in AVS_VideoFrame are shifted and a "baked" reference to the "pitch" now becomes the higher 32 bits of "offset", which is usually zero.

And that's only for x264. Anyway, when I'll make a test build, I will provide this "offsets are size_t instead of int" version as a separate test version.

If it could be changed it may work for current (int) and future (size_t) version of avisynth. Maybe it was inlined purposely, because older avisynth versions did not support avs_get_read_ptr_p through C interface (did not have time to dig into its history)?

To tell the truth, I had a look at the current avisynth_c.h from avs+ project, and although this specific call is O.K. in current version, there are two other problematic functions, namely
avs_get_row_size and avs_get_height (the ones without a "plane" parameter) so they apply on plane 0 (PLANAR_Y).

These functions are still directly accessing the AVS_VideoFrame contrary to the big warning "DO NOT USE THIS STRUCTURE DIRECTLY", so c interface header in avs+ is still inconsistent, at least for AVS_VideoFrame.
pinterf is offline  
Old 6th November 2017, 14:10   #3706  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,099
Ah, I hadn't considered that. Tricky.
TheFluff is offline  
Old 7th November 2017, 02:06   #3707  |  Link
Shirtfull
Registered User
 
Join Date: May 2007
Posts: 53
Quote:
Originally Posted by pinterf View Post
Thanks for the report, fixed. Rec601, Rec709 (limited range) RGB->Y conversion was affected. Regression since r2266 (a lot of high bit depth work at that time)
Rgb24 to yv12 seems incorrect as well, Frame served Progressive/interlaced testcube RGB24 from Virtualdub/VirtualDub_FilterMod to AvsPmod.

Rgb>yv12



Rgb>yv16

Shirtfull is offline  
Old 7th November 2017, 08:45   #3708  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,853
Quote:
Originally Posted by Shirtfull View Post
Rgb24 to yv12 seems incorrect as well, Frame served Progressive/interlaced testcube RGB24 from Virtualdub/VirtualDub_FilterMod to AvsPmod.

Rgb>yv12

Rgb>yv16
I can see something like a rotation blur on the yv12 sample. How did you achieve that exactly?
pinterf is offline  
Old 7th November 2017, 12:10   #3709  |  Link
Shirtfull
Registered User
 
Join Date: May 2007
Posts: 53
Code:
SetFilterMTMode("AVISource", x)  #2or3
AVISource("Path to frameserve file.avi").ConvertToYv12() #12 or 16
bob()
Prefetch (4)
Only difference between pics was the conversion.

Take out bob, they both look the same.
Shirtfull is offline  
Old 7th November 2017, 13:09   #3710  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,853
Quote:
Originally Posted by Shirtfull View Post
Code:
SetFilterMTMode("AVISource", x)  #2or3
AVISource("Path to frameserve file.avi").ConvertToYv12() #12 or 16
bob()
Prefetch (4)
Only difference between pics was the conversion.

Take out bob, they both look the same.
Specify interlaced=true in ConvertToYV12.
RGB->YV12 conversion is a two-phase conversion. First comes RGB->YV24, then YV24->YV12. This latter needs the hint that the clip is an interlaced one.
(remark: parameter "interlaced" is used only in conversions where yv12 is involved either as source or target)
pinterf is offline  
Old 7th November 2017, 13:39   #3711  |  Link
Shirtfull
Registered User
 
Join Date: May 2007
Posts: 53
Thanks, that fixed it.
Shirtfull is offline  
Old 7th November 2017, 16:27   #3712  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,145
Quote:
Originally Posted by Shirtfull View Post
Code:
SetFilterMTMode("AVISource", x)  #2or3
AVISource("Path to frameserve file.avi").ConvertToYv12() #12 or 16
bob()
Prefetch (4)
Only difference between pics was the conversion.

Take out bob, they both look the same.
Anyone know why it looks like some kind of rotation with the bob() in there? Seems weird.
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline  
Old 7th November 2017, 17:05   #3713  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 766
Quote:
Originally Posted by davidhorman View Post
Anyone know why it looks like some kind of rotation with the bob() in there? Seems weird.
I think VD does not have way to request interlaced conversion to YV12.
Looks better:
deinterlace (unfold)
convert format (yv12)
deinterlace (fold)
shekh is offline  
Old 8th November 2017, 16:50   #3714  |  Link
Shirtfull
Registered User
 
Join Date: May 2007
Posts: 53
The other way round, using VDFiltermod to generate cube and then frame-serving to avisynth. I recall it can only serve RGB.

Last edited by Shirtfull; 8th November 2017 at 16:54. Reason: Typo
Shirtfull is offline  
Old 8th November 2017, 22:45   #3715  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,439
It appears that Prefetch cannot be used in a script which uses Return.

Here is my test. The following script runs at the same speed with, or without, the Prefetch statement. It is clear that multi-threading is NOT being used.

Code:
LoadPlugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\Film Restoration\Script_and_Plugins\RemoveGrainSSE2.dll")
source=AVISource("E:\fs.avi").killaudio().ConvertToYV12()
output=MDegrain2i2(source,8,4,400,0)  
return output
Prefetch(5)
If I move Prefetch to the line before the "return output" statement, the script throws an error, "Invalid arguments to function 'Prefetch' ".

If I re-code so the script ends with the implied "last," as shown below, I get a 4x speedup (i.e., multi-threading is working), and don't get an error message. In other words, this works. However, it is a PITA to have to re-write scripts to avoid the return statement because I often want to do comparisons between the initial and final states of a denoising script, so it is useful to have a final "output" variable.

Is there a way to get Prefetch to work with Return, or I have I just found the only way?

Code:
LoadPlugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\Film Restoration\Script_and_Plugins\RemoveGrainSSE2.dll")
source=AVISource("E:\fs.avi").killaudio().ConvertToYV12()
MDegrain2i2(source,8,4,400,0)  
Prefetch(5)
johnmeyer is offline  
Old 9th November 2017, 00:19   #3716  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 4,629
Quote:
Originally Posted by johnmeyer View Post

Is there a way to get Prefetch to work with Return, or I have I just found the only way?
Don't use "return" , just call the variable directly

eg.

Code:
source = whateversource()
A = source().filter1()
B = source().filter2()

#source
A
#B
prefetch(5)
If you wanted "B" , comment out "A" and uncomment out B. If you wanted "source" - same idea, uncomment out source, comment out A and B
poisondeathray is offline  
Old 9th November 2017, 00:28   #3717  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,439
Quote:
Originally Posted by poisondeathray View Post
Don't use "return" , just call the variable directly

eg.

Code:
source = whateversource()
A = source().filter1()
B = source().filter2()

#source
A
#B
prefetch(5)
If you wanted "B" , comment out "A" and uncomment out B. If you wanted "source" - same idea, uncomment out source, comment out A and B
I just learned something: I thought I had to use "return." I need to re-read the AVISynth doc. Thanks!
johnmeyer is offline  
Old 9th November 2017, 00:30   #3718  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,099
Code:
return output.prefetch(5)
may or may not work
TheFluff is offline  
Old 9th November 2017, 00:44   #3719  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,399
AviSynth (plus as well as legacy) internally uses a clip variable "last" where an explicit assignment is omitted (and an implicit "return last" where any return was omitted). Explicitly, this would act like:

Code:
source = whateversource()
A = source().filter1()
B = source().filter2()

#source
last = A
#last = B
last.prefetch(5)
return last
AviSynth will probably even detect that B is never used, therefore never execute "B = source().filter2()".
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid

Last edited by LigH; 9th November 2017 at 00:47.
LigH is offline  
Old 9th November 2017, 00:52   #3720  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 9,144
I nearly suggested same as Fluffy, but as I've never used MT/Avs+, thought it might be daft.

Perhaps docs could be updated to reflect that Prefetch takes a [EDIT: compulsory] clip arg (if indeed it does), docs as given on Avisynth.org/Avs+ below.

Quote:
Enabling MT

The other difference is how you actually enable multithreading. Calling SetFilterMTMode() is not enough, it sets the MT mode, but the MT mode only has an effect if MT is enabled at all. Note this means you can safely include/import/autoload your SetFilterMTMode() calls in even single-threaded scripts, and they will not be messed up. Uhm, onto the point: You enable MT by placing a single call to Prefetch(X) at the *end* of your script, where X is the number of threads to use.
Example
Code:
# This line causes all filters that don't have an MT mode explicitly use mode 2 by default.
# Mode 2 is a relatively safe choice until you don't know most of your calls to be either mode 1 or 3.
# Compared with mode 1, mode 2 trades memory for MT-safety, but only a select few filters will work with mode 1.
SetFilterMTMode("DEFAULT_MT_MODE", 2)
or
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)

# FFVideoSource(), like most of all source filters, needs MT mode 3. 
# Note: starting  with AviSynth+ r2069, it will now automatically recognize source filters.
# If it sees a source filter which has no MT-mode specified at all, it will automatically use 
# mode 3 instead of the default MT mode.
SetFilterMTMode("FFVideoSource", 3)
or 
SetFilterMTMode("FFVideoSource", MT_SERIALIZED)

# Now comes your script as usual
FFVideoSource(...)
Trim(...)
QTGMC(...)
...

# Enable MT!
Prefetch(4)
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 9th November 2017 at 01:09.
StainlessS is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:24.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.