Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 9th August 2018, 18:46   #4201  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,547
This is based on my observation when developing VS but it'd surprise me greatly if AVS isn't guilty as well.

At some point the frame requests end up out of order and then the source filter has to seek. This is very slow in most of them. I ended up adding logic and a flag to indicate which filters really benefit from linear access and extra logic to make requests in order more of the time. It resulted in significant speedups.

I don't understand how Avs+ multithreading works but it's most likely close to the truth...
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline  
Old 11th August 2018, 12:07   #4202  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Short question for the AVS+ gurus:

I am aware that the "Prefetch(x)" command should be called last in the AVS script (except for a "Return()" call). All commands after "Prefetch" will not be multithreaded.

Right now I am in the process to migrate to AVS+, and of course I want everything to be automatic. In my older 32-bit StaxRip installation I added a "Prefetch" command as the last call in the script for all my templates. But as soon as I edit out commercials in the preview, StaxRip automatically adds a "Trim()" statement as the last command (after the "Prefetch" call).

My question: Does it matter at all? Can "Trim()" even be prefetched? I did a few tests with "Trim()" before and after "Prefetch()", but the speed was identical, and the output also had no artifacts for both cases.


Cheers
manolito
manolito is offline  
Old 11th August 2018, 12:57   #4203  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,803
In ripbot264 I do trimming after prefetch

Code:
#MT

#VideoSource

#Deinterlace

#Decimate

#Crop

#Resize

#Tonemap

#Levels

#Colours

#Denoise

#Custom

#Prefetch

#Subtitles

#AudioSource

#Triming

#ColorSpace

#Return
Atak_Snajpera is offline  
Old 11th August 2018, 14:04   #4204  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,746
Trim() is applied once to the whole clip, therefore it cannot be parallelized frame by frame. Practically, it will probably just set a "first output frame" and a "last output frame" attribute; done.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline  
Old 11th August 2018, 19:39   #4205  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Hello, I'm back again. I just want to say thanks again especially to Groucho2004 for his AviSynth installer and AVS meter. I'm the one who has Threadripper processor. Last February I believe I fixed my problem (low fps etc). Got 70fps (happy with that to be honest). But I've decided to reformat last 2 days and my FPS is back to 50fps. I did some googling and came to this forum again and fixed my problem.

I'm using old AviSynth+ version though. The latest versions have low fps too



Btw why the CPU Usage is always 0. The thread ripper is default and no overclock. This is the first time that the AviSynth script gives real FPS from the avsmeter (100+ fps in software too) unlike from my previous posts wherein avsmeter report 100 fps (70 fps in the same software). Thanks again
sausuke is offline  
Old 11th August 2018, 19:46   #4206  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by sausuke View Post
why the CPU Usage is always 0.
1. Post the script that gives you 0% CPU usage.
2. Update AVSMeter to the latest version, run "AVSMeter avsinfo -log" and post the log file.

What is the CPU usage in Task manager when you run the test?
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 11th August 2018, 20:03   #4207  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by Groucho2004 View Post
1. Post the script that gives you 0% CPU usage.
2. Update AVSMeter to the latest version, run "AVSMeter avsinfo -log" and post the log file.

What is the CPU usage in Task manager when you run the test?
Script
Quote:
MP_Pipeline("""
### platform: win32
AVISource("E:\2Encoded Files\sss.avi", audio=false)
ConvertToYV12(matrix="PC.709")
### ###
""")
I updated it to the latest v2.8.1

Here's the cpu usage


Log:
https://pastebin.com/pLZiYyCd
sausuke is offline  
Old 11th August 2018, 20:10   #4208  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by sausuke View Post
Code:
MP_Pipeline("""
### platform: win32
AVISource("E:\2Encoded Files\sss.avi", audio=false)
ConvertToYV12(matrix="PC.709")
### ###
""")
MP_Pipeline runs the script in a different process, that's why you get 0% in AVSMeter. Open the "Processes" tab in Task Manager and check the CPU usage for each process.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 11th August 2018, 20:14   #4209  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by Groucho2004 View Post
MP_Pipeline runs the script in a different process, that's why you get 0% in AVSMeter. Open the "Processes" tab in Task Manager and check the CPU usage for each process.
oh, I see, here's my top CPU usage while using avsmeter


sausuke is offline  
Old 11th August 2018, 20:49   #4210  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,746
Vegas is very busy, why do you run it at the same time you try to benchmark AviSynth?

What kind of AVI do you read, is it completely uncompressed, so reading from the disk consumes most of the time?
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline  
Old 11th August 2018, 21:19   #4211  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by LigH View Post
Vegas is very busy, why do you run it at the same time you try to benchmark AviSynth?

What kind of AVI do you read, is it completely uncompressed, so reading from the disk consumes most of the time?
I'm frameserving after I edit my videos. That's why my issue is complicated (for me). I'm not just converting a video (e.g video to handbrake).

I think I need to move on with the software to maximize threadripper. I'm just too attached with Vegas because I'm used to it. But the 100fps from 70fps is okay for me. I've tried other NLE but it's pain to setup with frameserver and the quality to size ratio is not good compare to x264.
sausuke is offline  
Old 12th August 2018, 12:19   #4212  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,803
Quote:
Originally Posted by sausuke View Post
I'm frameserving after I edit my videos. That's why my issue is complicated (for me). I'm not just converting a video (e.g video to handbrake).

I think I need to move on with the software to maximize threadripper. I'm just too attached with Vegas because I'm used to it. But the 100fps from 70fps is okay for me. I've tried other NLE but it's pain to setup with frameserver and the quality to size ratio is not good compare to x264.
I'm curious how many fps do you get while exporting using UT Video codec (YUV420) in Vegas?
Atak_Snajpera is offline  
Old 13th August 2018, 19:13   #4213  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,255
Is there some known problem with 'AviSynth+ 0.1 (r2728, MT, i386) (0.1.0.0)' and DeGrainMedian ?
I started with:
Code:
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\DGDecode.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\TDeint.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\TMM.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\MosquitoNR.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\RgTools.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\masktools2.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\degrainmedian.dll")
Import("I:\Hybrid\32bit\avisynthPlugins\Dehalo_alpha_mt.avsi")
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
# loading source: VTS_01_1.VOB
#  input color sampling YV12
#  input luminance scale tv
MPEG2Source(d2v="E:\Temp\vob_3b6af15ddd3b19a201adf2536e9a8f23_41.d2v")
# current resolution: 720x576
# deinterlacing
TDeint(slow=2)
# cropping to 720x550
Crop(0,12,0,-14)
# current resolution: 720x550 
# filtering
# deringing using MosquitoNR
MosquitoNR()
# dehaloing
DeHalo_alpha_mt()
# grain handling
DeGrainMedian(mode=2,norow=true)

# CUSTOM SCRIPT PART - position: Resize - START
function CustomResize(clip clp) {
  last=clp
  DeGrainMedian(limitY=3,limitUV=5,mode=2,norow=true)
}
# CUSTOM SCRIPT PART - position: Resize - END

CustomResize() # loading custom script content
# scaling
Spline36Resize(720,384)
# current resolution: 720x384
PreFetch(8)
return last
and when AVSMeter crashed and reported:
Code:
Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module:   C:\Windows\SysWOW64\KERNELBASE.dll
Address:  0x76C7DDC2
commenting out 'CustomResize', I tried with:
Code:
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\DGDecode.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\TDeint.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\TMM.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\MosquitoNR.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\RgTools.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\masktools2.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\degrainmedian.dll")
Import("I:\Hybrid\32bit\avisynthPlugins\Dehalo_alpha_mt.avsi")
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
# loading source: VTS_01_1.VOB
#  input color sampling YV12
#  input luminance scale tv
MPEG2Source(d2v="E:\Temp\vob_3b6af15ddd3b19a201adf2536e9a8f23_41.d2v")
# current resolution: 720x576
# deinterlacing
TDeint(slow=2)
# cropping to 720x550
Crop(0,12,0,-14)
# current resolution: 720x550 
# filtering
# deringing using MosquitoNR
MosquitoNR()
# dehaloing
DeHalo_alpha_mt()
# grain handling
DeGrainMedian(mode=2,norow=true)

# CUSTOM SCRIPT PART - position: Resize - START
function CustomResize(clip clp) {
  last=clp
  DeGrainMedian(limitY=3,limitUV=5,mode=2,norow=true)
}
# CUSTOM SCRIPT PART - position: Resize - END

#CustomResize() # loading custom script content
# scaling
Spline36Resize(720,384)
# current resolution: 720x384
PreFetch(8)
return last
and I got the same crash, but once I also commented the other DeGrainMedian call out too and tried:
Code:
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\DGDecode.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\TDeint.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\TMM.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\MosquitoNR.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\RgTools.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\masktools2.dll")
LoadPlugin("I:\Hybrid\32bit\avisynthPlugins\degrainmedian.dll")
Import("I:\Hybrid\32bit\avisynthPlugins\Dehalo_alpha_mt.avsi")
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
# loading source: VTS_01_1.VOB
#  input color sampling YV12
#  input luminance scale tv
MPEG2Source(d2v="E:\Temp\vob_3b6af15ddd3b19a201adf2536e9a8f23_41.d2v")
# current resolution: 720x576
# deinterlacing
TDeint(slow=2)
# cropping to 720x550
Crop(0,12,0,-14)
# current resolution: 720x550 
# filtering
# deringing using MosquitoNR
MosquitoNR()
# dehaloing
DeHalo_alpha_mt()
# grain handling
#DeGrainMedian(mode=2,norow=true)

# CUSTOM SCRIPT PART - position: Resize - START
function CustomResize(clip clp) {
  last=clp
  DeGrainMedian(limitY=3,limitUV=5,mode=2,norow=true)
}
# CUSTOM SCRIPT PART - position: Resize - END

#CustomResize() # loading custom script content
# scaling
Spline36Resize(720,384)
# current resolution: 720x384
PreFetch(8)
return last
the script isn't crashing anymore.

Memory usage is around 220MB all the time.

So I'm wondering:
a. is there a newer verison of degrainMedian ? (using v0.8.2 from 2006 atm.)
b. is this a known problem?
c. is there a fix for this?


Cu Selur

Ps.: The script itself works fine with normal Avisynth MT 32bit. (minus the Prefetch and SetFilterMTMode call and adding distributor(), SetMTMode(),..)
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 13th August 2018 at 19:30.
Selur is offline  
Old 13th August 2018, 23:27   #4214  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Selur View Post
So I'm wondering:
a. is there a newer verison of degrainMedian ? (using v0.8.2 from 2006 atm.)
b. is this a known problem?
c. is there a fix for this?
a. I'm not aware of a newer version

b. I can reproduce the problem. It may be that many of the inline ASM MMX functions in DeGrainMedian are missing the EMMS instruction. See here for some info.

c. I made a new build with the EMMS instructions included, let me know if it works for you (it does for me).
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 14th August 2018 at 10:12.
Groucho2004 is offline  
Old 13th August 2018, 23:38   #4215  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,492
Is this thread a good place to mention errors on the Wiki? I spotted this:

Quote:
If the input clip is field-based, the DoubleWeave filter operates like Weave, except that it produces double the number of frames: instead of combining fields 0 and 1 into frame 0, fields 2 and 3 into frame 1, and so on, it combines fields 0 and 1 into frame 0, fields 1 and 2 into frame 1, and so on. It does not change the [...] frame count.
http://avisynth.nl/index.php/DoubleWeave
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline  
Old 13th August 2018, 23:43   #4216  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by davidhorman View Post
Is this thread a good place to mention errors on the Wiki?
Probably the worst place.
If it's not too much hassle, why don't you create an account on avisynth.nl and fix it yourself? That's what I do (and many others) when I spot a mistake.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 14th August 2018, 00:03   #4217  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,492
Oh! I automatically assumed it wasn't open to registrations for some reason.
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline  
Old 14th August 2018, 04:33   #4218  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,255
@Groucho2004: Thanks! Did a quick test and it seems to fix the problem, will do some more testing and report in case I run into problems.
__________________
Hybrid here in the forum, homepage
Selur is offline  
Old 15th August 2018, 16:15   #4219  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Another (probably stupid) question from an AVS+ noob:

If MT is enabled by the "Prefetch(x)" command at the end of the script then all of the plugins before the Prefetch command will run in MT mode. Which of the 4 different MT modes gets used is specified by an explicit SetFilterMTMode() command or by a "SetMTMode.avsi" in the autoload folder. If none of the MT modes is specified then the default MT mode 2 (MT_MULTI_INSTANCE) will be used. Everybody agrees on this?

Now I found this about ColorMatrix as a comment in the current "SetMTMode.avsi":
Quote:
#note2: tried multiple files, seems to corrupt video even when it is the only filter in a script.
#tried mode 1, 2, and 3, none worked. however it works fine if MT isn't enabled.
So I would like to tell AVS+ to use ColorMatrix in "Single Threaded" mode while I still have a "Prefetch(x)" command at the end of the script. How would I do this? Is there any SetFilterMTMode() command which forces ST mode for a specified filter?


Cheers
manolito
manolito is offline  
Old 15th August 2018, 17:17   #4220  |  Link
videoh
Useful n00b
 
Join Date: Jul 2014
Posts: 1,667
E.g.:

SetFilterMTMode("FFVideoSource", MT_SERIALIZED)
videoh is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:48.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.