Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 7th March 2019, 00:39   #4561  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,498
https://forum.doom9.org/showthread.p...39#post1865439

Sorry I spoke too soon, the joy only lasted for 3 weeks...

Last night I tried to convert a longer movie (2 and a half hours), and I got the crashes again. And it was reproduceable.

My AVS script:
Code:
DSS2("D:\Black Mass.mkv", fps=50.000, preroll=15)
Crop(0,0, -Width % 4,-Height % 4)
ColorMatrix(source=0,dest=2)
RequestLinear(rlim=50, clim=50)
ConvertToYV12()
Spline36Resize(704,396)
FDecimate(25)

# Start Remove Logo 
Import("I:\Logo\Area.avs") #Defines the X and Y coordinates of the logo area
ConvertToYV12(interlaced=false)
Logo = crop(X1,Y1,X2-X1,Y2-Y1)
Above = Y1 > 0 ? crop(0,0,width(),Y1) : NOP()
Below = Y2 < height() ? crop(0,Y2,width(),height()-Y2) : NOP()
Left = X1 > 0 ? crop(0,Y1,X1,Y2-Y1) : NOP()
Right = X2 < width() ? crop(X2,Y1,width()-X2,Y2-Y1) : NOP()
ConvertToRGB32(Logo,interlaced=false)
LoadVirtualdubPlugin("E:\Programme\Virtualdub\Plugins\logoaway.vdf","VD_LogoAway")
# -----------------------------------------------------------------------------
VD_LogoAway( 3, 327685, 9895963, 0, 0, 5, 0, 66051, 131584, 10, "", "", "")
# -----------------------------------------------------------------------------
ConvertToYV12(interlaced=false)
IsClip(Left) ? StackHorizontal(Left, last) : NOP()
IsClip(Right) ? StackHorizontal(last, Right) : NOP()
IsClip(Above) ? StackVertical(Above, last) : NOP()
IsClip(Below) ? StackVertical(last, Below) : NOP()
last
# End Remove Logo

FineSharp()
avstp_set_threads(1)
Prefetch(4)
Trim(4248,161424)
This script is really not very complex. almosely's changes slowed it down quite a bit (adding Requestlinear after ColorMatrix and selecting MT_SERIALIZED for both ColorMatrix and RequestLinear), so the speed gain of MT was only about 1fps. Reducing the prefetch value to 2 did not make a difference, still I got crashes. Only disablng MT altogether made the script stable.

For my needs I do not see any advantage of using AVS+ over classic AVS 2.61. I reverted back to AVS classic, probably for good this time. (Unless someone comes up with a genius idea to use MT without crashes and still get a better speed).


Cheers
manolito
manolito is offline   Reply With Quote
Old 7th March 2019, 01:13   #4562  |  Link
VS_Fan
Registered User
 
Join Date: Jan 2016
Posts: 91
Quote:
Originally Posted by manolito View Post
... For my needs I do not see any advantage of using AVS+ over classic AVS 2.61. I reverted back to AVS classic, probably for good this time. (Unless someone comes up with a genius idea to use MT without crashes and still get a better speed).
I remember some years ago having problems with frame decimation in a Multi-Threaded environment (old AVS-MT) and trying innumerable different configurations for ‘RequestLinear’ to no avail.

What I successfully ended doing was:
  • Separating only the source, decimation with any other ‘MT_SERIALIZED’ filters in one Single-Threaded AVS script, No need of of using 'requestlinear';
  • Then I either: ‘mounted’ that first script with AVFS; or simply read it directly with “AVIFileSource” into a second MultiThreaded AVS script where I put every other processing filters, susceptible of speed gains because of multi-threading
I don’t think this is genius idea, but I sincerely hope it helps
VS_Fan is offline   Reply With Quote
Old 8th March 2019, 12:18   #4563  |  Link
an3k
Registered User
 
an3k's Avatar
 
Join Date: Oct 2006
Location: Munich, Germany
Posts: 164
I have a Plugin (DGHDRtoSDR) that is not yet listed in the mtmodes.avsi and I would like to test (and add) it. I've searched but not found the info thus I asked here.

1) What is the "single-thread" mtmode in which every plugin perfectly runs at (since it's "non-MT-behavior")? Is it NICE_FILTER or SERIALIZED?

2) What is the most performant mtmode we would love to have all plugins running at? MULTI_INSTANCE?

Thanks
an3k is offline   Reply With Quote
Old 8th March 2019, 15:49   #4564  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,069
Quote:
Originally Posted by VS_Fan View Post
I remember some years ago having problems with frame decimation in a Multi-Threaded environment (old AVS-MT) and trying innumerable different configurations for ‘RequestLinear’ to no avail.

What I successfully ended doing was:
  • Separating only the source, decimation with any other ‘MT_SERIALIZED’ filters in one Single-Threaded AVS script, No need of of using 'requestlinear';
  • Then I either: ‘mounted’ that first script with AVFS; or simply read it directly with “AVIFileSource” into a second MultiThreaded AVS script where I put every other processing filters, susceptible of speed gains because of multi-threading
I don’t think this is genius idea, but I sincerely hope it helps
It's might be simpler than that. As far as I can tell FDecimate isn't in mtmodes.avsi, but since it's a m-in-n decimation filter it's almost certainly very stateful and not at all threadsafe. Setting it to MT_SERIALIZED might very well fix the crashing.

That said, the filter chain in that script is so full of ancient serialized junk so far down into the chain that the only thing of note that runs parallelized at all is FineSharp (I don't think vdub filters are MT_NICE? who knows though), so it's no wonder it's not any faster. As a reminder, inserting a MT_SERIALIZED filter into a filter chain will effectively cause every filter upstream of it all the way up to the source filter to run serialized too, at least as far as performance goes.

Quote:
Originally Posted by an3k View Post
I have a Plugin (DGHDRtoSDR) that is not yet listed in the mtmodes.avsi and I would like to test (and add) it. I've searched but not found the info thus I asked here.

1) What is the "single-thread" mtmode in which every plugin perfectly runs at (since it's "non-MT-behavior")? Is it NICE_FILTER or SERIALIZED?

2) What is the most performant mtmode we would love to have all plugins running at? MULTI_INSTANCE?

Thanks
1) MT_SERIALIZED.

2) MT_NICE_FILTER.

See here for more details. As mentioned above, using a MT_SERIALIZED filter late in a chain effectively disables MT (you'll be so bottlenecked that MT doesn't help you much).

If you're using Vapoursynth though you can get some degree of parallelism even with serialized Avisynth filters, which you can't do in Avs+ itself. In Avs+, calling GetFrame on a serialized filter will essentially run every filter upstream of that point single threaded too, so requesting a frame will make the entire thing block until the entire chain is finished processing that frame, one filter at a time in a single thread. In VS, you still get a single instance of the serialized filter (in its own pretend environment) and it will only ever be asked to process one frame at a time, but upstream of it can run in parallel.
TheFluff is offline   Reply With Quote
Old 9th March 2019, 18:32   #4565  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,069
I don't know who maintains the MT modes list anymore, or if the PublishWithMe pad is still the single source of truth, but colormatrix should, uh, probably not be marked as a MT_NICE_FILTER. The publishwithme pad itself notes that it seems very broken even when MT_SERIALIZED. Newer versions are internally multithreaded, and it's always done a lot of funky stuff internally (like requesting frames from the constructor and Invoke()'ing a lot of stuff).

That said, don't use colormatrix, it's always been a piece of buggy garbage.

Last edited by TheFluff; 9th March 2019 at 18:35.
TheFluff is offline   Reply With Quote
Old 9th March 2019, 19:08   #4566  |  Link
DJATOM
Registered User
 
DJATOM's Avatar
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 152
Yeah, my friend used it in his scripts and it silently kills avs2yuv without any error. The only plugin that really piss me off...
__________________
Me on GitHub | My Telegram
DJATOM is offline   Reply With Quote
Old 9th March 2019, 20:31   #4567  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,498
Quote:
Originally Posted by TheFluff View Post
...but colormatrix should, uh, probably not be marked as a MT_NICE_FILTER.
Did you have a look at this post?
https://forum.doom9.org/showthread.p...79#post1865279

Quote:
ColorMatrix is running absolutely fine in MT-Mode within AVS+ (x64), but only it it's placed within a bubble of sequential frame order. So, to ensure that, RequestLinear has to be used and to be defined as MT_SERIALIZED. It makes no sense to use RequestLinear as MT_MULTI_INSTANCE. Therefore the following sript is working totally fine for me (I tested it multiple times with ST and MT encodings of the same clip) - no differences within the x264 logfiles. I did encounter differences when not using RequestLinear as MT_SERIALIZED of course.
Applying these changes made my conversions a lot more stable, until a couple of days ago I got crashes again while converting a longer movie. Now I additionally specified MT_SERIALIZED for FDecimate, the first 2 conversions went well, let's see if it lasts...
manolito is offline   Reply With Quote
Old 11th March 2019, 01:34   #4568  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,498
Did some more speed benchmark tests using a long 2:40 movie (Heat by Michael Mann). I basically used the script from a couple of posts above. I did get a small speed gain by moving the FDecimate(25) call to the top of the script right after the source filter, and so far everything was stable.

I also tested the same source using z_ConvertFormat replacing ColorMatrix, RequestLinear and the built-in resizer. It worked nicely, but even in the default MT_MULTI_INSTANCE mode it was about 0.5 fps slower than the other script (ColorMatrix and RequestLinear both in MT_SERIALIZED mode). So as long as the script with ColorMatrix is stable for me I see no reason to ditch it.

For the SetMTMode.avsi I don't know if anybody is actively maintaining it. I uploaded my modified version here:
https://www.sendspace.com/file/3pdsc8
All comments were removed, also all MT_MULTI_INSTANCE calls were deleted (not necessary, MULTI_INSTANCE is the default).


Cheers
manolito
manolito is offline   Reply With Quote
Old 11th March 2019, 02:55   #4569  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,810
Quote:
Originally Posted by manolito View Post
So as long as the script with ColorMatrix is stable for me I see no reason to ditch it.

Quality wise, ColorMatrix produces color splotches and noise compared to higher quality methods. It's more noticable on animation , gradients ; less noticable on live action, film grain sources
poisondeathray is offline   Reply With Quote
Old 11th March 2019, 05:47   #4570  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,498
Quote:
Originally Posted by poisondeathray View Post
Quality wise, ColorMatrix produces color splotches and noise compared to higher quality methods.
So far I am only aware of z_ConvertFormat as a "higher quality method" under AVS. Which other methods are you talking about?

Plus your statement about ColorMatrix producing color splotches and noise is quite new to me. So far I was only aware of Fluffy's statement that ColorMatrix was not suitable for MT under AVS due to questionable coding methods which offended his professional standards. Now you are talking about quality issues even when used in single threaded scripts.

Can you provide some proof of your statement? A source segment where a color conversion from Rec.709->Rec.601 reveals quality problems like color splotches and noise when using ColorMatrix?
manolito is offline   Reply With Quote
Old 11th March 2019, 06:11   #4571  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,810
Quote:
Originally Posted by manolito View Post
So far I am only aware of z_ConvertFormat as a "higher quality method" under AVS. Which other methods are you talking about?

Plus your statement about ColorMatrix producing color splotches and noise is quite new to me. So far I was only aware of Fluffy's statement that ColorMatrix was not suitable for MT under AVS due to questionable coding methods which offended his professional standards. Now you are talking about quality issues even when used in single threaded scripts.

Can you provide some proof of your statement? A source segment where a color conversion from Rec.709->Rec.601 reveals quality problems like color splotches and noise when using ColorMatrix?


Yes z.lib (z_ConvertFormat in avisynth), or AviSynthShader, probably Dither tools too. Basically anything else

709<=>601 either direction. Just compare ColorMatrix , it's easier to see on anime, cartoons . I'll post something tomorrow, there were some threads about this too.
poisondeathray is offline   Reply With Quote
Old 11th March 2019, 22:26   #4572  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,810
Quote:
Originally Posted by manolito View Post

Can you provide some proof of your statement? A source segment where a color conversion from Rec.709->Rec.601 reveals quality problems like color splotches and noise when using ColorMatrix?



@manolito - RE: colormatrix quality issues
https://forum.videohelp.com/threads/...709-to-rec-601

(the OP only provided screenshot, and it was converted back to YV12 for the "new" source but it's quite easy to see the problems)

Observations (not just this example, but in general):
- not version dependent, I tried a few colormatrix dll's , and the "ported" to ffmpeg filter exhibits the same issue. Probably partially due to 8bit conversion
- not settings dependent in terms of colormatrix clamping values (I usually set clamp=0 anyways, but pre/post clamp doesn't help)
- more noticable with certain color combinations , certain patterns that predispose to the issue. But source noise, film grain can often cover up the problems
- can occur 601=>709 too , eg. with DVD's when upscaling to HD . There were some DVD examples in other threads, I'll try to dig them up if you're interested
- the "better" high bit depth conversions are not because of dithering only (ie. it's not solely because of "covering up"); because if you disable dithering you still get "cleaner" image without the splotches and artifacts . But you can experiment with dithering algorithms back to 8bit . fmtc in vapoursynth has several options , it's easy to compare in avspmod tabs for example, with vsimport . avs+'s default convertbits(8) will use ordered dither .
poisondeathray is offline   Reply With Quote
Old 12th March 2019, 01:00   #4573  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,498
Thanks for digging out the VideoHelp thread, I went over it quickly, but I could not reproduce the findings.

My sources are always 8-bit with 4:2:0 chroma. And the conversion from HEVC HD to AVC SD keeps the video this way. No dithering involved, and the color space also stays the same.

Also I only convert films, no anime. I went over some of my conversions which used ColorMatrix carefully, and I could not detect any color artifacts. So my only valid reason to ditch ColorMatrix and use z_ConvertFormat instead would be a better MT speed, and right now I do not get this.


Cheers
manolito
manolito is offline   Reply With Quote
Old 12th March 2019, 08:12   #4574  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,183
Quote:
Originally Posted by poisondeathray View Post
avs+'s default convertbits(8) will use ordered dither .
ConvertBits does not dither, you have to specify dither=0 or dither=1.
pinterf is online now   Reply With Quote
Old 12th March 2019, 19:38   #4575  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,069
I guess I should point to the elephant in the room and ask why you're even converting to Rec601 at all, when metadata flagging has been the preferred solution for at least a decade. I have a sneaking suspicion the answer probably involves some kind of arcane playback contraption from the late 1990's, though.
TheFluff is offline   Reply With Quote
Old 12th March 2019, 21:44   #4576  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,124
Quote:
Originally Posted by TheFluff View Post
I guess I should point to the elephant in the room and ask why you're even converting to Rec601 at all, when metadata flagging has been the preferred solution for at least a decade. I have a sneaking suspicion the answer probably involves some kind of arcane playback contraption from the late 1990's, though.
He wanted to playback on his CRT TV
lansing is offline   Reply With Quote
Old 13th March 2019, 00:13   #4577  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,498
Quote:
Originally Posted by TheFluff View Post
I have a sneaking suspicion the answer probably involves some kind of arcane playback contraption from the late 1990's, though.
How in the world did you know that?

My playback device is an older (around 2010) Xtreamer Sidewinder connected to my CRT TV. It does not handle H.265, but it plays H.264 just fine. This means that my captured HEVC streams need to be converted to AVC anyways, and while I'm at it I can just as well reduce the resolution to SD 704x396 and decimate the framerate from 50fps to 25fps. This looks good on my TV.

My knowledge of the Rec709 and Rec601 standards is limited, but conventional wisdom says that HD is Rec709 while SD is Rec601. My captured sources are all 1080p50 and flagged as 709, so I figured that I need to convert them to 601 when I convert them to SD.

The data sheet of my Xtreamer unfortunately tells me nothing about which color it expects. The only way to check for the correct color is to switch between playback of the original stream through the DVB-T2 receiver and the converted SD stream through the Xtreamer. And this comparison does not show any color differences, so I guess I am doing it right...

Last edited by manolito; 13th March 2019 at 00:15.
manolito is offline   Reply With Quote
Old 13th March 2019, 00:33   #4578  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,249
Quote:
Originally Posted by manolito View Post
This means that my captured HEVC streams need to be converted to AVC anyways, and while I'm at it I can just as well reduce the resolution to SD 704x396 and decimate the framerate from 50fps to 25fps. This looks good on my TV.

My knowledge of the Rec709 and Rec601 standards is limited, but conventional wisdom says that HD is Rec709 while SD is Rec601. My captured sources are all 1080p50 and flagged as 709, so I figured that I need to convert them to 601 when I convert them to SD.
I'd use DitherTools for that process.

Edit - Example
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 13th March 2019 at 00:41.
Groucho2004 is offline   Reply With Quote
Old 13th March 2019, 11:16   #4579  |  Link
goorawin
Registered User
 
Join Date: Feb 2012
Posts: 67
Here is a script that works really well at converting 1920x1080x50p to 720x576x25i (DVD standard). You may need to change the SelectEvery, to get the field order correct. Also change the sharpen to suit the source material.

bicubicresize(1440,1152)
bicubicresize(720,1152,-.8,.6)
bicubicresize(720,576,-.8,.6)
blur(0.0,1.0)
sharpen(0.0,0.75)
assumeBff()
separatefields()
SelectEvery(4,1,2)
Weave()
Matrix(from=709, to=601)
goorawin is offline   Reply With Quote
Old 15th March 2019, 00:05   #4580  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,249
Quote:
Originally Posted by goorawin View Post
Matrix(from=709, to=601)[/I]
Is that pseudo code or an actual function?
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:04.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.