Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 15th August 2018, 19:12   #4221  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Are you serious?
MT_SERIALIZED equals Mode 3. And if I understand this correctly then Mode 3 is not single threaded. Yes I do know what parallel and serial means, but specifying Mode 3 is certainly not the same as disabling MT.
From the AVS+ Wiki:
Quote:
Mode 3 (MT_SERIALIZED) is evil.
It should only be used for source filters or filters which do not have a Clip parameter.

The quoted note said that he tried mode 1, 2 and 3 and none worked. It only worked if MultiThreading was not enabled.

So my question remains:
How do I force ColorMatrix to run single threaded without removing the Prefetch(x) command which would make my whole script run single threaded?

Last edited by manolito; 15th August 2018 at 19:32.
manolito is offline  
Old 15th August 2018, 19:51   #4222  |  Link
DJATOM
Registered User
 
DJATOM's Avatar
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 377
Yeah, colormatrix is just too broken, so I'd like to suggest you dithertools instead:
Quote:
Dither_convert_yuv_to_rgb (matrix="601", output="rgb48y")
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
Dither_convert_rgb_to_yuv (r, g, b, matrix="709", lsb=false, mode=0)
__________________
Me on GitHub
PC Specs: Ryzen 5950X, 64 GB RAM, RTX 2070
DJATOM is offline  
Old 15th August 2018, 20:26   #4223  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
or another alternative to colormatrix that is prefetch(x) safe in avs+mt would be avsresize using the colorspace_op parameter
poisondeathray is offline  
Old 15th August 2018, 21:45   #4224  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by DJATOM View Post
Yeah, colormatrix is just too broken
I have a hard time to believe that tritical has ever published something that is "just too broken"...

Anyhow, DitherTools is not an option for me. First of all I have no use for this high bitdepth stuff, and secondly DitherTools requires a SSE2 capable CPU which one of my computers does not have (and I want identical AviSynth configurations on all of my machines).

I just found that ColorMatrix has a "Threads" parameter:
Quote:
threads:

Sets the number of threads Colormatrix will use for processing. Can be any value greater than 0 and, for YUY2, less than the frame height, for YV12, less than the frame height divided by 2. If set to 0, ColorMatrix will automatically detect the number of available processors and set threads equal to that value.

default - 1 (int)
Could it help for AVS+ multithreading to increase the "Threads" value to the same value which is used by "Prefetch()"?


If I really cannot get ColorMatrix to work reliably with AVS+ multithreading then I am already tempted to revert back to classic AVS 2.60. Most of my AVS scripts (HD to SD conversions) use ColorMatrix, this needs to be absolutely reliable for me. Since I do not use the expanded bit depths and color formats of AVS+ all I really care for is the speed gain from multitasking. And my tests so far (using only filters which are not too complex) only showed a very moderate speed gain. Nothing I would trade for the risk of introducing artifacts which I might only detect much later.


Cheers
manolito
manolito is offline  
Old 15th August 2018, 22:26   #4225  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by manolito View Post
If I really cannot get ColorMatrix to work reliably with AVS+ multithreading then I am already tempted to revert back to classic AVS 2.60.
Can you post a script that shows the problem with ColorMatrix? I have never noticed any.

Anyway, I don't see the problem using MT_SERIALIZED with ColorMatrix. I just ran a few tests and the performance hit is virtually zero, even if I put the ColorMatrix call near the end of the script.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline  
Old 15th August 2018, 22:53   #4226  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by Groucho2004 View Post
Can you post a script that shows the problem with ColorMatrix? I have never noticed any.
No I can't because so far I also did never see any problems with multithreading ColorMatrix. Neither with the default MT_MULTI_INSTANCE nor with MT_NICE_FILTER which gets set when using the "MTMode.avsi" from here:
http://publishwith.me/ep/pad/view/ro.rDkwcdWn4k9/latest

What really turned on the red lights for me was the note in this AVSI that modes 1, 2 and 3 all caused artifacts. What should I make of this contradicting information? Is MT_SERIALIZED really safer?

The reason I am a little touchy on this subject is that I already had some bad experiences with ColorMatrix in the past. It was an older version which caused random artifacts on random sources, no error messages. It took me a while to find out that the newer version 2.5 had fixed the issues, but it was too late. After I finally detected the artifacts I did no longer have access to the source files.


Cheers
manolito
manolito is offline  
Old 16th August 2018, 00:20   #4227  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Methinks that you squeal long before you are bitten.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline  
Old 16th August 2018, 01:38   #4228  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by manolito View Post
I have a hard time to believe that tritical has ever published something that is "just too broken"...
No, ColorMatrix is definitely incredibly broken in several different ways and that definitely isn't unusual for tritical code. The man came up with some clever algorithms but he was definitely not someone whose code you'd ever want to read or maintain. ColorMatrix features an eclectic selection of incredibly poor programming decisions ranging from the usual inline assembler (with questionable speed benefit) that doesn't match the C implementation, to an eyewateringly awful D2V parser copypasted into the middle of the plugin, and finally to the (in this case) rather inconvenient fact that it has its own internal win32 multithreading, which is used even if you set threads=1. The actual processing is always run in a separate thread, so you can't actually make it single-threaded - it always uses at least two threads. That last fact probably broke Avs-MT's thread synchronization at some point, but I have no idea if it still does since very significant parts of the multithreading code was rewritten by ultim et al and then further modified by pinterf. I'm actually not sure if you can even entirely disable "MT" in Avs+ anymore. Writing comments in some obscure configuration file isn't a reasonable way to track and triage bug reports, in any case.

I'm pretty sure it works without issues in current Avs+ because this is the first thing I hear about artifacts and I know a lot of people use it with multithreading enabled in Avs+. However, there is really no reason to keep using ColorMatrix in 2018, since we have significantly better tools today. I'd suggest zimg (also referred to as "avsresize").

Last edited by TheFluff; 16th August 2018 at 02:00.
TheFluff is offline  
Old 16th August 2018, 01:54   #4229  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Fluffy, I'm always quite amazed at the depth of understanding you have of other peoples plugs,
I'm wondering if you could find the time to correct triticals 'not so good' decisions,
I'm quite sure many people would like a more robust and well more fluffy version of same.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline  
Old 16th August 2018, 02:07   #4230  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by StainlessS View Post
Fluffy, I'm always quite amazed at the depth of understanding you have of other peoples plugs,
I'm wondering if you could find the time to correct triticals 'not so good' decisions,
I'm quite sure many people would like a more robust and well more fluffy version of same.
No need, all of his useful plugins have already been ported (well, more like "reverse engineered and rewritten") to Vapoursynth by other people.

Note though that I don't have anything personal against tritical and I definitely don't deny that many of the plugins he wrote have been extremely influential and useful. As a professional software engineer though, his coding habits are deeply offensive to my sense of craftsmanship.

Last edited by TheFluff; 16th August 2018 at 02:12.
TheFluff is offline  
Old 16th August 2018, 02:13   #4231  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
reverse engineered and rewritten
Did triticals plugs come with source, usually ? (I think I have some of his source).

EDIT: Not everyone uses Vapoursynth, although as I understand it, some do.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 16th August 2018 at 02:16.
StainlessS is offline  
Old 16th August 2018, 02:17   #4232  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by StainlessS View Post
Did triticals plugs come with source, usually ? (I think I have some of his source).
Yes but it's completely unreadable and unmaintainable, and that's by far the biggest problem with all of his code. Myrsloik once referred to it as "open binary" because all the uncommented inline asm that doesn't match the C implementation and inscrutable numerical constants everywhere means that having the source code isn't really helpful at all when it comes to figuring out what the plugin is actually doing. You could have just disassembled the binary and you would've been in pretty much the same place.

If you want improved tritical plugins for Avisynth it'd be far, far easier to backport the Vapoursynth reimplementations than to try to fix the original code.

Last edited by TheFluff; 16th August 2018 at 02:22.
TheFluff is offline  
Old 16th August 2018, 02:25   #4233  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
OK, got ya.
I once tried figure out the AutoCrop thing by Gary (something beginning with B), and it was hell, real nasty stuff,
and that was only C/CPP, I guess that big T's efforts could be well more cryptic, Kassandro (not sure if I spelt right)
was well clever guy but also was a bit peculiar and somewhat messy and a little too secretive (undoc'ed functions and such).
Anyways, thanx for your answer, much appreciated.

EDIT: Just saw this
Quote:
If you want improved tritical plugins for Avisynth it'd be far, far easier to backport the Vapoursynth reimplementations than to try to fix the original code.
Worth considering, thank you.

EDIT: I know that I have been accused of writing "Machine Code" style C/CPP (by Feisty2), I really hope that Feisty
was/is just a little bit exaggerant in such accusation, I would hate to think that no-one at all could understand my
code, (well at least another C coder should be able to figure out what it was doing).
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 16th August 2018 at 02:48.
StainlessS is offline  
Old 16th August 2018, 11:11   #4234  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by TheFluff View Post
Writing comments in some obscure configuration file isn't a reasonable way to track and triage bug reports, in any case.

I'm pretty sure it works without issues in current Avs+ because this is the first thing I hear about artifacts and I know a lot of people use it with multithreading enabled in Avs+.
Well, this obscure configuration file is officially linked in the AVS+ Wiki. Among the authors and contributors you can find such "obscure" folks like Reel.Deal, tp7, Firesledge and real.finder.

Thanks for the hint that AVS+ MT was improved a lot during the last years, so Colormatrix may not be any problem today. I just saw that the note about Colormatrix was already present in this "obscure" AVSI in the first versions from 2015, so probably noone took the time to have another look at it recently.

I did comment out the line which specifies MT_NICE_FILTER for Colormatrix, so it will use the default MT_MULTI_INSTANCE now. Good idea or not?

StainlessS:
Quote:
Methinks that you squeal long before you are bitten.
You are damn right...
I have been bitten by an older version of Colormatrix before, and it was painful.


Cheers
manolito

Last edited by manolito; 16th August 2018 at 12:38.
manolito is offline  
Old 16th August 2018, 14:02   #4235  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by manolito View Post
Well, this obscure configuration file is officially linked in the AVS+ Wiki. Among the authors and contributors you can find such "obscure" folks like Reel.Deal, tp7, Firesledge and real.finder.
I meant obscure in the sense that it's not a place where people would go look for documentation about known issues, or think of as a place to update when such issues are fixed. For most people it's effectively equivalent to hardcoded internals. Since its version history is spotty at best it's also hard to figure out how old that comment really is, but it seems to have been there since mid-2014 at the very least. The same question you're asking now was brought up in this thread two years ago without any satisfactory answer. It's still there and spreading uncertainty to this day. The proper way of doing this kind of thing would be some kind of bug tracker.

Quote:
Originally Posted by manolito View Post
I did comment out the line which specifies MT_NICE_FILTER for Colormatrix, so it will use the default MT_MULTI_INSTANCE now. Good idea or not?
It's a pretty commonly used filter and everyone else who uses it with Avs+ most likely uses the default mode from that file. If you want to avoid unexpected and subtle problems, I would not recommend going out of your way to run a configuration that few other people use and test.

All that being said, I took a brief glance at the code again and I'd bet a fair sum that the reason it misbehaves (or used to misbehave) is extensive use of env->Invoke, especially in interlaced mode where it invokes InternalCache (which Avs+ (ab)uses a lot for its multithreading IIRC). Adding to the problem, the filters it invokes are invoked from the main thread, but when Avs+ then calls GetFrame the processing is actually done in a different thread inside ColorMatrix and I bet that has potential for interesting results - as far as I can remember Avs+ utilizes thread-local storage quite a bit. Still though, pinterf has done significant improvements to the robustness of the multithreading code over the last few years, especially when it comes to this kind of runtime shenanigans, so it may very well actually work now. Who knows, though!

Really though, don't use ColorMatrix. Why on earth would you invoke Limiter() instead of tacking on a simple min/max clamp to your pixel math? In the asm version it would literally be two instructions! Jeez.

zimg/avsresize works on ARM, so it should definitely work on a CPU without SSE2 too. No excuses.

Last edited by TheFluff; 16th August 2018 at 14:05.
TheFluff is offline  
Old 16th August 2018, 16:36   #4236  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by TheFluff View Post
zimg/avsresize works on ARM, so it should definitely work on a CPU without SSE2 too. No excuses.
Lots of excuses...
The zimg library might work without SSE2, the AVS wrapper DLL certainly does not.

The other excuse would be that the avsresize download package comes without any proper documentation for AVS users who have never touched VapourSynth (or are no diehard developers).

The HD to SD AVS script I use so far goes like this:
Code:
Source Filter
ColorMatrix(mode="Rec.709->Rec.601")
ConvertToYV12()
Spline36Resize(704,396)
How would I do the same thing with avsresize?

Looks like I would need to use the VS compatible calls. From the doc:
Code:
colorspace_op:          colorspace operation description
    Format is
        "matS[:transS[:primS[:rangeS]]]=>matD[:transD[:primD[:rangeD]]]"
This is something most AVS users including me will not understand. If someone wants avsresize to become more popular among AVS users then the first thing would be to write some proper documentation including usage examples (take a look at tritical's docs). I will stick with Colormatrix for the time being.


Cheers
manolito

Last edited by manolito; 16th August 2018 at 16:47.
manolito is offline  
Old 16th August 2018, 17:21   #4237  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by manolito View Post

The HD to SD AVS script I use so far goes like this:
Code:
Source Filter
ColorMatrix(mode="Rec.709->Rec.601")
ConvertToYV12()
Spline36Resize(704,396)
How would I do the same thing with avsresize?

Looks like I would need to use the VS compatible calls. From the doc:
Code:
colorspace_op:          colorspace operation description
    Format is
        "matS[:transS[:primS[:rangeS]]]=>matD[:transD[:primD[:rangeD]]]"
It's not that bad. Like avisynth , you can omit the other arguments and just use matrix (S is source, D is destination).
One difference in notation "170m" or "470bg" for sd matrix instead of "Rec601"; "709" instead of "Rec.709"

Code:
Source Filter
z_ConvertFormat(704,396, "YV12", colorspace_op="709=>170m",  resample_filter="spline36")

Another difference Fluffy alluded to is colormatrix clips (it doesn't "clamp", which implies "squishing", not hard clipping), unless you use clamp=0 (normally in avisynth classic I would use mode="Rec.709->Rec.601", clamp=0) . zlib/avsresize doesn't clip data
poisondeathray is offline  
Old 16th August 2018, 17:35   #4238  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Code:
colorspace_op="709=>470bg"
S and D stands for source and destination, "mat" means matrix, "trans" means transfer characteristics, "prim" means color primaries and range means range. As the short doc says, the string constants are in the Vapoursynth documentation.

e: poisondeathray got there first
TheFluff is offline  
Old 16th August 2018, 20:56   #4239  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,493
Is this expected/deliberate behaviour?

Code:
function _flipit(clip c) { # note the underscore
  return c.flipvertical
}

function flipit(clip c) {
  return c.fliphorizontal
}

version._flipit # note the underscore
The result is flipped horizontally, not vertically as you might expect. It seems that when faced with a function call with a leading _, Avisynth first matches it without the underscore, and only falls back to the underscored function (_flipit) if there is no non-underscored function (flipit).
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline  
Old 16th August 2018, 21:23   #4240  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
It is probably related to the function syntax {DllFileName}_{FunctionName} to select a function provided by a specific DLL when several DLLs in the range of loaded plugins (especially with a full auto-load directory) provide functions of the same name. An empty DLL name may match the AviSynth core.

You may prefer a function like flipit(clip c, bool vertical) instead.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid

Last edited by LigH; 16th August 2018 at 21:26.
LigH is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.