Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 19th February 2019, 09:04   #1  |  Link
silverwing
Registered User
 
Join Date: Nov 2017
Location: Russia, Nizhny Novgorod
Posts: 23
Can Avisynth Plus MT determine the number of processor cores and returns this value?

Hi, All!
One question.
Can Avisynth Plus MT determine the number of processor cores and returns this value? What function returns this value?
I need this value to substitute it into the Prefetch() function automatically (for multithreading).

I did it manually, but this is inconvenient, because I have to run the script on different computers with a different number of processor cores.

Is it possible?

Thank you for attention.
(Sorry for my bad English)
silverwing is offline   Reply With Quote
Old 19th February 2019, 10:35   #2  |  Link
tebasuna51
Moderator
 
tebasuna51's Avatar
 
Join Date: Feb 2005
Location: Spain
Posts: 5,473
Remember forum rule 8:

Quote:
8) No cross posting. Post your message once, to the appropriate forum and nowhere else or it will be locked or deleted without warning.
The same post deleted in AviSynth+ thread.
__________________
BeHappy, AviSynth audio transcoder, in Doom9 forums. NicAudio, BassAudio, audio decoders.
tebasuna51 is offline   Reply With Quote
Old 19th February 2019, 12:04   #3  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 4,095
Avisynth does not provide a function to query the number of CPU cores (yet).
A simple plugin could return the number of cores although that function should really be in the core.
__________________
Groucho's Avisynth Stuff

Inside every older person is a younger person wondering what the hell happened.
Groucho2004 is offline   Reply With Quote
Old 19th February 2019, 13:16   #4  |  Link
silverwing
Registered User
 
Join Date: Nov 2017
Location: Russia, Nizhny Novgorod
Posts: 23
Quote:
Originally Posted by Groucho2004 View Post
Avisynth does not provide a function to query the number of CPU cores (yet).
Sad. It would be a very useful feature.

Quote:
Originally Posted by Groucho2004 View Post
A simple plugin could return the number of cores although that function should really be in the core.
Is there such a plugin, Dear colleague?

Something like:
Code:
threads=CoreNumbers() + some_constant
Prefetch(threads)
or simple:
Code:
Prefetch( CoreNumbers() + some_constant )

Last edited by silverwing; 19th February 2019 at 13:19.
silverwing is offline   Reply With Quote
Old 19th February 2019, 13:32   #5  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,773
Poke this gentelman here https://forum.doom9.org/showthread.p...97#post1866097
Atak_Snajpera is offline   Reply With Quote
Old 19th February 2019, 14:07   #6  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,383
I thought that GetSystemEnv had already been added to Avs+, but not as of r2772.

but, can use this [from RT_Stats]
Code:
/*
    RT_GetSystemEnv(String envname)
     Returns a string from the System Environment with the env name of the string arg.
     Returns "", if cannot find environment variable of given name.
     eg, TEMP=RT_GetSystemEnv("TEMP") # to get TEMP folder path.
     and eg
     ComSpec=RT_GetSystemEnv("ComSpec") # might return "C:\WINDOWS\system32\cmd.exe"
     To see the environment variables from command console, type 'set'.
     http://en.wikipedia.org/wiki/Environment_variable


     See Also StickBoy GetSystemEnv() :- http://www.avisynth.nl/users/stickboy/

*/


Colorbars(width=960).killaudio

OS        = RT_GetSystemEnv("OS")
PROC_ARCH = RT_GetSystemEnv("PROCESSOR_ARCHITECTURE")
PROC_ID   = RT_GetSystemEnv("PROCESSOR_IDENTIFIER")
NUMPROC   = RT_GetSystemEnv("NUMBER_OF_PROCESSORS")

S = "OS                     = '" + OS        + "'\n" +
  \ "Processor Architecture = '" + PROC_ARCH + "'\n" +
  \ "Processor Identifier   = '" + PROC_ID   + "'\n" +
  \ "Processor Count        = '" + NumProc   + "'"


Subtitle(S,Font="Courier New",lsp=0)
NOTE, you may get different results if called from x86 or x64 process [from PROCESSOR_ARCHITECTURE].
EDIT: Dont know if NUMBER_OF_PROCESSORS includes hyperthreading or not, ie logical as well as physical cores (suspect that it will include both).

__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 19th February 2019 at 15:44.
StainlessS is offline   Reply With Quote
Old 19th February 2019, 14:47   #7  |  Link
silverwing
Registered User
 
Join Date: Nov 2017
Location: Russia, Nizhny Novgorod
Posts: 23
Quote:
Originally Posted by StainlessS View Post
I thought that GetSystemEnv had already been added to Avs+, but not as of r2772.

but, can use this
Thank you! My CPU has hyperthreading. Your script shows correctly the number of logical cores. Good workaround!

BTW.

I can't find in the source code (Avisynth Plus, downloaded from github) are internal functions like GetProcessInfo(), etc.
(It is possible that looking in the wrong direction. Someone tell me where to look?)

Otherwise, I would have put something like:
Code:
GetLogicalProcessorInformation
(Microsoft Sysinfo API).

Name? Simple! GetLogicalProcessorCores (). Add this code, compile, run.
And "In the bag"!

Last edited by silverwing; 19th February 2019 at 15:01.
silverwing is offline   Reply With Quote
Old 19th February 2019, 15:02   #8  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,773
Keep in mind that AviSynth MT works better with prefetch equal to number of cores not logical cpus.
Atak_Snajpera is offline   Reply With Quote
Old 19th February 2019, 15:29   #9  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,383
From Avisynth.cpp

Code:
size_t  __stdcall ScriptEnvironment::GetProperty(AvsEnvProperty prop)
{
  switch(prop)
  {
  case AEP_FILTERCHAIN_THREADS:
    return (prefetcher != NULL) ? prefetcher->NumPrefetchThreads()+1 : 1;
  case AEP_PHYSICAL_CPUS:
    return GetNumPhysicalCPUs();
  case AEP_LOGICAL_CPUS:
    return std::thread::hardware_concurrency();
  case AEP_THREAD_ID:
    return 0;
  case AEP_THREADPOOL_THREADS:
    return thread_pool->NumThreads();
  case AEP_VERSION:
    return AVS_SEQREV;
  default:
    this->ThrowError("Invalid property request.");
    return std::numeric_limits<size_t>::max();
  }

  assert(0);
}
Also see GetProperty in avisynth.h

EDIT: Not available for Avs Standard v2.60 (dont know about v2.61).
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 19th February 2019 at 15:52.
StainlessS is offline   Reply With Quote
Old 19th February 2019, 15:34   #10  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,434
Quote:
Originally Posted by Atak_Snajpera View Post
Keep in mind that AviSynth MT works better with prefetch equal to number of cores not logical cpus.
This has been discussed before, but in my experience this is not always true...
My CPU is an Intel Core i5 with 2 physical cores plus Hyperthreading, so it has 4 logical cores. The machine has 8GB RAM, and I use only AVS+ 32-bit.

Myrsloik proposed this formula for the number of threads:
Quote:
It's possible that the default number of threads should be something like max(physical cores, min(logical threads, 8)) for x86
For my CPU this would mean 4. And indeed with AVS+ MT 32-bit Prefetch(4) so far has always been faster than Prefetch(2).


I use Stickboy's GetSystemEnv plugin to determine the NUMBER_OF_PROCESSORS environment variable which is 4 (win7-64). And this can be used directly with the Prefetch command. Like this:
Quote:
Prefetch(Max(Int(Value(GetSystemEnv("NUMBER_OF_PROCESSORS")))/2, MIN(Int(Value(GetSystemEnv("NUMBER_OF_PROCESSORS"))),8)))
I know it looks ugly, but the typecast from string to int is required. Another problem is that the command assumes that the CPU has HyperThreading and HyperThreading is active.


Cheers
manolito

Last edited by manolito; 22nd February 2019 at 00:24.
manolito is offline   Reply With Quote
Old 19th February 2019, 15:35   #11  |  Link
silverwing
Registered User
 
Join Date: Nov 2017
Location: Russia, Nizhny Novgorod
Posts: 23
Atak_Snajpera, This is a purely technical dispute. And I will not go into it, although I tried both the number of logical cores and the number of physical cores.
The question is different - to get the total number of processor(s) cores. StainlessS's script works. I just have to wait until this feature is added to the core of Avisynth Plus.
I hope this happens soon.
silverwing is offline   Reply With Quote
Old 22nd February 2019, 00:47   #12  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,434
Quote:
Originally Posted by Atak_Snajpera View Post
Keep in mind that AviSynth MT works better with prefetch equal to number of cores not logical cpus.
A couple more speed benchmarks show that (at least on my machine) the results are different whether you only check the script with AVSMeter, or if you check the overall speed together with an encoder (I use X264 32-bit, the --threads param was not used so all available cores were used by X264).

It happened quite often that the script by itself was faster when using the number of physical cores for prefetch. But the overall speed was always faster when using all cores. (My Core i5 has 2 physical cores plus Hyperthreading, so I used either 2 or 4 as the prefetch values).


I am still thinking of a simple way to translate Myrsloik's formula into an AVS script automatically. My current prefetch command is this:
Quote:
Prefetch(Max(Int(Value(GetSystemEnv("NUMBER_OF_PROCESSORS")))/2, MIN(Int(Value(GetSystemEnv("NUMBER_OF_PROCESSORS"))),8)))
This only works if HyperThreading is active, though.
Getting the logical cores is easy, I can use the environment variable NUMBER_OF_PROCESSORS. The problem is getting the number of physical cores.

So far I have come up with this:
The WMIC command can report the physical cores. I found a batch file which will export this value into an environment variable:

Quote:
set PHYSICAL_CORES=0 & for /f "tokens=2 delims==" %%d in ('wmic cpu get NumberOfCores /value ^| findstr "="') do @set /A PHYSICAL_CORES+=%%d >NUL
Then the GetSystemEnv plugin can retrieve this variable.

To run this batch file I could use CallCmd. Haven't tried it yet, but I am not too sure if the environment variable which the batch file creates will be in the same process environment as the AVS script.

Anyone with a better (and simpler) idea?


Cheers
manolito


//EDIT//
After getting some test results from Selur and doing some serious thinking I decided to make it simpler. Going above 8 prefetch threads for AVS+ should only make sense in some rare situations, so I simplified Myrsloik's original algo:
Quote:
max(physical cores, min(logical threads, 8))
to
Quote:
min(logical threads, 8)
This will use the number of logical threads, but only up to 8. This should give good results in most cases. The prefetch command would be:
Quote:
Prefetch(Min(Int(Value(GetSystemEnv("NUMBER_OF_PROCESSORS"))),8))

Last edited by manolito; 22nd February 2019 at 18:42.
manolito is offline   Reply With Quote
Old 22nd February 2019, 01:35   #13  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 4,095
Quote:
Originally Posted by manolito View Post
The problem is getting the number of physical cores.
This has become even more difficult with contemporary CPUs such as Ryzen. I went back to using Veselin Georgiev's libcpuid C-library in my programs which reports that value reliably and is frequently updated with the latest CPUs.

Looking at some comments (about Ryzen in this case) in the source of libcpuid shows how complex things have become:
Quote:
/* Ryzen 3 has SMT flag, but in fact cores count is equal to threads count.
Ryzen 5/7 reports twice as many "real" cores (e.g. 16 cores instead of 8) because of SMT. */
/* On PPR 17h, page 82:
CPUID_Fn8000001E_EBX [Core Identifiers][15:8] is ThreadsPerCore
ThreadsPerCore: [...] The number of threads per core is ThreadsPerCore+1 */
That nice library could of course be integrated into a plugin or Avisynth itself.
__________________
Groucho's Avisynth Stuff

Inside every older person is a younger person wondering what the hell happened.
Groucho2004 is offline   Reply With Quote
Old 22nd February 2019, 12:14   #14  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,773
Quote:
Originally Posted by manolito View Post
This has been discussed before, but in my experience this is not always true...
My CPU is an Intel Core i5 with 2 physical cores plus Hyperthreading, so it has 4 logical cores. The machine has 8GB RAM, and I use only AVS+ 32-bit.

Myrsloik proposed this formula for the number of threads:


For my CPU this would mean 4. And indeed with AVS+ MT 32-bit Prefetch(4) so far has always been faster than Prefetch(2).


I use Stickboy's GetSystemEnv plugin to determine the NUMBER_OF_PROCESSORS environment variable which is 4 (win7-64). And this can be used directly with the Prefetch command. Like this:


I know it looks ugly, but the typecast from string to int is required. Another problem is that the command assumes that the CPU has HyperThreading and HyperThreading is active.


Cheers
manolito
With cpu with higher number of cores for example 8 if I use prefetch with value higher than 8 then my script starts to choke and thus reducing average fps. This happens on Ryzen 7 and on my Xeon e5-2690.
Atak_Snajpera is offline   Reply With Quote
Old 22nd February 2019, 12:35   #15  |  Link
silverwing
Registered User
 
Join Date: Nov 2017
Location: Russia, Nizhny Novgorod
Posts: 23
Quote:
Originally Posted by Groucho2004 View Post
That nice library could of course be integrated into a plugin or Avisynth itself.
"Avisynth itself" - best method. Out-of-box - no extra stuff!
I hope that a respected printerf will heed our wishes.
silverwing is offline   Reply With Quote
Old 22nd February 2019, 13:17   #16  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 4,095
Quote:
Originally Posted by silverwing View Post
"Avisynth itself" - best method. Out-of-box - no extra stuff!
I hope that a respected printerf will heed our wishes.
And if printerf doesn't, pinterf might.
__________________
Groucho's Avisynth Stuff

Inside every older person is a younger person wondering what the hell happened.
Groucho2004 is offline   Reply With Quote
Old 22nd February 2019, 13:34   #17  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 2,434
Quote:
Originally Posted by Atak_Snajpera View Post
With cpu with higher number of cores for example 8 if I use prefetch with value higher than 8 then my script starts to choke and thus reducing average fps. This happens on Ryzen 7 and on my Xeon e5-2690.
According to Myrsloik's formula for such a CPU with 8 physical cores the result for Prefetch would be 8. What I am not sure about is the behavior for CPUs with a lot more than 8 physical cores, let's say 16.

With 16 physical cores the formula will give out 16, doesn't matter if HyperThreading is active or not. Is this good, or shouldn't the max number of Prefetch threads be capped, maybe to a maximum of 8?


Cheers
manolito
manolito is offline   Reply With Quote
Old 22nd February 2019, 14:20   #18  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 4,095
Quote:
Originally Posted by silverwing View Post
"Avisynth itself" - best method. Out-of-box - no extra stuff!
Using external libraries also has its drawbacks. You have to regularly check for updates (which in case of libcpuid is quite frequent since AMD/Intel keep releasing new CPUs). This adds to the workload for the person who maintains the project that uses the library. Also, what if the developer of the library stops development? It can be a PITA.
__________________
Groucho's Avisynth Stuff

Inside every older person is a younger person wondering what the hell happened.
Groucho2004 is offline   Reply With Quote
Old 22nd February 2019, 14:47   #19  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,773
Quote:
Originally Posted by manolito View Post
According to Myrsloik's formula for such a CPU with 8 physical cores the result for Prefetch would be 8. What I am not sure about is the behavior for CPUs with a lot more than 8 physical cores, let's say 16.

With 16 physical cores the formula will give out 16, doesn't matter if HyperThreading is active or not. Is this good, or shouldn't the max number of Prefetch threads be capped, maybe to a maximum of 8?


Cheers
manolito
Like somebody explained to me reduction in speed is probably due to cache misses. I predict that on Threadripper we would see the same degradation in script processing.
Atak_Snajpera is offline   Reply With Quote
Old 22nd February 2019, 15:05   #20  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 1,642
Quote:
Originally Posted by Groucho2004 View Post
And if printerf doesn't, pinterf might.
We should be using pinterf_s these days.
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Reply

Tags
avisynth processor cores

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.