Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 9th August 2019, 07:23   #841  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,876
Yes, I mean that only for indexing, none of the filters for the final video need to be included ("minimal" in the sense to only load the [audio and] video). Like Atak_Snajpera suggested.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 9th August 2019, 09:25   #842  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,359
Quote:
Originally Posted by LigH View Post
Yes, I mean that only for indexing, none of the filters for the final video need to be included ("minimal" in the sense to only load the [audio and] video). Like Atak_Snajpera suggested.
Right. As I mentioned, since one most likely already has a script, indexing will be triggered by using avsr with the switch "-i" which simply displays the clip properties.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 9th August 2019, 19:37   #843  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 543
Quote:
Originally Posted by Groucho2004 View Post
What do you mean by "minimal script"?
Quote:
Originally Posted by Groucho2004 View Post
Right. As I mentioned, since one most likely already has a script,
Having non-minimal script could significantly increase loading time if complicated filters are used, such as waifu or QTGMC.

I always use a bare source script to generate the index, and later a fully functional script to do the actual encode.

Anyway, you can always use avs4x26x like I mentioned above to automatically create the script (in memory) and generate the index.
MeteorRain is offline   Reply With Quote
Old 11th August 2019, 17:23   #844  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 598
Quote:
Originally Posted by FranceBB View Post
Uh... may I ask you an x86 version, please? :P
Link updated in the previous post. The other thread as well.
HolyWu is offline   Reply With Quote
Old 11th August 2019, 19:04   #845  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 5,876
Praise the HolyWu!
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 11th August 2019, 21:22   #846  |  Link
Taurus
Registered User
 
Taurus's Avatar
 
Join Date: Mar 2002
Location: Krautland
Posts: 856
Quote:
Originally Posted by HolyWu View Post
Link updated in the previous post. The other thread as well.
Taurus is offline   Reply With Quote
Old 13th August 2019, 13:54   #847  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,910
I have another feature request.
Store index file in compressed form. Currently index file can be extremely large in comparison to .ffindex.
video.mkv is remuxed Alien Covenant DVD (2h movie). Just take a look how tiny .ffindex is


Large index file is not a problem if you load that file locally from SSD. Normally It should take less than 1s. However it is a different story if you have to load that file directly via network 100Mbps LAN or wi-fi n.

Quote:
video=LWLibavVideoSource("\\XEON-PC\RipBot264temp\job1\video.mkv",threads=0,cachefile="\\XEON-PC\RipBot264temp\job1\video.mkv.lwi")
That large uncompressed index file significantly delays starting of encoding process. It gets even worse if more than one server is downloading that file.

example


Now 6 servers are downloading 200MiB index file at the same time. (1.2GiB total). Assuming that real transfer rate for LAN100Mbps is only ~10MiB/s we can easily predict that encoding process will be delayed by about 2 minutes!

So my suggestion is to always compress index file with some basic zip algorithm. LZMA (normal) can compress that index file to ~5MiB while Zip to ~10MiB.
Time saving will be immediately noticeable. Delay will be measured in seconds instead of minutes.

Last edited by Atak_Snajpera; 13th August 2019 at 14:20.
Atak_Snajpera is offline   Reply With Quote
Old 13th August 2019, 15:45   #848  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 1,956
Quote:
Originally Posted by Atak_Snajpera View Post
I have another feature request.
Store index file in compressed form. Currently index file can be extremely large in comparison to .ffindex.
video.mkv is remuxed Alien Covenant DVD (2h movie). Just take a look how tiny .ffindex is


Large index file is not a problem if you load that file locally from SSD. Normally It should take less than 1s. However it is a different story if you have to load that file directly via network 100Mbps LAN or wi-fi n.



That large uncompressed index file significantly delays starting of encoding process. It gets even worse if more than one server is downloading that file.

example


Now 6 servers are downloading 200MiB index file at the same time. (1.2GiB total). Assuming that real transfer rate for LAN100Mbps is only ~10MiB/s we can easily predict that encoding process will be delayed by about 2 minutes!

So my suggestion is to always compress index file with some basic zip algorithm. LZMA (normal) can compress that index file to ~5MiB while Zip to ~10MiB.
Time saving will be immediately noticeable. Delay will be measured in seconds instead of minutes.
FFMS2 uses zlib and preprocessing of the data (store the differences between timestamps/frame numbers instead of absolute timestamps/frame numbers). Huffman coding based things already work wonders on that kind of data. Just a hint.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 13th August 2019, 16:59   #849  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 619
Quote:
Originally Posted by HolyWu View Post
Link updated in the previous post. The other thread as well.
Thank you!
__________________
Broadcast Encoder
LinkedIn
FranceBB is offline   Reply With Quote
Old 14th August 2019, 19:56   #850  |  Link
Morku
Registered User
 
Join Date: Jul 2012
Posts: 161
Edit: Nevermind.

Last edited by Morku; Yesterday at 20:43.
Morku is offline   Reply With Quote
Old 14th August 2019, 20:37   #851  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: Germany
Posts: 5,628
Did you verify that it actually outputs double width? There was a patch for native high bit depth support.
stax76 is online now   Reply With Quote
Old 14th August 2019, 23:47   #852  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 543
Quote:
Originally Posted by Atak_Snajpera View Post
I have another feature request.
Store index file in compressed form. Currently index file can be extremely large in comparison to .ffindex.
video.mkv is remuxed Alien Covenant DVD (2h movie). Just take a look how tiny .ffindex is


Large index file is not a problem if you load that file locally from SSD. Normally It should take less than 1s. However it is a different story if you have to load that file directly via network 100Mbps LAN or wi-fi n.



That large uncompressed index file significantly delays starting of encoding process. It gets even worse if more than one server is downloading that file.

example


Now 6 servers are downloading 200MiB index file at the same time. (1.2GiB total). Assuming that real transfer rate for LAN100Mbps is only ~10MiB/s we can easily predict that encoding process will be delayed by about 2 minutes!

So my suggestion is to always compress index file with some basic zip algorithm. LZMA (normal) can compress that index file to ~5MiB while Zip to ~10MiB.
Time saving will be immediately noticeable. Delay will be measured in seconds instead of minutes.
Removing audio part (I do all the time) will save tones of space. Not just transferring, but loading and parsing the index itself could take up to minutes for a large file, until you remove the audio part from the index.

It's as simple as removing lines containing Type=1 or Channel. You can use sed to manipulate the file.

Last edited by MeteorRain; 14th August 2019 at 23:54.
MeteorRain is offline   Reply With Quote
Old 15th August 2019, 12:51   #853  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,910
Quote:
Originally Posted by MeteorRain View Post
Removing audio part (I do all the time) will save tones of space. Not just transferring, but loading and parsing the index itself could take up to minutes for a large file, until you remove the audio part from the index.

It's as simple as removing lines containing Type=1 or Channel. You can use sed to manipulate the file.
Thanks for your tip! After processing by sed I got indeed much smaller file



Nevertheless It is still too much for me. (4s extra delay on LAN100). Additionally to your 'hack' I decided to keep that index file in compressed form (LZMA-normal) for encoding server app. So instead of using index file directly now server has to first decompress index file to some temp folder.
Atak_Snajpera is offline   Reply With Quote
Old 17th August 2019, 15:08   #854  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 598
L-SMASH-Works-r935+31-20190820
  • LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
  • Print indexing progress to stderr.
  • Tell lavf to discard unwanted packets so they needn't be demuxed.
  • Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
  • Automatically re-create the index file when the file size of the source file doesn't match.

Last edited by HolyWu; Today at 05:43.
HolyWu is offline   Reply With Quote
Old 17th August 2019, 15:52   #855  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,910
Quote:
Originally Posted by HolyWu View Post
L-SMASH-Works-r935+31-20190817
  • LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
  • Print indexing progress to stdout.
  • Tell lavf to discard unwanted packets so they needn't be demuxed.
  • Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
  • Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.
LOL! You are reading in my mind! I had to always change that from local path (c:\something\...) to network path (\\XEON-PC\...)

Now only one thing is missing. Compression of index file like in FFMS2 index file. Thank you.

Last edited by Atak_Snajpera; 17th August 2019 at 16:11.
Atak_Snajpera is offline   Reply With Quote
Old 17th August 2019, 17:37   #856  |  Link
HolyWu
Registered User
 
HolyWu's Avatar
 
Join Date: Aug 2006
Location: Taiwan
Posts: 598
Quote:
Originally Posted by Atak_Snajpera View Post
Now only one thing is missing. Compression of index file like in FFMS2 index file. Thank you.
Sorry. It probably won't happen, at least from me.
HolyWu is offline   Reply With Quote
Old 17th August 2019, 17:40   #857  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: Germany
Posts: 5,628
@HolyWu

Very exciting improvements. I'm sure it's not only GUI authors that have been waiting for this long time.
stax76 is online now   Reply With Quote
Old 17th August 2019, 17:45   #858  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 930
Quote:
Originally Posted by HolyWu View Post
L-SMASH-Works-r935+31-20190817
  • LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
  • Print indexing progress to stdout.
  • Tell lavf to discard unwanted packets so they needn't be demuxed.
  • Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
  • Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.
THX
Would it be possible to add a gpu parameter? I saw that there is a supported HW decoders list, so maybe just a cuvid_gpu/qsv_gpu = true/false parameter could be added to avoid nvidia intel gpu detection? It should then set the hevc, h264, mjpeg, etc. decoder accordingly.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database || https://github.com/avisynth-repository

Last edited by ChaosKing; 17th August 2019 at 17:47.
ChaosKing is offline   Reply With Quote
Old 17th August 2019, 18:13   #859  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,910
Quote:
Originally Posted by HolyWu View Post
Sorry. It probably won't happen, at least from me.
Ok. No problem.
Atak_Snajpera is offline   Reply With Quote
Old 17th August 2019, 21:30   #860  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Germany
Posts: 619
Thank you for the update, once again!
__________________
Broadcast Encoder
LinkedIn
FranceBB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.