Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th February 2017, 14:41   #21  |  Link
Danette
Registered User
 
Join Date: Apr 2013
Posts: 179
You can find (hopefully) the mpg file at:

https://www.sendspace.com/file/j4w9y1

It has every 5th frame duplicated (I IVTC’d an NTSC file, but did not decimate for purposes of the test). This is not anime.

When I run the first pass through Vdub, I get the attached dupinfo and times data. As you can see, each frame is listed in the sequence, i.e.; no duplicates found by ExactDeDub. Same thing happens when I run the source AVI file. I have also found that I need to re-load the ExactDeDub filter into AvsPmod each time I make a script change to avoid an error message, if that’s meaningful.

I have not tried removing the duplicates with ExactDeDup, as yet, since the dupinfo data seems wrong. I assume that I would change the firstpass variable to “False” and re-run the file(?).

ndjamena: I didn't try your mod, as this is not anime.
Attached Files
File Type: txt dupinfo.txt (5.5 KB, 14 views)
File Type: txt times.txt (18.3 KB, 13 views)
Danette is offline   Reply With Quote
Old 17th February 2017, 15:03   #22  |  Link
ndjamena
Registered User
 
Join Date: Sep 2012
Posts: 366
ExactDeDup removes frames that are EXACTLY the same. IVTC’d frames are NOT exactly the same, the MPEG encoding process plus the IVTC’d leaves the "duplicate" frames slightly different so ExactDeDup doesn't touch them.

ReDup IS DeDup, you can use it to create a VFR MKV just like you can with DeDup. The difference for your purposes would be the ability to provide a separate clip for detection purposes or a second one if desired. The point of that is that you can provide a denoised clip for detection purposes without needing the denoising applied to the output. The secondary detection clip is so you can use a different denoising process for the comparison...

something like
fft3dfilter(bt=2)
for the first detection clip and
reverse.fft3dfilter(bt=2).reverse
for the second

I would have added more options if I'd been able to continue.

All of which is completely besides the point. The point being, EXACT DEDUP IS THE WRONG FILTER FOR YOUR PURPOSES.

DeDup or even just the decimation features of TDecimate would be more appropriate.

If it's simple NTSC with every 5th frame duplicated I'm not sure why you'd need any of these filters anyway, you'd be better of simply changing the frame rate by removing the 5th frames. ExactDeDup would be completely unnecessary for that, as would be any flavour of DeDup.
ndjamena is offline   Reply With Quote
Old 17th February 2017, 15:35   #23  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,018
Ndjamena,
Check out framesel, framerep.
Framesel can extract a list of frames, framerep can put them back from whence they came.
All your repdupe thing needs to do create the frames file.

Mobile.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 17th February 2017, 20:33   #24  |  Link
Danette
Registered User
 
Join Date: Apr 2013
Posts: 179
Thanks to you both. It sounds like non-anime duplicates are not exact and, so I can see where ExactDeDup won’t work. I tried ReDup, ndjamena, but stumble on what to enter for the “counts” variable (no instructions in the download). I get an error message, in AvsPmod, saying “the named argument “counts” to ReDup had the wrong type.” What should be entered here?

I also tried DeDup, but get the error message, in AvsPmod, that “DeDup: failed to open logfile” with the script written as: DeDup(log=test.dup.txt), which is patterned on the authors instructions for version .17. I tried looking around for a newer release, but couldn’t locate one.

Regarding your question:
Quote:
Originally Posted by ndjamena View Post
I'm not sure why you'd need any of these filters anyway, you'd be better of simply changing the frame rate by removing the 5th frames.
You’re right, for this example I can remove that 5th frame with Tdecimate, but I created that simply to verify functionality for the different functions you two have offered. My intent is to use something that works on a badly mangled file that has random duplicates, sometimes 3-4 sequential duplicates that Tdecimate won’t address.
Danette is offline   Reply With Quote
Old 18th February 2017, 03:50   #25  |  Link
ndjamena
Registered User
 
Join Date: Sep 2012
Posts: 366
TDecimate Mode 3 and Mode 5 both can create VFR Matroska files.

DeDup(log=test.dup.txt)

Do you mean DeDup(log="test.dup.txt")?

I hope so. DeDup only throws that error if "log" equals NULL.

It shouldn't be hard to modify DeDup to have a default log file name if one isn't provided.

ReDup is as buggy as hell, "counts" is only used for actual ReDup when you're returning frames to the video.

These are the functions:

env->AddFunction("DeDup", "c[threshold]f[threshold2]f[range2]i[trigger2]f[show]b[dec]b[maxcopies]i[maxdrops]i[decwhich]i[log]s[times]s[counts]s[timesin]s[ovr]s[debug]s", Create_DeDup, 0);
env->AddFunction("ReDup", "c[counts]s", Create_ReDup, 0);
env->AddFunction("DupMC", "c[chroma]b[log]s[search]i[alt]c[alt2]c[maxaltthresh]f[minaltthresh]f", Create_DupMetric, 0);

Decimation function is still called DeDup, the ReDup function reverses it and "counts" is the file that stores which frames were removed. It's optional, you can provide "counts" or "times" or both...

Anyway, I believe your problems with DeDup is are being cause by you providing a null variable as an argument.

This is the TDecimate instructions:

https://avisynth.org.ru/docs/english..._tdecimate.htm

DeDup:
http://akuvian.org/src/avisynth/dedup/dedup.txt

The problem with DeDup is that it tends to be impossible to find a value that removes the dups but doesn't remove non-dup frames as well, since the values overlap. I was trying to correct that in ReDup but I think the dll is useless as it is.
ndjamena is offline   Reply With Quote
Old 19th February 2017, 00:29   #26  |  Link
Danette
Registered User
 
Join Date: Apr 2013
Posts: 179
DeDup

Well, with the help of StainlessS and ndjamena, I found the solution to convert my badly telecined film into progressive that eliminates all duplicates, provides smooth video play with synced audio. I thought I’d post the results in case others find it useful.

This was an NTSC film that had several telecined patterns (3:2 and 2:1) as well as duplicate frames between some of the telecined frames. Although there may be better ways to do this (and I would be interested if there are any), here is what worked:

1. Before any IVTC, I used DeDup on the source to remove all duplicates between the telecined frames
2. On the resulting file, I then applied TFM and Tdecimate(cycler=2) - the default cycle=5 takes care of both telecine patterns).

My next step is to figure out how to do this in a batch process, but that should be simple at this point.
Danette is offline   Reply With Quote
Old 19th February 2017, 19:26   #27  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,179
I only skimmed the thread so I don't know if you publicly posted a sample of the video you were trying to IVTC. However, with patterns that don't repeat, or which have patterns that change, you can often get a good result by dramatically increasing the cycle size (to 20 or more) and then changing the TDecimate paramters to remove the most similar from each cycle.
johnmeyer is offline   Reply With Quote
Old 3rd October 2017, 11:47   #28  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
Hello,

Would like to know if by any chance there is a x64 version of the modified .dll file done by StainlessS ?
There is a x64 for the original .dll in version.03 but for this one i have to run it using MP_Pipeline and it probably hurts the performances on the first pass :/ .

Another question is how the "maxdupcount" settings works exactly, probably my English defaults me to understand properly this :
"Default 20. The number of maximum consecutive duplicate frames corresponding to a single frame of the output; i.e. a maximum of (maxdupcount - 1) frames will be removed from a run of duplicates before another frame is kept."

If i set it to let's say 10 and i have a sequence with 50 exact same frames it will keep one frame then delete (10-1) identical frames then keep one frame and again delete (10-1) identical frames and keep another frame etc ... and in the end will leave only 5 identical frames in the video sequence output, or am i understanding incorrectly ?

If i want no duplicate frames at all i just need to set this value really high or maybe something like "-1" will be more efficient ?

( Also does this parameters have any sort of impact on the performances during the first pass please ? if i set it really high like 500 or more will it hurts ? )

i done only a few tests for now as it takes really a long time to do the first pass on the footage i tested so i would like to be sure before starting to apply this on my projects ( and sorry if the question sounds dumb )

Thanks in advance.
Yanak is offline   Reply With Quote
Old 4th October 2017, 10:49   #29  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,018
ill try knock up 64bit within coupla days.
mobile.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 4th October 2017, 13:51   #30  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
It will be awesome if you can do it ,

thanks in advance .
Yanak is offline   Reply With Quote
Old 5th October 2017, 07:42   #31  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,018
Yanak, see MediaFire or SendSpace below in my sig.
Set maxdupcount to a big number eg 1000000 to switch it off.

Code:
ExactDedup 0.05 (2017-05-17) (StainlessS)
  Using Avs+ Header.
  Convert to VS2008, x64 compile.
EDIT: Oops, shoulda been (2017-10-05)
EDIT: Avisynth v2.6 only (did not do v2.58 plug), and Avs+ x64.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 5th October 2017 at 08:05.
StainlessS is offline   Reply With Quote
Old 5th October 2017, 11:01   #32  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
Thanks a lot for this,
as expected it goes a bit faster than using it in Mp_Pipeline, that's great .

Only thing different i noticed is that previously using Mp_Pipline in my script launched from Staxrip program it was creating the text files and using them in my Temp destination folder directly, now it creates the txt files directly inside the Staxrip folder, but during the second pass it cannot find them and i get an error message telling me it can't read the files.

But this is fixed if indicate a full path for the txt files creation on 1st pass and use the same full path for the .txt files during the second pass so not a problem at all.
Not sure what is the behavior using the 0.4 version with avisynth x86, outside MP_Pipeline i mean.

Thanks again a lot for this, you can't imagine how useful it will be for me.
Yanak is offline   Reply With Quote
Old 5th October 2017, 11:21   #33  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,018
Functionality did not change at all from v0.4, simple recompile (although v2.6 and using avs+ header now).

I did not play with paths at all, paths used to open files are exactly as provided, in all versions.

You can use in script (req RT_Stats, 64bit beta in your case), RT_GetFullPathName(filename) in any script to change relative path to full path.
If you use RT_Stats 64bit, and have any probs, let me know.

EDIT: I'll probably have XP64 set up to compile/test 64 bit stuff, within a week or two. (initially set up now, but no service packs or
other additions).
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 5th October 2017 at 12:15.
StainlessS is offline   Reply With Quote
Old 5th October 2017, 12:14   #34  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
Thanks,

Probably it worked differently when I was using it in Mp_Pipeline compared to when used directly, never tried in a x86 the 0.4 version but i'll give a try for the 0.3 native x64 version later to compare behaviors, anyways not a problem in this case, just had to setup a full path for my filter and it works now very nicely with a speed boost on the files i tested so it's perfect.

Thanks again for all, this thing will serve me nicely
Yanak is offline   Reply With Quote
Old 5th October 2017, 12:40   #35  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
Just run a couple of tests using the 0.3 x64 version and it is the same indeed, so it is Mp_Pipleine that acts differently when creating/storing or reading the txt files, apparently creating them where the avs script is called and reading them there too, will keep this in mind for the next times i'll need similar things in Mp_Pipeline,

Thanks again.
Yanak is offline   Reply With Quote
Old 14th October 2017, 13:25   #36  |  Link
Ben_Nicholls
Registered User
 
Join Date: Aug 2017
Location: Charlbury, UK
Posts: 18
Rather than outputting the duplicate frames to a timecode file to merge with an MKV creating a VFR video would it be possible to set it to use SVPflow to interpolate them to a specified CFR, thus creating smooth (non-ghosted) interpolated new frames?

It could also be expanded upon in order to allow creation of video at whatever desired rate you want without having to use SVP twice (once to remove duplicates, then again to change the FPS to the desired output).

So for example if you have a video that was shot at 24fps, but has been encoded at 25fps (creating 1 duplicate every second) but want to create 30fps CRF content for YouTube, you could set it to interpolate the detected duplicate frames, and then interpolate all the frames until the desired frame rate was achieved.
Ben_Nicholls is offline   Reply With Quote
Old 14th October 2017, 16:01   #37  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,018
@Ben_Nicholls,
Assuming that above is aimed at me, sorry but no, It's not my plug and I dont really wanna adopt it (I just spotted several bugs and fixed them).
I did not touch times code stuff and have no idea how it works, nor do I use SPV, and likewise know nothing of its working.
Got way too much on my plate already and am actually trying to get away from Avisynth stuff.
Again sorry.

EDIT: This any good:- https://forum.doom9.org/showthread.p...65#post1764865
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 14th October 2017 at 16:26.
StainlessS is offline   Reply With Quote
Old 14th October 2017, 16:51   #38  |  Link
Ben_Nicholls
Registered User
 
Join Date: Aug 2017
Location: Charlbury, UK
Posts: 18
Quote:
Originally Posted by StainlessS View Post
@Ben_Nicholls,
Assuming that above is aimed at me, sorry but no, It's not my plug and I dont really wanna adopt it (I just spotted several bugs and fixed them).
I did not touch times code stuff and have no idea how it works, nor do I use SPV, and likewise know nothing of its working.
Got way too much on my plate already and am actually trying to get away from Avisynth stuff.
Again sorry.

EDIT: This any good:- https://forum.doom9.org/showthread.p...65#post1764865
No problem mate, I dont expect free coding to cater for my every editing whim! If I remember correctly I had some issues getting the script you linked to working, I will give it another try. Recently did a massive overhaul of my plug-ins and switched from the Avisynth 2.6 MT to Avisynth+ 2508, so I will try it out again. I will let you know how it turns out!
Ben_Nicholls is offline   Reply With Quote
Old 25th July 2018, 16:18   #39  |  Link
JoyBell
Registered User
 
Join Date: Jun 2017
Posts: 10
Completely Lost

I have been reading this thread for a few days. I have come to the conclusion that I am missing the basic information as to where to start.

I have no idea where to attempt to use this code.

Let me back up.

I use StaxRip and would like to use ExactDeDup to take in a CFR source file and output a VFR file. I am thinking on Animatics or perhaps Cartoons as many frames are the same.

I don't see how ExactDeDup can be use as an Avisynth filter in Staxrip.

Is this a Script I run as a separate tool to just remux the source and then use that in Staxrip? If so, how do I do that?
I have been asking in my groups discord channel for a few days now and have gotten no where. Thanks for help!
JoyBell is offline   Reply With Quote
Old 26th July 2018, 02:03   #40  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
Hello,

Will try to explain wiht all steps, hopefully it will do the work.

Previously in Staxrip i was using the event commands combined with a template to make the 1st pass and scan file for dupes, but since yesterday it is possible to now execute the filter in a single pass with the help of a little new plugin found here : https://forum.doom9.org/showthread.php?t=175616 ( Thx again StainlessS )

Download the file here : http://www.mediafire.com/file/xyvpru...80724.zip/file
Inside the zip file Avisynth+_x64 folder you find the ForceProcess_x64.dll file, extract it and put it inside avisynth+ plugin64+ folder, located here :
"C:\Program Files\AviSynth+\plugins64+"

This done we will create a filter inside Staxrip to use ExactDeDup, in the list of filters make a right click and then on the dropdown menu click on profiles, like here :



On the new window scroll a bit until you see the [Misc] category , and there you can copy paste the following :
Code:
Remove Dup Frames =
    if (Exist("%working_dir%times.txt")) {
    ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
    maxdupcount=1000000, keeplastframe=false, show=false)	
    }
    else {
    ExactDedup(firstpass=true, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
    maxdupcount=1000000, keeplastframe=false, show=false)
    ForceProcess()
    ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
    maxdupcount=1000000, keeplastframe=false, show=false)
    }
It should then look like this :

If needed adjust a bit the code for things like keeplasframe setting etc.
( Filter can be put in another category, it's up to you, just found Misc one to fit )

Once this done you have now a new filter added into StaxRip and can now select the new created filter and add it to the list :


When adding this filter for the first time into a project it may look like Staxrip is freezing a bit and it can take a bit of time before it recovers, in fact immediately when you add the filter the new little plugin "ForceProcess" starts his work and analyze the video file to find the duplicate frames and create the data text files, so just be a bit patient in case it takes too long, but the filter code i provided is made so that if the timestamps "times.txt" file already exist it will not scan it again, if the scan was done and the file is present on the temp folder it will just use the tiemstamp file directly.

Last thing to do is on the MKV container to set up the path for the timestamps, click on "Container Options" :


And then go to the "Options" Tab and select the times.txt file, by default clicking the little button like in the pic below should send you directly to the folder where the file is located :

just select the times.txt file there.

I did not found a way to automate this part yet, adding additional custom switches in a dedicated MKV profile doesn't seems to work for this particular one so you will have to select the timestamps manually each time, unless i find a solution for this one of those days...


After this you are done, filter is created, when needed just add the "Remove Dup Frames" filter, wait a little for video to be scanned for duplicate frames and finally go to the mkv container options and select the times.txt file.

From there the rest is up to you for encoding parameters etc, once encoded the MKV should be a variable framerate file where all duplicate frames have been eliminated and the container having the mkv timestamps.

Hope it helps a bit

PS : forgot to say but of course you also need to download ExactDeDup file and put the win64 dll in the same plugin64+ folder as the dll in the top of this post

Last edited by Yanak; 26th July 2018 at 02:44.
Yanak is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.