ExactDeDup

[Danette]

Hopefully, someone here knows how to use ExactDeDup. I am trying it, for the first time, and am running into the error message:
"Error requesting frame 239
ExactDeDup: Trying to compute frames out of order. Did you try to seek during processing?"

I've tried many different source videos and it's always the same frame # 239 cited in the message.

The script for the first pass is:
ExactDedup(firstpass=true, dupinfo="dupinfo.txt", times="times.txt", maxdupcount=20, keeplastframe=false)

[StainlessS]

Works for me, however its best to delete both times.txt and dupinfo.txt before you start (if they already exist).
There is at least one bug on the source,

    if (dupfile)
    {
      fclose(dupfile);
      timesfile = NULL; // should be dupfile=NULL;
    }
    if (timesfile)
    {
      fclose(timesfile);
      timesfile = NULL;
    }
    if (framemap)
    {
      delete[] framemap;
      framemap = NULL;
    }

Will leave timesfile open.

What version of Avisynth are you using, and how are you playing file ?

Also, zip has been deleted from the TasVideo thread as linked on wiki, perhaps there is a reason for deletion.
http://avisynth.nl/index.php/ExactDedup

[Danette]

Thanks, StainlessS.

Neither of the txt files exist because it doesn't get that far. I can’t change the code, unless you can give me an idea as to how to do that, since I am simply loading the ExactDedup.dll plugin.

Originally Posted by StainlessS

What version of Avisynth are you using, and how are you playing file ?

I use Avisynth 2.6 and play it through AvsPmod.

Further help would be appreciated.

[StainlessS]

I'm doing mod for 32 bit only.
Fixed several more issues, and will not write file until fully scanned (so you dont have to open in text editor to change mode, would start writing file again).
Also support for v2.6 planar (would not work with YV12, at least would not scan U, and V).

[StainlessS]

Ok Danette, see SendSpace link below this post in my sig. (not on MediaFire, having problems uploading anything).

ExactDedup v0.03
Copyright (c) 2011 Steve Melenchuk, Arick Chan
==============================================

0. Table of Contents
--------------------
  1. Introduction
  2. Usage
  3. Licensing

1. Introduction
---------------
ExactDedup is an AVISynth 2.5 plugin providing a filter intended to remove
frames that are exact duplicates of each other, leaving only the first and
(optionally) last frames of a run intact, and generates a Matroska v2
timecodes file with timing information for the ensuing stream.

Due to the particulars of the AVISynth API, ExactDedup requires two passes
to function efficiently. It could theoretically be designed to run with
one pass, but this would result in a very long startup delay while the
plugin precomputes all of the duplicate frame information and timecodes.

ExactDedup theoretically supports any input colorspace, but has been
tested only with RGB colorspaces.

You will need the Visual C++ 2008 runtimes (i.e. MSVCR90.dll) to be able
to run this plugin.

2. Usage
--------

ExactDedup(clip [, bool firstpass] [, string dupinfo], [, string times]
           [, int maxdupcount] [, bool _keeplastframe] [,bool show])

Options:
- firstpass    : Default true. Controls which pass the plugin is running.

                 The first pass generates a mapping of output frames to
                 input frames (stored in the file specified by dupinfo) and
                 Matroska v2 timecodes (stored in the file specified by
                 times); seeking during this pass is impossible.

                 The second pass generates a video stream containing only
                 the frames specified in the dupinfo file. (Seeking is
                 permissible during this pass.)

- dupinfo      : Default "dupinfo.txt". The name of a temporary file
                 generated by the first pass and read by the second pass;
                 each line corresponds to a frame in the output video
                 stream, and specifies the frame number of the frame in
                 the input stream that will be used as that output frame.

- times        : Default "times.txt". The name of the Matroska v2 timecodes
                 file that will be output by the first pass.

- maxdupcount  : Default 20. The number of maximum consecutive duplicate
                 frames corresponding to a single frame of the output; i.e.
                 a maximum of (maxdupcount - 1) frames will be removed from
                 a run of duplicates before another frame is kept.

- keeplastframe: Default false. Keeps the last frame of a run of duplicates.
                 If a run of duplicates is larger than maxdupcount, this
                 keeps only the last frame of the entire run, not of the
                 individual maxdupcount-size blocks.

- Show         : Default false. If True, shows a little info on second pass.

3. Licensing
------------
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

#####################################################################

v0.4 Mod by StainlessS, 16 Feb 2017.
Avisynth v2.6 plugin.
Fixed a number of bugs.
Added Support for v2.6 Planar (also added U & V chroma scan for YV12, only scanned luma).
Added Show Arg.
Write times and dupinfo files at end of scan,so you dont have to edit script in text editor before second pass.
No runtime dependency.

ssS

EDIT: Avisynth v2.6 32 bit only dll, incl source + original v0.3 source.

[Danette]

Wow! It looks like you did some good work to bring this up to speed.

I tried your link, but can only see four files there, all from 2016.

[StainlessS]

Oops sorry. Was outside of the Public StainlessS folder, its there now. (still having Mediafire problems).

[Danette]

Thanks: downloaded fine.

Running it, the same error message appears, but with two differences now:

1. Instead of frame 279 showing, it is now frame 195 (for any video source - I've tried dgi, directshow and AVI for various videos).

2. Once I receive the error message, above, if I click anywhere in the AvsPmod window, the video appears, without the error message, but there is no processing by ExactDeDup that I can see.

I'm guessing that ExactDeDup is stumbling during the first pass attempt (no dupinfo or times txt files are being generated) and is bypassed when I click after the error message. Before trying your update, it could not be bypassed and I had to shutdown AvsPmod to escape.

[StainlessS]

How does it fair if you use eg VirtualDub ? (perhaps AvsPMod related).

Are you using any kind of Multi-threaded version of Avisynth v2.6 ?

ExactDeDup demands total linear access, if MT accessing frames in scrambled order, it will produce the described error message.

The text files are now only produced after completed successful scan (was a bit of a problem before, as if you load into eg AVSPMod
for second pass [without first editing the avs file for 2nd pass], it would immediately start writing the pass 1 files again).

EDIT: Also, can anyone else reproduce error with AvsPMod ?

[StainlessS]

Danette. Can you try out "TEST_1" version on SendSpace, I've added some debug information
(will require DebugView to see debugging stuff:- https://technet.microsoft.com/en-us/...debugview.aspx )

First few lines of GetFrame()

PVideoFrame __stdcall ExactDedup::GetFrame(int n, IScriptEnvironment* env) {
    dprintf("GetFrame(%d)",n);
    n=(n<0)?0:(n>=vi.num_frames)?vi.num_frames-1:n;     // Range limit n frame to valid range
    if (firstpass) {
        if(framemap!=NULL) {                            // FrameMap is still allocated and not yet written to files
            if(n==0&&prevframe==0) {
                framemap[n]=0;                          // Special case frame 0
            } else {
                if (n != prevframe+1) {                 // Must be strictly linear
                    dprintf(" *** PrevFrame=%d n=%d *** n is not PrevFrame + 1",prevframe,n);
                    delete [] framemap;
                    framemap=NULL;
                    env->ThrowError("ExactDedup: Trying to compute frames out of order. Did you try to seek during processing?");
                }
                prevframe = n;

Debug output in blue

Post last few lines of debug info.

EDIT: And you are not trying to use any filters after ExactDeDup, pass==1, are you ?

[Danette]

While you were typing, I tried your first suggestions. I'll try your new download, but thought you'd be interested in the results from your earlier suggestion:

Originally Posted by StainlessS

How does it fair if you use eg VirtualDub ? (perhaps AvsPMod related).

Just tried pushing the script through Vdub. Vdub locked-up. So, I trimmed it to a 5-10 second segment to see if it would process. Then, after the 2 minutes, Vdub generated the error message:
“ffmpeg: decode video took longer than 2 minutes reading frame 0, aborted”

Script used was:
DGSource("C:\Users\Main\Desktop\Videos\MD.dgi")
Trim(4386,4695)
ExactDeDup()

Originally Posted by StainlessS

Are you using any kind of Multi-threaded version of Avisynth v2.6 ?

No, just the standard 2.6, build: Mar 31 2015

[StainlessS]

What colorspace is it ?, wondering why ffmpeg was mentioned (and what audio).

EDIT: Also, make sure that old plugin is gone from your plugs directory.

[Danette]

Will have to postpone this, but will work on the debug later and report. In the meantime here is info on colorspace and more:

[Danette]

Ahhhh ...you gave me a clue with:

Originally Posted by StainlessS

Also, make sure that old plugin is gone from your plugs directory.

I must not have copied it successfully. Deleted the old one from the plugins folder and copied the one from your link and now it seems to work fine (your original work is good). Txt files are generated and no error messages.

All I have to do now is to figure out how to work with it, which I'll do tomorrow.

Much thanks for staying with me through this and for working the new code.

[StainlessS]

Okay Dokey.

I added a whole load more debugging stuff now on SendSpace as Test_2, dont download.
I'll update on SendSpace in about an hour (bit busy).

[StainlessS]

OK, update zip on SendSpace, left all debugging in but disabled (uncomment Compiler.h #define BUG to switch back on).
Made a few small mods but not worth a version bump.

[Danette]

Well, now I’m into usage issues.

I have a dgi file with, clearly, many frames that are duplicates. I run ExactDeDup and the DupInfo.txt file is created. When I open that txt file, every single frame is listed (0-309). There are telecined frames in the file, but there are also duplicate frames. Not a single gap in the frame sequence exists so, I assume, that indicates that ExactDeDup found none of those duplicates.

I also created an AVI progessive file that has intentional duplicates in it (detelecined but not decimated), but get the same error message as reported, above, whenever AVI’s are sourced. So, it seems, that AvI’s cannot be used.

Also, when sourcing the dgi file, I loaded TFM and Tdecimate ahead of ExactDeDup, but that generates the error message as well. Is that a no-no?

[ndjamena]

??

If it's anything like DeDup metrics are recorded for every frame, it calculates which frames to discard later using the info from the txt file.

If you were talking about the time.txt file then that would be different...

[StainlessS]

ExactDedup calculates no metrics whatsoever. It decides based puely on exact frame contents, ie exits scan early if finds a single
pixel of any channel that is different where it is taken as not a dup, otherwise a dupe.

Can you post you problem avi, I'll have a look at it, but works fine for me on my test clip.
I got exact same results from original plug as modded one, but I had no problems getting original to work ok.
On my AVI test clip anime, it discarded about 1000, frames out of 4000.

It outputs a file of frames to keep, and also Matroska version 2 times for each of those frames.

First few entries for my test file, frames to keep.

0
6
7
8
9
11
12
13
14
16
17
18
19
21
22
23
24
26
27
28
29
31

And some times

# timecode format v2
0.000000000
200.200000000
233.566666667
266.933333333
300.300000000
367.033333333
400.400000000
433.766666667
467.133333333
533.866666667
567.233333333
600.600000000
633.966666667
700.700000000
734.066666667
767.433333333
800.800000000
867.533333333
900.900000000
934.266666667
967.633333333
1034.366666667
1067.733333333

Source file was 1920x1080 @ 29.97
EDIT: Ok my numbers were a bit off, test file had 4042 frames and kept 3371 of them.

[ndjamena]

I just tested it on an animated sequence sped up from 25fps to 50fps using changefps.

Although it successfully removed all the frames changefps added to the clip, I can't find a single instance of any of the original frames being removed from the clip, despite the fact that there are quite a few static scenes where each of the frames are almost visually indistinguishable from each other.

Given that's how it works, if he's really run a video through TFM it's unlikely ANY of the frames would be detected as duplicates.

I think he needs a different filter.

This was my attempt at re-writing DeDup:

http://www.mediafire.com/file/mfs7y8w7npv80mf/ReDup.zip

It's called ReDup, its purpose was to be run on animation to remove duplicates before processing, then add them back again once processing was complete. It was for when you're using something like Remation or another combination of filters that take forever to run.

Unfortunately my PTSD got in the way and I could never finished it... it was supposed to do much more than it does. I'm pissed off that I never managed to finish it.

It's based on the old DeDup code so it's got the same bugs, it contains the same functions as Dedup, but other than the ability to add the frames back in in the end it can also take an alternate clip for detection and even a second alternate clip to be compared to the first.

*sigh*

He probably just needs DeDup though, there's a version featuring fixed code floating on the forum somewhere.