Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage
Register FAQ Calendar Today's Posts Search

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th July 2018, 14:02   #41  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 275
The least i can do, if it can help a bit

I am now trying to have a feedback message from Staxrip program when it finished the scan of the video but no success for now, just so it does not look like freezing when processing the entire video to get duplicate frames data and get a small popup notification when it's done, previously i used a template dedicated for this associated with event commands :
- When i needed this plugin i selected the dedicated template
- The videos loaded in this template triggered an event command , launching avs2pipemod64 -benchmark and processing the analyze, and Staxrip window was showing the progress of the benchmark,
- Once analyzed another event command was replacing the first pass filter by the second pass filter automatically

Might post this method too if needed, at least you have a feedback on what Staxrip is doing and not like it is drawing and freezing, i don't mind myself but might be less scary


BTW i did not had time to start benchmarking things yet plus it's way too hot those last days to run the PC in a room for a long time, room goes up to 30°c inside after barely one hour using the PC with light tasks... other rooms stays at 26-27 which is too high already, but will do soon tho no worries.
Yanak is offline   Reply With Quote
Old 26th July 2018, 15:42   #42  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 275
For this other method you don't need ForceProcess.dll plugin, only ExactDedup x64 dll is needed.

First let's create a template dedicated for this task :
- Select your preferred encoding template ( or the one who starts wiht Staxrip usually )
- Go to Project > Save as Template > and give it a dedicated name like "Remove Duplicate Frames":


On this template we will directly add the filter code, same as previously but lighter code, in the list of filters make a right click and then on the dropdown menu click on profiles, and copy paste this on the Misc Section :

Code:
Remove Duplicate Frames =
    ExactDedup(firstpass=True, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
    maxdupcount=1000000, keeplastframe=false, show=false)
Note that you can't have both this one and previous one at the same time, well it is possible but need to change the filter name... Best is to stay with one or the other method


Once the filter created we will add it to our project and to our new created template :
Right click on the list of filters Add > Misc > Remove Duplicate Frames

Then we save our template with this filter added:
Project, Save as Template > we leave the name we put earlier "Remove Duplicate Frames" and just click OK


Now we will create event commands triggers :
go to tools > Advanced > Event Commands > click on "+ Add"
On this first one we will set it like this :


The command line is :
Code:
"%app:avs2pipemod%" -benchmark "%script_file%"
The rest is easy to copy from the picture

Once done we will again create another event command, same as before click on +Add, and this time we will set it like this :

Unlike on this pic for the name above the code line it needs to be Remove Duplicate Frames and not "Remove Dup Frames", forgot to change it sorry

Notice the little button on the orange arrow, that's where you click to bring the editor, easier than trying to type in the tiny field, in the editor you paste this :

Code:
ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \
maxdupcount=1000000, keeplastframe=false, show=false)
Lastly check the 2 little boxes to make them active :



That's it, The next time you will need to use this filter go to templates, select the template named "Remove Duplicate Frames" , and then the rest it automated,when you will browse or drag and drop a video into staxrip the first pass will start and you will see something like this while the video is being analyzed for duplicate frames :


Once finished the second event command we created will be triggered and automatically replace the first pass filter by the second pass one in the list of filters and video from there have all the duplicate frames removed, you still have like in the previous method to go to the MKV container options and select the times.txt file for the timestamps but that's all.

( It is possible using the template method to save the times.txt file to a specific location that will always be the same, not in the temp folder, and then save the template with the MKV setting for the timestamp path saved too, this will be a full auto process, if needed just ask, or in both the filters codes of this page set a static path for the times.txt file , and in MKV options set this same path too then save the template, again ask if needed or interested )

At least here you see what Staxrip is doing, like i said i don't mind myself but maybe it's better and less scary idk


Last edited by Yanak; 26th July 2018 at 15:51.
Yanak is offline   Reply With Quote
Old 26th July 2018, 22:30   #43  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,695
I probably should have posted this back when this thread first started.

The following is the simple script I use to detect duplicates, and near-duplicates. I haven't used Dedup, ExactDeDup, or StanlessS' mods, but this certainly can and does detect exact duplicates perfectly, and does a very nice job with near-duplicates. I usually use this code in conjunction with some sort of replacement logic, either by taking the frame number list and operating on that in a second pass, or by simply adding some conditional logic to the function. I've posted these frame replacement scripts in other doom9 threads.

The script is based entirely on YDifference, a built-in AVISynth function that compares the luma of every pixel in the current frame with every pixel in the previous frame and then does a sum of absolute differences to obtain a metric. This is a very useful function for scene change detection, motion detection, and any other situation where you want to get a metric that relates to the amount of visual change between frames.

YDifference sure ain't perfect, but it is amazing how often it gets the job done.

Here's my very simple-minded approach to finding duplicates. It is obviously hard-wired and not written as a proper function, so you have to go through and change things as needed (e.g., assumeBFF). I never could be bothered to clean it up.
Code:
#This script finds either exact duplicate frames (blankthreshold=0) or near duplicates 
#and then outputs the frame numbers of the dups.

global blankthreshold=3

filename = "e:\output_duplicate_frames.txt"
AVISource("e:\fs.avi").killaudio()
i=AssumeBFF.ConvertToYV12

#This line below will output EVERY frame that is below threshold, which results in LOTS of frames
#Normally you don't do this, but it is included for those who want this capability.
#WriteFileIf(last, filename,  "(YDifferenceFromPrevious(i)<=blankthreshold)", "current_frame+1", append = false)

#The line below writes the FIRST frame that falls below the threshold
WriteFileIf(last, filename,  "(YDifferenceFromPrevious(i)>blankthreshold)&&YDifferenceToNext(i)<=blankthreshold", "current_frame", append = false)

#Use this instead of WriteFile in order to determine blankthreshold
#ScriptClip("Subtitle(String(YDifferenceFromPrevious(i)))")
johnmeyer is offline   Reply With Quote
Old 28th July 2018, 08:11   #44  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 275
Glad to see you get it working,

About the error reported , what version of ExactDeDup are your using please ? The "show" parameter was added in version 0.04 apparently :

Code:
v0.4 Mod by StainlessS, 16 Feb 2017.
Avisynth v2.6 plugin.
Fixed a number of bugs.
Added Support for v2.6 Planar (also added U & V chroma scan for YV12, only scanned luma).
Added Show Arg.
Write times and dupinfo files at end of scan,so you dont have to edit script in text editor before second pass.
No runtime dependency.
Latest version of this plugin is v0.05, direct link to StainlessS last version :
http://www.mediafire.com/file/8184da6dp4hsdcq/

And for :
%working_dir%\times.txt it should be %working_dir%times.txt without the \ in the middle, not sure if it's just a typo you made while posting on the forum or if you entered this in the code, i checked again and in my code samples posted here it seems to be correct without the /, unless i am not seeing it.

You can quickly test this :
Just open a video in Staxrip no matter the video, template or filters, just open one video so it creates temp files,
Then go to the filters editor and just for the test add a line :
%working_dir%
then go to the preview tab :

does it show the correct path there ? i mean is the path indicated the location where Staxrip just created some .AVS and a .LOG file for the video you opened in it ?
( think about removing this %working_dir% line from the profiles editor created just for the test after it's done ^^ )

I'm using Staxrip v 1.7.0.6, the latest one released by Stax76 and it works there, not sure what version of the program you are using.
You can check it in Help > Info

As for the duplicates frames removed it depends of the sources and the quality, this plugin removes exact duplicates, sometimes if your source material is already an altered source, already encoded i mean, there can be frame that were originally the same but during the encoding some little artifacts were added so it ends up in 2 different frames for the plugin, but for things like gaming capture unaltered/raw footage there is often a lot of duplicate frames.


@johnmeyer, i'll give it a try one of those days, might serve in some occasions , thank you

Last edited by Yanak; 28th July 2018 at 08:15.
Yanak is offline   Reply With Quote
Old 29th July 2018, 14:26   #45  |  Link
Wilbert
Moderator
 
Join Date: Nov 2001
Location: Netherlands
Posts: 6,364
@JoyBell, Please respect the forum rules. No talk about illegal downloaded stuff.

Last edited by Wilbert; 29th July 2018 at 19:58.
Wilbert is offline   Reply With Quote
Old 9th August 2018, 12:12   #46  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 275
? I have been out for a bit and see this now, cannot remember exactly the last posts but i don't recall any piracy discussion, i see that my last post was removed in the process too but i'm sure at 100% that i never spoke of piracy related things...

Anyways...

Came to continue the conversation started here, makes more sense to post it here on ExactDedup dedicated thread.

Took me way longer to experiment things than expected as being doing some other stuff irl, plus really hot weather all those days did not push me to make the PC run a lot when i had some free time to burn, but as promised i recompiled it, on some files it gains nicely during the 1st pass, the 2nd pass is always more or less the same, on some others video files there is close to no gain, probably because i reach the hardware limit before the software part.

A part of the tests results :
Code:
***********************************
#### Script ###
ColorBars(width = 1920, height = 1080, pixel_type = "YV12", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999)
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               10000 (0 - 9999)
FPS                             263.7
Time (elapsed):                 00:00:37.920
###########   1st pass Compiled dll   ##########
Frames processed:               10000 (0 - 9999)
FPS                             745.6
Time (elapsed):                 00:00:13.411

###########   2nd pass Original dll   ########## 
Too fast for metrics
###########   2nd pass Compiled dll   ##########
Too fast for metrics

***********************************
#### Script ###
ColorBars(width = 1920, height = 1080, pixel_type = "RGB32", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999)
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               10000 (0 - 9999)
FPS                             66.34
Time (elapsed):                 00:02:30.736
###########   1st pass Compiled dll   ##########
Frames processed:               10000 (0 - 9999)
FPS                             187.2
Time (elapsed):                 00:00:53.432

###########   2nd pass Original dll   ########## 
Too fast for metrics
###########   2nd pass Compiled dll   ##########
Too fast for metrics

***********************************
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Format profile                           : OpenDML
File size                                : 6.10 GiB
Duration                                 : 2 min 36 s
Overall bit rate                         : 336 Mb/s
Format                                   : Fraps
Codec ID                                 : FPS1
Duration                                 : 2 min 36 s
Bit rate                                 : 334 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 60.000 FPS
Color space                              : YUV
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 2.685
Stream size                              : 6.07 GiB (100%)
***********************************
#### Script ###
FFVideoSource("video.avi", colorspace = "YUV420P8", \
              cachefile = "video.ffindex")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               9366 (0 - 9365)
FPS                             373.1
Time (elapsed):                 00:00:25.102
###########   1st pass Compiled dll   ##########
Frames processed:               9366 (0 - 9365)
FPS                             426.9
Time (elapsed):                 00:00:21.939

###########   2nd pass Original dll   ########## 
Frames processed:               5647 (0 - 5646)
FPS                             271.0
Time (elapsed):                 00:00:20.836
###########   2nd pass Compiled dll   ##########
Frames processed:               5647 (0 - 5646)
FPS                             272.1
Time (elapsed):                 00:00:20.753

***********************************
#### Script ###
DSS2("video.avi")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               9365 (0 - 9364)
FPS                             92.93
Time (elapsed):                 00:01:40.779
###########   1st pass Compiled dll   ##########
Frames processed:               9365 (0 - 9364)
FPS                             116.5
Time (elapsed):                 00:01:20.355

###########   2nd pass Original dll   ########## 
Frames processed:               5647 (0 - 5646)
FPS                             73.85
Time (elapsed):                 00:01:16.464
###########   2nd pass Compiled dll   ##########
Frames processed:               5647 (0 - 5646)
FPS                             73.65
Time (elapsed):                 00:01:16.672

***********************************
#### Script ###
AviSource("video.avi", Audio = False)
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               9366 (0 - 9365)
FPS                             73.12
Time (elapsed):                 00:02:08.086
###########   1st pass Compiled dll   ##########
Frames processed:               9366 (0 - 9365)
FPS                             101.8
Time (elapsed):                 00:01:31.967

###########   2nd pass Original dll   ########## 
Frames processed:               5647 (0 - 5646)
FPS                             83.01
Time (elapsed):                 00:01:08.028
###########   2nd pass Compiled dll   ##########
Frames processed:               5647 (0 - 5646)
FPS                             83.06
Time (elapsed):                 00:01:07.991

***********************************
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Format profile                           : OpenDML
File size                                : 3.88 GiB
Duration                                 : 55 s 933 ms
Overall bit rate                         : 596 Mb/s
Format                                   : Fraps
Codec ID                                 : FPS1
Duration                                 : 55 s 933 ms
Bit rate                                 : 594 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 60.000 FPS
Color space                              : YUV
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 4.778
Stream size                              : 3.87 GiB (100%)
***********************************
#### Script ###
FFVideoSource("video.avi", colorspace = "YUV420P8", \
              cachefile = "video.ffindex")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               3356 (0 - 3355)
FPS                             357.9
Time (elapsed):                 00:00:09.378
###########   1st pass Compiled dll   ##########
Frames processed:               3356 (0 - 3355)
FPS                             387.7
Time (elapsed):                 00:00:08.657

###########   2nd pass Original dll   ########## 
Frames processed:               2455 (0 - 2454)
FPS                             298.9
Time (elapsed):                 00:00:08.214
###########   2nd pass Compiled dll   ##########
Frames processed:               2455 (0 - 2454)
FPS                             298.8
Time (elapsed):                 00:00:08.217

***********************************
#### Script ###
DSS2("video.avi")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               3355 (0 - 3354)
FPS                             78.28
Time (elapsed):                 00:00:42.858
###########   1st pass Compiled dll   ##########
Frames processed:               3355 (0 - 3354)
FPS                             84.04
Time (elapsed):                 00:00:39.923

###########   2nd pass Original dll   ########## 
Frames processed:               2454 (0 - 2453)
FPS                             62.36
Time (elapsed):                 00:00:39.352
###########   2nd pass Compiled dll   ##########
Frames processed:               2454 (0 - 2453)
FPS                             62.50
Time (elapsed):                 00:00:39.264

***********************************
#### Script ###
AviSource("video.avi", Audio = False)
***********************************
###########   1st pass Original dll   ##########
Frames processed:               3356 (0 - 3355)
FPS                             64.34
Time (elapsed):                 00:00:52.159
###########   1st pass Compiled dll   ##########
Frames processed:               3356 (0 - 3355)
FPS                             77.35
Time (elapsed):                 00:00:43.387

###########   2nd pass Original dll   ##########
Frames processed:               2455 (0 - 2454)
FPS                             65.33
Time (elapsed):                 00:00:37.579
###########   2nd pass Compiled dll   ##########
Frames processed:               2455 (0 - 2454)
FPS                             65.33
Time (elapsed):                 00:00:37.576

***********************************
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 839 MiB
Duration                                 : 50 s 0 ms
Overall bit rate                         : 141 Mb/s
Format                                   : Fraps
Codec ID                                 : FPS1
Duration                                 : 50 s 0 ms
Bit rate                                 : 139 Mb/s
Width                                    : 1 280 pixels
Height                                   : 1 024 pixels
Display aspect ratio                     : 5:4
Frame rate                               : 30.000 FPS
Color space                              : YUV
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 3.545
Stream size                              : 831 MiB (99%)
***********************************
#### Script ###
FFVideoSource("video.avi", colorspace = "YUV420P8", \
              cachefile = "video.ffindex")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               1500 (0 - 1499)
FPS                             474.5
Time (elapsed):                 00:00:03.161
###########   1st pass Compiled dll   ##########
Frames processed:               1500 (0 - 1499)
FPS                             492.5
Time (elapsed):                 00:00:03.046

###########   2nd pass Original dll   ########## 
Frames processed:               1500 (0 - 1499)
FPS                             492.3
Time (elapsed):                 00:00:03.047
###########   2nd pass Compiled dll   ##########
Frames processed:               1500 (0 - 1499)
FPS (min | max | average):      54.79 | 6789 | 501.7
Time (elapsed):                 00:00:02.990

***********************************
#### Script ###
DSS2("video.avi")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               1500 (0 - 1499)
FPS                             161.7
Time (elapsed):                 00:00:09.274
###########   1st pass Compiled dll   ##########
Frames processed:               1500 (0 - 1499)
FPS                             162.0
Time (elapsed):                 00:00:09.259

###########   2nd pass Original dll   ########## 
Frames processed:               1500 (0 - 1499)
FPS                             161.5
Time (elapsed):                 00:00:09.290
###########   2nd pass Compiled dll   ##########
Frames processed:               1500 (0 - 1499)
FPS                             161.6
Time (elapsed):                 00:00:09.284

***********************************
#### Script ###
AviSource("video.avi", Audio = False)
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               1500 (0 - 1499)
FPS                             139.5
Time (elapsed):                 00:00:10.749
###########   1st pass Compiled dll   ##########
Frames processed:               1500 (0 - 1499)
FPS                             154.1
Time (elapsed):                 00:00:09.734

###########   2nd pass Original dll   ########## 
Frames processed:               1500 (0 - 1499)
FPS                             161.5
Time (elapsed):                 00:00:09.288
###########   2nd pass Compiled dll   ##########
Frames processed:               1500 (0 - 1499)
FPS                             161.5
Time (elapsed):                 00:00:09.286

***********************************
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 492 MiB
Duration                                 : 2 min 28 s
Overall bit rate                         : 27.8 Mb/s
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main@L8.5@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 2 min 28 s
Width                                    : 960 pixels
Height                                   : 540 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
***********************************
#### Script ###
DSS2("video.mkv")
***********************************
###########   1st pass Original dll   ########## 
Frames processed:               7415 (0 - 7414)
FPS                             348.4
Time (elapsed):                 00:00:21.286
###########   1st pass Compiled dll   ##########
Frames processed:               7415 (0 - 7414)
FPS                             354.5
Time (elapsed):                 00:00:20.915

###########   2nd pass Original dll   ########## 
Frames processed:               6962 (0 - 6961)
FPS                             334.5
Time (elapsed):                 00:00:20.815
###########   2nd pass Compiled dll   ##########
Frames processed:               6962 (0 - 6961)
FPS                             334.8
Time (elapsed):                 00:00:20.793

***********************************
Hosting it here for the moment until attached x64 dll is approved : https://www84.zippyshare.com/v/CdNTNTbC/file.html
Attached Files
File Type: zip ExactDeDup_x64.zip (90.0 KB, 52 views)
Yanak is offline   Reply With Quote
Old 9th August 2018, 12:28   #47  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
Originally Posted by Yanak View Post
? I have been out for a bit and see this now, cannot remember exactly the last posts but i don't recall any piracy discussion
I too was puzzled by post #46 at the time it was posted.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:26.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.