Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
26th July 2018, 14:02 | #41 | Link |
Registered User
Join Date: Oct 2011
Posts: 275
|
The least i can do, if it can help a bit
I am now trying to have a feedback message from Staxrip program when it finished the scan of the video but no success for now, just so it does not look like freezing when processing the entire video to get duplicate frames data and get a small popup notification when it's done, previously i used a template dedicated for this associated with event commands : - When i needed this plugin i selected the dedicated template - The videos loaded in this template triggered an event command , launching avs2pipemod64 -benchmark and processing the analyze, and Staxrip window was showing the progress of the benchmark, - Once analyzed another event command was replacing the first pass filter by the second pass filter automatically Might post this method too if needed, at least you have a feedback on what Staxrip is doing and not like it is drawing and freezing, i don't mind myself but might be less scary BTW i did not had time to start benchmarking things yet plus it's way too hot those last days to run the PC in a room for a long time, room goes up to 30°c inside after barely one hour using the PC with light tasks... other rooms stays at 26-27 which is too high already, but will do soon tho no worries. |
26th July 2018, 15:42 | #42 | Link |
Registered User
Join Date: Oct 2011
Posts: 275
|
For this other method you don't need ForceProcess.dll plugin, only ExactDedup x64 dll is needed.
First let's create a template dedicated for this task : - Select your preferred encoding template ( or the one who starts wiht Staxrip usually ) - Go to Project > Save as Template > and give it a dedicated name like "Remove Duplicate Frames": On this template we will directly add the filter code, same as previously but lighter code, in the list of filters make a right click and then on the dropdown menu click on profiles, and copy paste this on the Misc Section : Code:
Remove Duplicate Frames = ExactDedup(firstpass=True, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \ maxdupcount=1000000, keeplastframe=false, show=false) Once the filter created we will add it to our project and to our new created template : Right click on the list of filters Add > Misc > Remove Duplicate Frames Then we save our template with this filter added: Project, Save as Template > we leave the name we put earlier "Remove Duplicate Frames" and just click OK Now we will create event commands triggers : go to tools > Advanced > Event Commands > click on "+ Add" On this first one we will set it like this : The command line is : Code:
"%app:avs2pipemod%" -benchmark "%script_file%" Once done we will again create another event command, same as before click on +Add, and this time we will set it like this : Unlike on this pic for the name above the code line it needs to be Remove Duplicate Frames and not "Remove Dup Frames", forgot to change it sorry Notice the little button on the orange arrow, that's where you click to bring the editor, easier than trying to type in the tiny field, in the editor you paste this : Code:
ExactDedup(firstpass=false, dupinfo="%working_dir%dupinfo.txt", times="%working_dir%times.txt", \ maxdupcount=1000000, keeplastframe=false, show=false) That's it, The next time you will need to use this filter go to templates, select the template named "Remove Duplicate Frames" , and then the rest it automated,when you will browse or drag and drop a video into staxrip the first pass will start and you will see something like this while the video is being analyzed for duplicate frames : Once finished the second event command we created will be triggered and automatically replace the first pass filter by the second pass one in the list of filters and video from there have all the duplicate frames removed, you still have like in the previous method to go to the MKV container options and select the times.txt file for the timestamps but that's all. ( It is possible using the template method to save the times.txt file to a specific location that will always be the same, not in the temp folder, and then save the template with the MKV setting for the timestamp path saved too, this will be a full auto process, if needed just ask, or in both the filters codes of this page set a static path for the times.txt file , and in MKV options set this same path too then save the template, again ask if needed or interested ) At least here you see what Staxrip is doing, like i said i don't mind myself but maybe it's better and less scary idk Last edited by Yanak; 26th July 2018 at 15:51. |
26th July 2018, 22:30 | #43 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,695
|
I probably should have posted this back when this thread first started.
The following is the simple script I use to detect duplicates, and near-duplicates. I haven't used Dedup, ExactDeDup, or StanlessS' mods, but this certainly can and does detect exact duplicates perfectly, and does a very nice job with near-duplicates. I usually use this code in conjunction with some sort of replacement logic, either by taking the frame number list and operating on that in a second pass, or by simply adding some conditional logic to the function. I've posted these frame replacement scripts in other doom9 threads. The script is based entirely on YDifference, a built-in AVISynth function that compares the luma of every pixel in the current frame with every pixel in the previous frame and then does a sum of absolute differences to obtain a metric. This is a very useful function for scene change detection, motion detection, and any other situation where you want to get a metric that relates to the amount of visual change between frames. YDifference sure ain't perfect, but it is amazing how often it gets the job done. Here's my very simple-minded approach to finding duplicates. It is obviously hard-wired and not written as a proper function, so you have to go through and change things as needed (e.g., assumeBFF). I never could be bothered to clean it up. Code:
#This script finds either exact duplicate frames (blankthreshold=0) or near duplicates #and then outputs the frame numbers of the dups. global blankthreshold=3 filename = "e:\output_duplicate_frames.txt" AVISource("e:\fs.avi").killaudio() i=AssumeBFF.ConvertToYV12 #This line below will output EVERY frame that is below threshold, which results in LOTS of frames #Normally you don't do this, but it is included for those who want this capability. #WriteFileIf(last, filename, "(YDifferenceFromPrevious(i)<=blankthreshold)", "current_frame+1", append = false) #The line below writes the FIRST frame that falls below the threshold WriteFileIf(last, filename, "(YDifferenceFromPrevious(i)>blankthreshold)&&YDifferenceToNext(i)<=blankthreshold", "current_frame", append = false) #Use this instead of WriteFile in order to determine blankthreshold #ScriptClip("Subtitle(String(YDifferenceFromPrevious(i)))") |
28th July 2018, 08:11 | #44 | Link |
Registered User
Join Date: Oct 2011
Posts: 275
|
Glad to see you get it working,
About the error reported , what version of ExactDeDup are your using please ? The "show" parameter was added in version 0.04 apparently : Code:
v0.4 Mod by StainlessS, 16 Feb 2017. Avisynth v2.6 plugin. Fixed a number of bugs. Added Support for v2.6 Planar (also added U & V chroma scan for YV12, only scanned luma). Added Show Arg. Write times and dupinfo files at end of scan,so you dont have to edit script in text editor before second pass. No runtime dependency. http://www.mediafire.com/file/8184da6dp4hsdcq/ And for : %working_dir%\times.txt it should be %working_dir%times.txt without the \ in the middle, not sure if it's just a typo you made while posting on the forum or if you entered this in the code, i checked again and in my code samples posted here it seems to be correct without the /, unless i am not seeing it. You can quickly test this : Just open a video in Staxrip no matter the video, template or filters, just open one video so it creates temp files, Then go to the filters editor and just for the test add a line : %working_dir% then go to the preview tab : does it show the correct path there ? i mean is the path indicated the location where Staxrip just created some .AVS and a .LOG file for the video you opened in it ? ( think about removing this %working_dir% line from the profiles editor created just for the test after it's done ^^ ) I'm using Staxrip v 1.7.0.6, the latest one released by Stax76 and it works there, not sure what version of the program you are using. You can check it in Help > Info As for the duplicates frames removed it depends of the sources and the quality, this plugin removes exact duplicates, sometimes if your source material is already an altered source, already encoded i mean, there can be frame that were originally the same but during the encoding some little artifacts were added so it ends up in 2 different frames for the plugin, but for things like gaming capture unaltered/raw footage there is often a lot of duplicate frames. @johnmeyer, i'll give it a try one of those days, might serve in some occasions , thank you Last edited by Yanak; 28th July 2018 at 08:15. |
9th August 2018, 12:12 | #46 | Link |
Registered User
Join Date: Oct 2011
Posts: 275
|
? I have been out for a bit and see this now, cannot remember exactly the last posts but i don't recall any piracy discussion, i see that my last post was removed in the process too but i'm sure at 100% that i never spoke of piracy related things...
Anyways... Came to continue the conversation started here, makes more sense to post it here on ExactDedup dedicated thread. Took me way longer to experiment things than expected as being doing some other stuff irl, plus really hot weather all those days did not push me to make the PC run a lot when i had some free time to burn, but as promised i recompiled it, on some files it gains nicely during the 1st pass, the 2nd pass is always more or less the same, on some others video files there is close to no gain, probably because i reach the hardware limit before the software part. A part of the tests results : Code:
*********************************** #### Script ### ColorBars(width = 1920, height = 1080, pixel_type = "YV12", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999) *********************************** ########### 1st pass Original dll ########## Frames processed: 10000 (0 - 9999) FPS 263.7 Time (elapsed): 00:00:37.920 ########### 1st pass Compiled dll ########## Frames processed: 10000 (0 - 9999) FPS 745.6 Time (elapsed): 00:00:13.411 ########### 2nd pass Original dll ########## Too fast for metrics ########### 2nd pass Compiled dll ########## Too fast for metrics *********************************** #### Script ### ColorBars(width = 1920, height = 1080, pixel_type = "RGB32", staticframes = false).killaudio().assumefps(60, 1).trim(0,9999) *********************************** ########### 1st pass Original dll ########## Frames processed: 10000 (0 - 9999) FPS 66.34 Time (elapsed): 00:02:30.736 ########### 1st pass Compiled dll ########## Frames processed: 10000 (0 - 9999) FPS 187.2 Time (elapsed): 00:00:53.432 ########### 2nd pass Original dll ########## Too fast for metrics ########### 2nd pass Compiled dll ########## Too fast for metrics *********************************** Format : AVI Format/Info : Audio Video Interleave Format profile : OpenDML File size : 6.10 GiB Duration : 2 min 36 s Overall bit rate : 336 Mb/s Format : Fraps Codec ID : FPS1 Duration : 2 min 36 s Bit rate : 334 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate : 60.000 FPS Color space : YUV Bit depth : 8 bits Bits/(Pixel*Frame) : 2.685 Stream size : 6.07 GiB (100%) *********************************** #### Script ### FFVideoSource("video.avi", colorspace = "YUV420P8", \ cachefile = "video.ffindex") *********************************** ########### 1st pass Original dll ########## Frames processed: 9366 (0 - 9365) FPS 373.1 Time (elapsed): 00:00:25.102 ########### 1st pass Compiled dll ########## Frames processed: 9366 (0 - 9365) FPS 426.9 Time (elapsed): 00:00:21.939 ########### 2nd pass Original dll ########## Frames processed: 5647 (0 - 5646) FPS 271.0 Time (elapsed): 00:00:20.836 ########### 2nd pass Compiled dll ########## Frames processed: 5647 (0 - 5646) FPS 272.1 Time (elapsed): 00:00:20.753 *********************************** #### Script ### DSS2("video.avi") *********************************** ########### 1st pass Original dll ########## Frames processed: 9365 (0 - 9364) FPS 92.93 Time (elapsed): 00:01:40.779 ########### 1st pass Compiled dll ########## Frames processed: 9365 (0 - 9364) FPS 116.5 Time (elapsed): 00:01:20.355 ########### 2nd pass Original dll ########## Frames processed: 5647 (0 - 5646) FPS 73.85 Time (elapsed): 00:01:16.464 ########### 2nd pass Compiled dll ########## Frames processed: 5647 (0 - 5646) FPS 73.65 Time (elapsed): 00:01:16.672 *********************************** #### Script ### AviSource("video.avi", Audio = False) *********************************** ########### 1st pass Original dll ########## Frames processed: 9366 (0 - 9365) FPS 73.12 Time (elapsed): 00:02:08.086 ########### 1st pass Compiled dll ########## Frames processed: 9366 (0 - 9365) FPS 101.8 Time (elapsed): 00:01:31.967 ########### 2nd pass Original dll ########## Frames processed: 5647 (0 - 5646) FPS 83.01 Time (elapsed): 00:01:08.028 ########### 2nd pass Compiled dll ########## Frames processed: 5647 (0 - 5646) FPS 83.06 Time (elapsed): 00:01:07.991 *********************************** Format : AVI Format/Info : Audio Video Interleave Format profile : OpenDML File size : 3.88 GiB Duration : 55 s 933 ms Overall bit rate : 596 Mb/s Format : Fraps Codec ID : FPS1 Duration : 55 s 933 ms Bit rate : 594 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate : 60.000 FPS Color space : YUV Bit depth : 8 bits Bits/(Pixel*Frame) : 4.778 Stream size : 3.87 GiB (100%) *********************************** #### Script ### FFVideoSource("video.avi", colorspace = "YUV420P8", \ cachefile = "video.ffindex") *********************************** ########### 1st pass Original dll ########## Frames processed: 3356 (0 - 3355) FPS 357.9 Time (elapsed): 00:00:09.378 ########### 1st pass Compiled dll ########## Frames processed: 3356 (0 - 3355) FPS 387.7 Time (elapsed): 00:00:08.657 ########### 2nd pass Original dll ########## Frames processed: 2455 (0 - 2454) FPS 298.9 Time (elapsed): 00:00:08.214 ########### 2nd pass Compiled dll ########## Frames processed: 2455 (0 - 2454) FPS 298.8 Time (elapsed): 00:00:08.217 *********************************** #### Script ### DSS2("video.avi") *********************************** ########### 1st pass Original dll ########## Frames processed: 3355 (0 - 3354) FPS 78.28 Time (elapsed): 00:00:42.858 ########### 1st pass Compiled dll ########## Frames processed: 3355 (0 - 3354) FPS 84.04 Time (elapsed): 00:00:39.923 ########### 2nd pass Original dll ########## Frames processed: 2454 (0 - 2453) FPS 62.36 Time (elapsed): 00:00:39.352 ########### 2nd pass Compiled dll ########## Frames processed: 2454 (0 - 2453) FPS 62.50 Time (elapsed): 00:00:39.264 *********************************** #### Script ### AviSource("video.avi", Audio = False) *********************************** ########### 1st pass Original dll ########## Frames processed: 3356 (0 - 3355) FPS 64.34 Time (elapsed): 00:00:52.159 ########### 1st pass Compiled dll ########## Frames processed: 3356 (0 - 3355) FPS 77.35 Time (elapsed): 00:00:43.387 ########### 2nd pass Original dll ########## Frames processed: 2455 (0 - 2454) FPS 65.33 Time (elapsed): 00:00:37.579 ########### 2nd pass Compiled dll ########## Frames processed: 2455 (0 - 2454) FPS 65.33 Time (elapsed): 00:00:37.576 *********************************** Format : AVI Format/Info : Audio Video Interleave File size : 839 MiB Duration : 50 s 0 ms Overall bit rate : 141 Mb/s Format : Fraps Codec ID : FPS1 Duration : 50 s 0 ms Bit rate : 139 Mb/s Width : 1 280 pixels Height : 1 024 pixels Display aspect ratio : 5:4 Frame rate : 30.000 FPS Color space : YUV Bit depth : 8 bits Bits/(Pixel*Frame) : 3.545 Stream size : 831 MiB (99%) *********************************** #### Script ### FFVideoSource("video.avi", colorspace = "YUV420P8", \ cachefile = "video.ffindex") *********************************** ########### 1st pass Original dll ########## Frames processed: 1500 (0 - 1499) FPS 474.5 Time (elapsed): 00:00:03.161 ########### 1st pass Compiled dll ########## Frames processed: 1500 (0 - 1499) FPS 492.5 Time (elapsed): 00:00:03.046 ########### 2nd pass Original dll ########## Frames processed: 1500 (0 - 1499) FPS 492.3 Time (elapsed): 00:00:03.047 ########### 2nd pass Compiled dll ########## Frames processed: 1500 (0 - 1499) FPS (min | max | average): 54.79 | 6789 | 501.7 Time (elapsed): 00:00:02.990 *********************************** #### Script ### DSS2("video.avi") *********************************** ########### 1st pass Original dll ########## Frames processed: 1500 (0 - 1499) FPS 161.7 Time (elapsed): 00:00:09.274 ########### 1st pass Compiled dll ########## Frames processed: 1500 (0 - 1499) FPS 162.0 Time (elapsed): 00:00:09.259 ########### 2nd pass Original dll ########## Frames processed: 1500 (0 - 1499) FPS 161.5 Time (elapsed): 00:00:09.290 ########### 2nd pass Compiled dll ########## Frames processed: 1500 (0 - 1499) FPS 161.6 Time (elapsed): 00:00:09.284 *********************************** #### Script ### AviSource("video.avi", Audio = False) *********************************** ########### 1st pass Original dll ########## Frames processed: 1500 (0 - 1499) FPS 139.5 Time (elapsed): 00:00:10.749 ########### 1st pass Compiled dll ########## Frames processed: 1500 (0 - 1499) FPS 154.1 Time (elapsed): 00:00:09.734 ########### 2nd pass Original dll ########## Frames processed: 1500 (0 - 1499) FPS 161.5 Time (elapsed): 00:00:09.288 ########### 2nd pass Compiled dll ########## Frames processed: 1500 (0 - 1499) FPS 161.5 Time (elapsed): 00:00:09.286 *********************************** Format : Matroska Format version : Version 4 / Version 2 File size : 492 MiB Duration : 2 min 28 s Overall bit rate : 27.8 Mb/s Format : HEVC Format/Info : High Efficiency Video Coding Format profile : Main@L8.5@Main Codec ID : V_MPEGH/ISO/HEVC Duration : 2 min 28 s Width : 960 pixels Height : 540 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 50.000 FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits *********************************** #### Script ### DSS2("video.mkv") *********************************** ########### 1st pass Original dll ########## Frames processed: 7415 (0 - 7414) FPS 348.4 Time (elapsed): 00:00:21.286 ########### 1st pass Compiled dll ########## Frames processed: 7415 (0 - 7414) FPS 354.5 Time (elapsed): 00:00:20.915 ########### 2nd pass Original dll ########## Frames processed: 6962 (0 - 6961) FPS 334.5 Time (elapsed): 00:00:20.815 ########### 2nd pass Compiled dll ########## Frames processed: 6962 (0 - 6961) FPS 334.8 Time (elapsed): 00:00:20.793 *********************************** |
9th August 2018, 12:28 | #47 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
I too was puzzled by post #46 at the time it was posted.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
|
|