RemoveDirt - pfmod

pinterf · 14th March 2019, 19:42

RemoveDirt v1.0 (20250108)
https://github.com/pinterf/RemoveDirt/releases/tag/v1.0
Milestone: Vapoursynth (API v4) support

RemoveDirt v0.9.2 (20190324)
https://github.com/pinterf/RemoveDir...ses/tag/v0.9.2

Link to the previous topic:
https://forum.doom9.org/showthread.p...threadid=70856

Code:

# RemoveDirt - pfmod
A plugin for removing dirt from film clips.

Port of classic RemoveDirt 0.9 to Avisynth v2.6 interface (x86/x64), adding new color spaces, bugfixes.

Functions in plugin: RestoreMotionBlocks and SCSelect.
This mod does not support other filters appearing in 1.0beta.

Note: Previous v0.9 DLL versions named differently (RemoveDirtT.DLL, RemoveDirtSSE2.DLL) should be deleted from your plugin folder.

- (20190324 v0.9.2)
  - RestoreMotionBlocks: 10-16 bit support. Relevant threshold and noise parameters are bit depth independent.
  - minor speedup
  - SCSelect: add support for 10-16 bits and 32 bit float clips
  - SCSelect: add support for planar RGB
  - FIX: SCSelect: make it work properly for large frames (>8MPixel)
  - FIX: SCSelect: Makes use the whole frame: now counts the rightmost non-mod32 pixels as well.
  - Clang 7.0.1 support (LLVM) with Visual Studio 2017 (15.9.9) (LLVM 8.0.0: not tried yet)
    - Install LLVM 7.0.1 (http://releases.llvm.org/download.html, Windows pre-built libraries)
    - Install Clang Power Tools & LLVM Compiler Toolchain
      - https://marketplace.visualstudio.com/items?itemName=caphyon.ClangPowerTools
      - https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.llvm-toolchain
  - update html docs
  - add clang-built DLLs to the released version

- (20190314 v0.9.1)
  - project moved to github: https://github.com/pinterf/RemoveDirt
  - built using Visual Studio 2017
  - x64 build for Avisynth+
  - Added version resource to DLL
  - Changed to AVS 2.6 plugin interface
  - Fix: RestoreMotionBlocks: grey=false: it was copying 8x4 chroma pixels for YV12 and 8x8 for YUY2 instead of 4x4 and 8x4 blocks
  - Fix: SCSelect: Old v0.9 SSE2 code omitted difference checking for every second 8 columns
  - Removed MMX code, now requires SSE2
  - Reports MT Modes for Avisynth+: MT_SERIALIZED for SCSelect
  - Reports MT Modes for Avisynth+: MT_MULTI_INSTANCE for RestoreMotionBlocks (may not be any faster)
  - Added Y, YV16 and YV24 support besides existing YV12 and planar-hacked-YUY2

- (20190312)
  Initial source v0.9 (2005/05/07) moved to VS2017 project 

Links
=====
Project: https://github.com/pinterf/RemoveDirt
Forum: https://forum.doom9.org/showthread.php?t=176199
Additional info: http://avisynth.nl/index.php/RemoveDirt

pinterf · 14th March 2019, 19:51

Finally I used the original Avisynth source, the VapourSynth port had bugs, I'll point to those places on my fork later next week.

poisondeathray · 14th March 2019, 20:02

Thanks

SaurusX · 14th March 2019, 20:18

This is much appreciated.

filler56789 · 14th March 2019, 20:49

Thanks!

Reel.Deel · 14th March 2019, 21:05

Thank you pinterf, I will update the wiki in the next couple of days.

ChaosKing · 14th March 2019, 22:38

I think the VS port wasn't finished yet. The one dll I found crashes often. Thx for the release.

I always wondered how hard it is to make "hybrid plugins" now which supports vs and avs+, since avs+ now supports 16bit and the same color spaces? Some plugins like this exists: knlmeans, f3kdb, ffms2
Or is it just too cumbersome because of the different threading models?

p.s. You can also install RemoveDirt via avsrepo now

real.finder · 15th March 2019, 04:19

been waiting for this, thanks

Myrsloik · 15th March 2019, 14:21

Quote:

Originally Posted by ChaosKing

I think the VS port wasn't finished yet. The one dll I found crashes often. Thx for the release.

I always wondered how hard it is to make "hybrid plugins" now which supports vs and avs+, since avs+ now supports 16bit and the same color spaces? Some plugins like this exists: knlmeans, f3kdb, ffms2
Or is it just too cumbersome because of the different threading models?

p.s. You can also install RemoveDirt via avsrepo now

It's generally very simple. The threading models are extremely similar.

pinterf · 24th March 2019, 15:20

New build.
Download RemoveDirt v0.9.2
Just now: experimental (and for me quicker) clang build inside.
Original code is almost completely rewritten by now.

Code:

- (20190324 v0.9.2)
  - RestoreMotionBlocks: 10-16 bit support. Relevant threshold and noise parameters are bit depth independent.
  - minor speedup
  - SCSelect: add support for 10-16 bits and 32 bit float clips
  - SCSelect: add support for planar RGB
  - FIX: SCSelect: make it work properly for large frames (>8MPixel)
  - FIX: SCSelect: Makes use the whole frame: now counts the rightmost non-mod32 pixels as well.
  - Clang 7.0.1 support (LLVM) with Visual Studio 2017 (15.9.9) (LLVM 8.0.0: not tried yet)
    - Install LLVM 7.0.1 (http://releases.llvm.org/download.html, Windows pre-built libraries)
    - Install Clang Power Tools & LLVM Compiler Toolchain
      - https://marketplace.visualstudio.com...langPowerTools
      - https://marketplace.visualstudio.com...llvm-toolchain
  - update html docs
  - add clang-built DLLs to the released version

real.finder · 24th March 2019, 19:34

Quote:

Originally Posted by pinterf

New build.
Download RemoveDirt v0.9.2
Just now: experimental (and for me quicker) clang build inside.
Original code is almost completely rewritten by now.

any plan to port RemoveGrainT to x64? it only one missed for run RemoveDirtMC_SE in x64

pinterf · 25th March 2019, 09:58

Quote:

Originally Posted by real.finder

any plan to port RemoveGrainT to x64? it only one missed for run RemoveDirtMC_SE in x64

Which function is used?

real.finder · 25th March 2019, 18:43

Quote:

Originally Posted by pinterf

Which function is used?

TemporalRepair in TemporalDeGrain_SE

pinterf · 6th April 2019, 08:04

Quote:

Originally Posted by real.finder

TemporalRepair in TemporalDeGrain_SE

I'm near the end of porting (including to make the source human readable, re-creating plain C language versions, these works take a huge amount of time as well). Right now I put it in the RemoveDirt DLL, I woudn't create a new DLL just for the sake of this single function. Opinion?

FranceBB · 27th March 2019, 23:55

Quote:

Originally Posted by pinterf

New build.
Download RemoveDirt v0.9.2
Just now: experimental (and for me quicker) clang build inside.
Original code is almost completely rewritten by now.

Code:

- (20190324 v0.9.2)
  - RestoreMotionBlocks: 10-16 bit support. Relevant threshold and noise parameters are bit depth independent.
  - minor speedup
  - SCSelect: add support for 10-16 bits and 32 bit float clips
  - SCSelect: add support for planar RGB
  - FIX: SCSelect: make it work properly for large frames (>8MPixel)
  - FIX: SCSelect: Makes use the whole frame: now counts the rightmost non-mod32 pixels as well.
  - Clang 7.0.1 support (LLVM) with Visual Studio 2017 (15.9.9) (LLVM 8.0.0: not tried yet)
    - Install LLVM 7.0.1 (http://releases.llvm.org/download.html, Windows pre-built libraries)
    - Install Clang Power Tools & LLVM Compiler Toolchain
      - https://marketplace.visualstudio.com...langPowerTools
      - https://marketplace.visualstudio.com...llvm-toolchain
  - update html docs
  - add clang-built DLLs to the released version

Thank you!

Out of curiosity, what are the benefits of the Clang compiled version over the normal x86 version?

Thank you in advance.

pinterf · 28th March 2019, 15:17

Clang sometimes generates quicker code, it seems that it features with smarter optimizer than MS. For example the above mentioned TemporalRepair has modes that run 10-15% faster in clang version than with MS built DLL.
I was also looking at the generated assembly code based on pure C source. clang was able to vectorize (parallel computing of pixel data using mm registers) situations where MS simply stuck with linear one-by-one processing.

tormento · 28th March 2019, 16:28

Quote:

Originally Posted by pinterf

Clang sometimes generates quicker code, it seems that it features with smarter optimizer than MS.

It would be nice to try your MVTools, MaskTools and RgTools with Clang...

real.finder · 28th March 2019, 21:15

I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016

tormento · 29th March 2019, 12:07

Quote:

Originally Posted by real.finder

I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016

XP is 18 years old. Time for it to go out from parents' home

pinterf · 29th March 2019, 12:29

Quote:

Originally Posted by real.finder

I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016

LLVM (clang) dropped XP support three years ago, since version 3.8. Now I'm using latest version (8.0)

14th March 2019, 19:51	#2 \| Link
pinterf Registered User Join Date: Jan 2014 Posts: 2,542	Finally I used the original Avisynth source, the VapourSynth port had bugs, I'll point to those places on my fork later next week. __________________ AviSynth+ on github, Other repos: RgTools, Masktools2, MvTools2, TIVTC, Average

14th March 2019, 22:38	#7 \| Link
ChaosKing Registered User Join Date: Dec 2005 Location: Germany Posts: 1,827	I think the VS port wasn't finished yet. The one dll I found crashes often. Thx for the release. I always wondered how hard it is to make "hybrid plugins" now which supports vs and avs+, since avs+ now supports 16bit and the same color spaces? Some plugins like this exists: knlmeans, f3kdb, ffms2 Or is it just too cumbersome because of the different threading models? p.s. You can also install RemoveDirt via avsrepo now __________________ AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK \|\| VapourSynth Database Last edited by ChaosKing; 14th March 2019 at 22:51.

15th March 2019, 04:19	#8 \| Link
real.finder Registered User Join Date: Jan 2012 Location: Mesopotamia Posts: 2,694	been waiting for this, thanks __________________ See My Avisynth Stuff

28th March 2019, 15:17	#16 \| Link
pinterf Registered User Join Date: Jan 2014 Posts: 2,542	Clang sometimes generates quicker code, it seems that it features with smarter optimizer than MS. For example the above mentioned TemporalRepair has modes that run 10-15% faster in clang version than with MS built DLL. I was also looking at the generated assembly code based on pure C source. clang was able to vectorize (parallel computing of pixel data using mm registers) situations where MS simply stuck with linear one-by-one processing. __________________ AviSynth+ on github, Other repos: RgTools, Masktools2, MvTools2, TIVTC, Average

28th March 2019, 21:15	#18 \| Link
real.finder Registered User Join Date: Jan 2012 Location: Mesopotamia Posts: 2,694	I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016 __________________ See My Avisynth Stuff

14th March 2019, 20:02	#3 \| Link
poisondeathray Registered User Join Date: Sep 2007 Posts: 5,669	Thanks

14th March 2019, 20:18	#4 \| Link
SaurusX Registered User Join Date: Feb 2017 Posts: 149	This is much appreciated.

14th March 2019, 20:49	#5 \| Link
filler56789 SuperVirus Join Date: Jun 2012 Location: Antarctic Japan Posts: 1,458	Thanks!

14th March 2019, 21:05	#6 \| Link
Reel.Deel Registered User Join Date: Mar 2012 Location: Texas Posts: 1,678	Thank you pinterf, I will update the wiki in the next couple of days.

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode