Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th January 2008, 21:51   #1  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
ExInpaint plugin - Exemplar-Based Image Inpainting

Exemplar-Based Image Inpainting - removing large objects from images.

Selected objects has been replaced by a visually plausible background that mimics the appearance of the source region. Unlike other inpainting tools (logo removers), this filter does not blur image "holes". It tries to replace holes by similar texture elements (patches, exemplars, blocks) from frame image area, taking into account the image special points (with estimated confidence and priority). Block inpainting is started at hole border and propagates to its internal area, with isophote as a direction.

Algorithm is based on article: Object Removal by Exemplar-Based Inpainting. A. Criminisi, P. Perez, K. Toyama. In Proc. Conf. Comp. Vision Pattern Rec., Madison, WI, Jun 2003. http://research.microsoft.com/vision...isi_cvpr03.pdf

Code is based on project of Qiushuang Zhang
http://www.cc.gatech.edu/grads/q/qsz...inpainting.htm

First version of ExInpaint plugin is slow, not optimized, especially for large radius.

It is spatial filter developed for still images. Only current frame is used. So, we can see temporal instability in this version.

Potentially it may be used in video for:
-film restoration (spots and scratches removal);
-logo removal;
-filling of empty areas (borders) after motion stabilization (compensation);
-filling damaged areas after VHS capture;
-frame increasing without resizing :-)

It can be implemented differently, may be as a part of MVTools. Current version is mostly for experiment and testing.

For logo removing and similar tasks you can also try XLogo, AVSInpaint, Greycstoration inpainting filters.
Fizick is offline   Reply With Quote
Old 19th January 2008, 06:26   #2  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
this is pretty amazing actually :|

i think i might hack this to do scratch concealment.

[edit]

seems to be RGB only?

it works really well! try this:

Code:
stackhorizontal(last.crop(0,0,64,last.height),last.blankclip(width=4,color=$0000ff),last.crop(68,0,652,last.height))#blankclip(width=652,color=$000000))

exinpaint(last,last,radius=16,xsize=3,ysize=3,color=$0000ff)
the stackhorizontal is to simulate a film scratch. it could be replaced by an automated scratch detector using some masks.
__________________
sucking the life out of your videos since 2004

Last edited by Mug Funky; 19th January 2008 at 06:34.
Mug Funky is offline   Reply With Quote
Old 23rd January 2008, 08:04   #3  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
little question:
Am I overlooking something? (can't find a avisynth filter, just a download for papers and a command line executeable)

btw. tested the executeable with a little smaple I tool form google picture search:
Original: http://www.postimage.org/image.php?v=aVV6qzA
Masked: http://www.postimage.org/image.php?v=aVV6Ay9
Result: http://www.postimage.org/image.php?v=aVV6Fxr

Cu Selur
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 23rd January 2008 at 08:26.
Selur is offline   Reply With Quote
Old 23rd January 2008, 08:28   #4  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Fizick's plugins
foxyshadis is offline   Reply With Quote
Old 23rd January 2008, 08:33   #5  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 7,259
thanks

http://avisynth.org.ru/ seems to be down atm, would be nice if someone who got the filter&co could up it to megaupload or something like it.
-> works again
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 23rd January 2008 at 12:54.
Selur is offline   Reply With Quote
Old 24th January 2008, 02:49   #6  |  Link
Konrad Klar
Registered User
 
Konrad Klar's Avatar
 
Join Date: Nov 2005
Location: Wałbrzych, Poland
Posts: 105
@Fizick
Thanks for plugin.
Two questions:
1. Does it make difference in speed which colorspace is used (RGB24, RGB32, YV12)?
2. Is there any chance in near future for multithreaded version of ExInpaint? It does not work (at least for me) with MT() nor with SetMTMode(). Maybe because it takes more than one clip as argument.
Konrad Klar is offline   Reply With Quote
Old 24th January 2008, 16:59   #7  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
1.1 You can measure it yourself
I almost finished next speed-optimized version (with SSE MMX).
RGB32, RGB24, YV12 are ready. But YUY2 optimizing is not trivial task in MMX.
2.1. I do not develep m.t. filters (do not have hardware).
2.2. exinpaint can be used with some tricks with mask.
2.3. But according to algo the filter inpaints one block per a time (iteration).
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 13th April 2008, 18:48   #8  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
I found that I developed version 0.2 at 26 January 2008

-even patch sizes instead of odd (with some approriate changes)
-assembler MMX optimization of PatchTexture
-use SAD instead of SSD
-local max priority updating
-option to estimate search radius automatically
-added YUY2 format and YUV24
-added dilation of mask

But I am afraid this version is not finally finished (beta or even alpha stage).
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 15th April 2008, 18:11   #9  |  Link
TSchniede
Registered User
 
Join Date: Aug 2006
Posts: 77
Some sort of overlapping for block prevention would be nice too.

And some addition for the documentation: the color for yv12-mode is based on the binary representation of the mask color NOT the converted color of RGB. So $FFFFFF is quite colorful instead of white.

Example:
Code:
enlargedclip=clip.converttorgb().addborders(8,8,8,8,color=$FFFFFF).converttoyv12(matrix="pc.601")
exinpaint(enlargedclip,enlargedclip,color=$FF8080)
remark: the color conversion introduces a 1-pixel line around the image which will not get inpainted.This is only for demonstration.
TSchniede is offline   Reply With Quote
Old 23rd April 2008, 17:20   #10  |  Link
AnnaFan777
Registered User
 
Join Date: Apr 2008
Posts: 51
Thank you, it's very promising.

Hope the temporal instability can be fixed soon.

btw, the border of inpaint area look sharper, somewhat
noticeble if you look carefully, now if only we can
blend the adjacent pixels a bit to make it more natural.
AnnaFan777 is offline   Reply With Quote
Old 18th September 2008, 20:17   #11  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Today I found that I forget to release version 0.2 beta in april (again!)
Prabably becourse of not many responces....

So, released now.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 19th September 2008, 00:07   #12  |  Link
mikeytown2
Resize Abuser
 
mikeytown2's Avatar
 
Join Date: Apr 2005
Location: Seattle, WA
Posts: 623
I've used it a couple of times on photos, thanks for the update!
mikeytown2 is offline   Reply With Quote
Old 3rd October 2008, 00:00   #13  |  Link
Revgen
Registered User
 
Join Date: Sep 2004
Location: Near LA, California, USA
Posts: 1,545
I'm trying to figure out how to remove a logo with this tool. I've yet to figure out how.

Any clues or tips would be appreciated.

Here's a sample with a logo I'm trying to remove.

http://www.sendspace.com/file/5atrwo
__________________
Pirate: Now how would you like to die? Would you like to have your head chopped off or be burned at the stake?

Curly: Burned at the stake!

Moe: Why?

Curly: A hot steak is always better than a cold chop.
Revgen is offline   Reply With Quote
Old 3rd October 2008, 00:05   #14  |  Link
mikeytown2
Resize Abuser
 
mikeytown2's Avatar
 
Join Date: Apr 2005
Location: Seattle, WA
Posts: 623
U try this?
http://avisynth.org/mediawiki/Rm_logo

ExInpaint is commented out, but you can uncomment it and try it out.
mikeytown2 is offline   Reply With Quote
Old 6th January 2011, 23:47   #15  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Here is the message i have got recently from Jean (i have no clear answer to it):
Quote:
... i'd like to ask you a bit about
exinpaint which i found in your plugins list. I really like that tool
and it produces good output; However, i wanted to ask you, how an
implementation into MVTools could look like (i'm new to avisynth, so
i'm not very skilled yet)

I used exinpaint to do the same in your example with some borders,
sadly every frame the the inpainting is changed, even if the edges are
very similar. Thats why i think an implementation into mvtools could be
very useful, i just have no idea how.

Let's imagine we have a video with 800x600px. Now we add 20px border at
the top, and 20px border at the bottom. Now i use exinpaint with 24px
top and 24px bottom to get the border's inpainted (produces a good
result so far) though on the next frame this inpainting changed while
the rest of the frame did not change. (there's maybe only 10% in the
whole frame changing, the only thing i can see is some dust in the
middle and a girl moving the mouth cuz of talking, - the dust doesn't
reach any of the borders, it's really in the middle) so, what i would
need, would be something to "stabilize" the inpainting. An idea i'd
have would be:


- Create a Mask in the same size around the mask (yeah, i'm
complicated explainer) so if we have a border of 24 px at the top, we
make a mask of it with 28 px which gets inpainted. Now we create a
second mask BELOW of it with 28px in which we check for motion - if
there's no motion use the inpainting of the previous frame. That's just
an example if we do something at the top. Same functionality should be
right, left and at the bottom, though individually (because, while at
the top nothing might change, at the bottom something could change, etc)

You think that could work? And you got any idea how to implement
something like that?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 7th January 2011, 00:03   #16  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
A black/white decision (either there is motion, or there is not) doesn't sound very promising. The forseeable result is false locking when the frame is slightly moving, with the funky garbling randomly kicking in when the holy motion threshold happens to be exceeded.

The better question is why ExInpaint delivers vastly different inpainting results when hardly anything changes between the frames ...
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 7th January 2011, 00:04   #17  |  Link
chani
Registered User
 
Join Date: Dec 2010
Posts: 2
Hey,

thanks for posting fizzick, that was me, with the mail, i couldn't post earlier here, because of the "you have to be 5 days registered to write"-rule. However, back to the problem/idea:

I also talked with manao and kuukunen on IRC, it came out that what i wanted to do is most likely not possible. Because i simply can't take the result of the previous frame (at least i have no idea how, i tried with imagewrite and imageread, though that didnt work because the image is only read once, and not on every request) i tried with trim, duplicateframe.. So.. There's some additional magic needed i guess :-)

Some people might say that this is useless at all, because it wouldn't give good results, however, the result's i've seen in my testing clip were good (especially if you just want to do that at borders. The less you want to add to the borders the better - This might be useful for turning 4/3 into 16/9 video. My idea was just to make masks around the borders, so that we have a left, top, bottom and right mask, and just checking whether there's motion in a mask - if there is, do inpainting, if not, use the previously inpainted one. Dunno. I weren't able to get something like that working and i was told it's impossible, so i gave up yet. Though if anyone else has any idea about it ... i'd be interested.
__________________
a computer without windows is like a chocolatecake without mustard
chani is offline   Reply With Quote
Old 7th January 2011, 00:30   #18  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
i think the algo would need to be rewritten almost completely - we'd be adding a new dimension to it.

the idea of using mvtools' vector field (and associated confidence levels, scene detection, etc etc) is pretty exciting though. the algorithm could track it's choices of exemplars (is this correct terminology?) along a contiguous scene, and update them locally when there's large enough changes, then possibly revert back if the scene returns to how it was earlier...

i have no idea how to do this

exinpaint already works beautifully for removing film scratches, but must be combined with a very heavy temporal blur (mv'd of course) to become invisible. also it can't handle very thick scratches without becoming apparent.

i've saved the edges of quite a few films where the camera mag was unkind to them - those black vertical lines just vanish and leave no trace if you zero in well enough. i love it when clients think you've just performed an act of magic.
__________________
sucking the life out of your videos since 2004
Mug Funky is offline   Reply With Quote
Old 2nd March 2011, 13:51   #19  |  Link
Floatingshed
Registered User
 
Join Date: Nov 2008
Posts: 322
This looks promising, can somebody please give me a few pointers to use it to remove film scatches? Thanks.
Floatingshed is offline   Reply With Quote
Old 22nd February 2013, 09:14   #20  |  Link
fbert
Registered User
 
Join Date: Feb 2013
Posts: 1
exemplar based inpainting

I have problems to do about exemplar based inpainting..
I'm doing my final project and confused about some formula written on paper object removal exemplar based inpainting..
Could you help me?
My problems are about to descript formula..
I'm confused about three formula:
1. Np... Is it just giving 1 to source region and. 0 to target region?
2. Dp = (np*(symbol of inverted triangle) Ip ) / alpha
What is the mean of (symbol of inverted triangle) Ip. What should i do about that?
3. Psi Q... How to get that...
fbert is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:07.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.