Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th October 2021, 18:51   #1  |  Link
Reclusive Eagle
Registered User
 
Join Date: Oct 2021
Posts: 51
Edge mask help

So currently I am not able to generate a decent edge mask that is sensitive enough to pick up all the lines while not picking up noise.

The noise is not that much of an issue. I can just denoise the mask but that feathers the mask edges creating issues like broken lines and reintroducing the masked out noise.
Here are some images the original image so you can see what I am working with. The edge mask and end result.

Basically what I am doing is denoising the clip but also selectively sharpening only edges and applying the mask to keep the edges from blurring due to BM3D's denoise.
The problem is, as you can tell in the Result. The mask is not sensitive enough to pick up all lines. Which leads to warped edges like the one circled in blue.

This is prevalent throughout the entire clip and I am losing details.
Currently I am using Animemask to make the edgemask. I have tried Sobel etc but they all are even less accurate creating horrible lines.
Is there someway I can increase sensitivity of masks? Or does anyone have any functions to create better masks? I've been trying these at Kageru. However, half of them I can't get to work and the other half aren't sensitive enough to pick up all the lines.

Also the Convolution matrix's create green masks. Idk why. Not an issue, just makes things harder to see.
I just want to be able to make a sensitive mask and then manipulate it to get what I want in white. I have tons of experience with Photoshop masks-
I just don't know how to use them through commands. Small things I would just brush out and end of story but you can't do that here.

Any advice? My project can't move forward without solving these basic issues.
Attachments Pending Approval
File Type: jpg original.jpg
File Type: jpg basic.jpg
File Type: jpg Result2.jpg

Last edited by Reclusive Eagle; 25th October 2021 at 19:34. Reason: Images
Reclusive Eagle is offline   Reply With Quote
Old 25th October 2021, 18:56   #2  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,400
Post your pictures to an external image host if you ever want comments.

Green masks are due to the UV planes being mostly 0 after processing. Only process the Y plane and it'll look better.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 25th October 2021, 19:36   #3  |  Link
Reclusive Eagle
Registered User
 
Join Date: Oct 2021
Posts: 51
Quote:
Originally Posted by Myrsloik View Post
Post your pictures to an external image host if you ever want comments.

Green masks are due to the UV planes being mostly 0 after processing. Only process the Y plane and it'll look better.
Yup, posted them to imgur. That's the thing though I am only processing the Y plane. Both 1 and 2 values give me the blue and red chroma layers so It's like? and if I convert it to RGB I get color noise in the gray mask. Either way, that's the least of my issues
Reclusive Eagle is offline   Reply With Quote
Old 26th October 2021, 00:10   #4  |  Link
Greenhorn
Registered User
 
Join Date: Apr 2018
Posts: 56
I'm aware that green masks are the smallest of your problems, but they can also very easily be solved by extracting luma from the clip before processing, something like:

clip = core.ffms2.Source(r"path/goes/here") ## or whatever your script looks like before masking
luma = core.std.ShufflePlanes(clip, 0, vs.GRAY)
## run your masking functions on "luma"
mask = (your masking function of choice here)
## denoise "clip"
denoised_clip = (your denoising function here)
core.std.MaskedMerge(denoised_clip, clip, mask)

of course, you might already be doing things this way, or a better way.

Anyway, I think there are still a lot of tools at your disposal. Just to throw a few things out there:

-- There are still more masking functions to try. Here's a few:
-- https://github.com/HomeOfVapourSynth...urSynth-TCanny
-- https://github.com/Ichunjo/vardefunc...ter/vardefunc/ (The "mask" submodule)
-- https://github.com/Irrational-Encodi...er/kagefunc.py (The "retinex_edgemask" function)

And there are definitely more, especially among ported avisynth filters I don't have much experience with as a relative newbie.

-- Prefiltering the clip before creating the mask; ie, creating a copy of it, and running a weak denoise or something like core.std.Median() to try and remove only the most extreme grain, and making the mask from the filtered clip.

-- Binarizing the mask (core.std.Binarize or core.std.BinarizeMask) to exclude low values and force everything else to full white.

-- Expanding the created mask with functions like core.std.Maximum(), core.std.Inflate(), or mt_expand_multi from havsfunc

Generally I've found that there's no free lunch though. Some masking functions are more accurate than others, but more sensitivity = more noise in the mask.

Also, regarding the denoising itself: I'm not an expert on such subjects, but I've found that BM3D can start deleting details at surprisingly low strengths, so I'd do as much experimentation with its parameters as I could before proceeding further. You might want to tweak it per-scene; while that's laborious, so is fiddling with masking functions.

I hope something in all of that is helpful.
Greenhorn is offline   Reply With Quote
Old 26th October 2021, 13:33   #5  |  Link
Reclusive Eagle
Registered User
 
Join Date: Oct 2021
Posts: 51
Quote:
Originally Posted by Greenhorn View Post
I'm aware that green masks are the smallest of your problems, but they can also very easily be solved by extracting luma from the clip before processing, something like:

clip = core.ffms2.Source(r"path/goes/here") ## or whatever your script looks like before masking
luma = core.std.ShufflePlanes(clip, 0, vs.GRAY)
## run your masking functions on "luma"
mask = (your masking function of choice here)
## denoise "clip"
denoised_clip = (your denoising function here)
core.std.MaskedMerge(denoised_clip, clip, mask)

of course, you might already be doing things this way, or a better way.

Anyway, I think there are still a lot of tools at your disposal. Just to throw a few things out there:

-- There are still more masking functions to try. Here's a few:
-- https://github.com/HomeOfVapourSynth...urSynth-TCanny
-- https://github.com/Ichunjo/vardefunc...ter/vardefunc/ (The "mask" submodule)
-- https://github.com/Irrational-Encodi...er/kagefunc.py (The "retinex_edgemask" function)

And there are definitely more, especially among ported avisynth filters I don't have much experience with as a relative newbie.

-- Prefiltering the clip before creating the mask; ie, creating a copy of it, and running a weak denoise or something like core.std.Median() to try and remove only the most extreme grain, and making the mask from the filtered clip.

-- Binarizing the mask (core.std.Binarize or core.std.BinarizeMask) to exclude low values and force everything else to full white.

-- Expanding the created mask with functions like core.std.Maximum(), core.std.Inflate(), or mt_expand_multi from havsfunc

Generally I've found that there's no free lunch though. Some masking functions are more accurate than others, but more sensitivity = more noise in the mask.

Also, regarding the denoising itself: I'm not an expert on such subjects, but I've found that BM3D can start deleting details at surprisingly low strengths, so I'd do as much experimentation with its parameters as I could before proceeding further. You might want to tweak it per-scene; while that's laborious, so is fiddling with masking functions.

I hope something in all of that is helpful.

That's actually extremely helpful. I am new to Vapour and AVI synth (only discovered them last week and basically have been pouring every hour of the day into learning Python, reading technical documentation etc) so any advice is amazing.

But yes BM3D does alter details which is why I need to use a mask with it. Dfftest denoises the scene perfectly however, it only denoises about 45% of the noise. And even less if you upscale before denoise. Which is where BM3D comes in because it can eliminate all noise at the cost of sharpness and accuracy which is what I am trying to find a way around.

It doesn't help that I am working with a PAL video with the best source coming from 5k bit rate dvd's from 2007. Do you have any advice beyond denoising and sharpening for this siuation?

I find upscaling before sharpening does eliminate a lot of the sharpened noise. But would I benefit from "oversampling" the video by denoising at higher resolution? E.g I scale it to 4k, apply all filters and then downscale to 1080p? I assume this would eliminate Alising at least.
Reclusive Eagle is offline   Reply With Quote
Old 26th October 2021, 13:38   #6  |  Link
Reclusive Eagle
Registered User
 
Join Date: Oct 2021
Posts: 51
Quote:
Originally Posted by Greenhorn View Post
I'm aware that green masks are the smallest of your problems, but they can also very easily be solved by extracting luma from the clip before processing, something like:

clip = core.ffms2.Source(r"path/goes/here") ## or whatever your script looks like before masking
luma = core.std.ShufflePlanes(clip, 0, vs.GRAY)
## run your masking functions on "luma"
mask = (your masking function of choice here)
## denoise "clip"
denoised_clip = (your denoising function here)
core.std.MaskedMerge(denoised_clip, clip, mask)

of course, you might already be doing things this way, or a better way.

Anyway, I think there are still a lot of tools at your disposal. Just to throw a few things out there:

-- There are still more masking functions to try. Here's a few:
-- https://github.com/HomeOfVapourSynth...urSynth-TCanny
-- https://github.com/Ichunjo/vardefunc...ter/vardefunc/ (The "mask" submodule)
-- https://github.com/Irrational-Encodi...er/kagefunc.py (The "retinex_edgemask" function)

And there are definitely more, especially among ported avisynth filters I don't have much experience with as a relative newbie.

-- Prefiltering the clip before creating the mask; ie, creating a copy of it, and running a weak denoise or something like core.std.Median() to try and remove only the most extreme grain, and making the mask from the filtered clip.

-- Binarizing the mask (core.std.Binarize or core.std.BinarizeMask) to exclude low values and force everything else to full white.

-- Expanding the created mask with functions like core.std.Maximum(), core.std.Inflate(), or mt_expand_multi from havsfunc

Generally I've found that there's no free lunch though. Some masking functions are more accurate than others, but more sensitivity = more noise in the mask.

Also, regarding the denoising itself: I'm not an expert on such subjects, but I've found that BM3D can start deleting details at surprisingly low strengths, so I'd do as much experimentation with its parameters as I could before proceeding further. You might want to tweak it per-scene; while that's laborious, so is fiddling with masking functions.

I hope something in all of that is helpful.
Side note how do you actually use retinex edgmask? It can't just be
kagefunc.retinex_edgemask(Clip, sigma=1)
As the only functions?


Also am I doing something wrong here? I'm trying out vardefunc mask. Installed it and here is my code

import vardefunc as vf
clip= vf.get_mask(clip, lthr = 0.0, multi= 1.0)

AttributeError: module 'vardefunc.mask' has no attribute 'get_mask'

Last edited by Reclusive Eagle; 26th October 2021 at 14:16.
Reclusive Eagle is offline   Reply With Quote
Old 26th October 2021, 15:30   #7  |  Link
Greenhorn
Registered User
 
Join Date: Apr 2018
Posts: 56
Quote:
Originally Posted by Reclusive Eagle View Post
Side note how do you actually use retinex edgmask? It can't just be
kagefunc.retinex_edgemask(Clip, sigma=1)
As the only functions?


Also am I doing something wrong here? I'm trying out vardefunc mask. Installed it and here is my code

import vardefunc as vf
clip= vf.get_mask(clip, lthr = 0.0, multi= 1.0)

AttributeError: module 'vardefunc.mask' has no attribute 'get_mask'
For the first, if I imported kagefunc as kgf it'd just be kgf.retinex_edgemask(clip), yeah

For script collections with submodules you have to specify the submodule. For that one it'd be

vf.mask.Kirsch().get_mask(clip)

where Kirsch() can be replaced with any of the other methods in the mask.py file. (The various classes defined as Class MaskName(EdgeDetect))

Also, apparently most of the masks in that actually came from G41Fun (https://github.com/Vapoursynth-Plugins-Gitify/G41Fun), so maybe give its EdgeDetect function a spin.
Greenhorn is offline   Reply With Quote
Old 26th October 2021, 16:14   #8  |  Link
Reclusive Eagle
Registered User
 
Join Date: Oct 2021
Posts: 51
Quote:
Originally Posted by Greenhorn View Post
For the first, if I imported kagefunc as kgf it'd just be kgf.retinex_edgemask(clip), yeah

For script collections with submodules you have to specify the submodule. For that one it'd be

vf.mask.Kirsch().get_mask(clip)

where Kirsch() can be replaced with any of the other methods in the mask.py file. (The various classes defined as Class MaskName(EdgeDetect))

Also, apparently most of the masks in that actually came from G41Fun (https://github.com/Vapoursynth-Plugins-Gitify/G41Fun), so maybe give its EdgeDetect function a spin.
Thank you so much! Your help has been amazing!
There really is no info on edgemasks at all for Vapoursynth so I actually think you've accidently made the first guide.

Also, I figured out if you upscale before denoising with BM3D then downscale back to original resolution before maskmerge BM3D
does not modify any of the line work but still denoises effectively.

And since you can also use any clip as reference, denoising with Dfftest, upsacling and using that as a reference for BM3D final before downscaling again.
Should result in clean denoising no matter how aggressive the settings

Last edited by Reclusive Eagle; 26th October 2021 at 16:16.
Reclusive Eagle is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 12:18.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.