help improving and scaling this anime (DVD R1 Source)

[OvejaNegra]

Im trying to upscale this anime to 1440 X 1080 using

nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=1, etype=0, pscrn=1, threads=0, opt=0, fapprox=15,cshift="spline36resize",fwidth=1440,fheight=1080)

Results for now are great (suggestions welcomed) but first i want to clean a little bit.

The source has some bright halos on the RIGHT of the lines, removing them using dehalo / finedehalo gives me acceptable resuls, but no good. That without (off course) killing the lines or creating blurring.

Halos look like this (post-scale result, halos processed on original resolution- green rectangles):


After dehalo, this is the result (the best i could do)



I want to apply proToon to darken the lines, but i have a problem: Using protoon AFTER dehalo gives me some broken lines. Using protoon BEFORE dehalo gives me more clean lines, but affects dehalo results.
Is possible to "create" the protoon mask before the dehalo pass and apply it later (i don't know if i'm making the right question)?

This is the best i could do without both filters opposing each other:



Another question: The chroma has a lot of noise, i was trying to saturate a little bit but it looks horrible and noisy. Also, it adds chroma noise to dark / white areas: Is possible to create a mask with the lines, the black areas and the white areas to "protect them" during saturation"? (i know the example picture is over saturated, is intentional to show the noise).



Making it look so clean like a flash animation or a modern anime is too much but i want to try. Thanks

[real.finder]

DeHaloHmod

[brucethemoose]

Would long encoding times be an issue?

[OvejaNegra]

Sorry for delay. No encoding times is not issue.
In the end FineDehalo gave me acceptable results and now im working with protoon.
Also im testing Satmask for saturating the dull parts of the image (like Vibrance).
I need something like that for the dark / white zones -> Making the dark zones really dark and some overbright zones, not that bright.
Also, the white zones has some blue tint, so maybe white point correction will be necesary.
Advices are always welcome.

[SSH4]

First rules of up-scaling anime - never use R1 sources. Use only R2 Japan sources that have 2-3x times bigger bit-rate than 99,99% of R1 sh*t.

[brucethemoose]

Originally Posted by OvejaNegra

Sorry for delay. No encoding times is not issue.
In the end FineDehalo gave me acceptable results and now im working with protoon.
Also im testing Satmask for saturating the dull parts of the image (like Vibrance).
I need something like that for the dark / white zones -> Making the dark zones really dark and some overbright zones, not that bright.
Also, the white zones has some blue tint, so maybe white point correction will be necesary.
Advices are always welcome.

If you really mean that (I'm talking ~2 FPS or worse), Waifu and Vine are pretty amazing:

http://www.screenshotcomparison.com/comparison/185123

(Pupils are messed up in that screenshot due to too much dehaloing, but you get the idea)

I had less luck with FineDehalo, but my halo problems were much more severe.

[OvejaNegra]

sorry for delay.
I compared waifu with my setup, and i prefer mine, waifu looks.... wierd
What's vine? Can't find anything about it

I know about R1 VS R2 DVDs, but R1 i what i have

[brucethemoose]

Originally Posted by OvejaNegra

sorry for delay.
I compared waifu with my setup, and i prefer mine, waifu looks.... wierd
What's vine? Can't find anything about it

I know about R1 VS R2 DVDs, but R1 i what i have

A very, very slow VapourSynth dehaloer. I'm in the wrong forum section for that :P

Be sure to turn down denoising to 0 if you try Waifu again, or just turn it off with -1. Anything higher just messes it up, even in my source.



Anyway, my problem with dehaloing is that it blurs/breaks some dark lines. Since halos are usually white, my plan is to mask out the filter in areas below a darkness threshold, to save the black lines... maybe you could try something similar.

[OvejaNegra]

i have the same problem, i can make the halos dissapear, at the cost of blurring dark lines.
That was why i was thinking on creating a mask with the dark lines before the dehalo filters, and using that mask after to "paste" the dark lines over the filtered clip. But i'm new with the mask filters
so im trying.

[real.finder]

Originally Posted by brucethemoose

A very, very slow VapourSynth dehaloer. I'm in the wrong forum section for that :P

Be sure to turn down denoising to 0 if you try Waifu again, or just turn it off with -1. Anything higher just messes it up, even in my source.



Anyway, my problem with dehaloing is that it blurs/breaks some dark lines. Since halos are usually white, my plan is to mask out the filter in areas below a darkness threshold, to save the black lines... maybe you could try something similar.

I already did this, but the Thread writer seems ignore my post

http://avisynth.nl/index.php/DeHaloHmod < but don't have all Parameters and updates

[brucethemoose]

Originally Posted by real.finder

I already did this, but the Thread writer seems ignore my post

http://avisynth.nl/index.php/DeHaloHmod < but don't have all Parameters and updates

Thanks. I'll try it too if I can import it to VS.

[OvejaNegra]

I already did this, but the Thread writer seems ignore my post

Slow down please, yes i tried your filter but i was not able to remove the halos with it, maybe some advice from you will help.

DeHaloHmod(Radius=8, Str=16, Maska=false, strong=true, mode=2, thr=32, analog=false, dirty=false, smooth=false, anime=true, pel2=false, usedeen=false, maskpel2=false, thr2=32,exdehalo="FineDehalo(rx=2, ry=1, thmi=5, thma=128, thlimi=10, thlima=250, darkstr=1.0, brightstr=2.0, showmask=0, contra=0.0, excl=true, edgeproc=0.0)")

Those gave me the best results, and yes i tried with and without and external dehaloer.

Also, my goal is a little different: Make a mask from the lines, do several stages of filtering (not just dehalos) and "paste" the black lines after all that:
For example: Playing with sat mask some times make the lines around saturated objects dark red or dark blue. I suposse this is for the lower resolution chroma, but im not going to stretch this to 4:4:4

Thanks

[real.finder]

Originally Posted by OvejaNegra

Slow down please, yes i tried your filter but i was not able to remove the halos with it, maybe some advice from you will help.

DeHaloHmod(Radius=8, Str=16, Maska=false, strong=true, mode=2, thr=32, analog=false, dirty=false, smooth=false, anime=true, pel2=false, usedeen=false, maskpel2=false, thr2=32,exdehalo="FineDehalo(rx=2, ry=1, thmi=5, thma=128, thlimi=10, thlima=250, darkstr=1.0, brightstr=2.0, showmask=0, contra=0.0, excl=true, edgeproc=0.0)")

Those gave me the best results, and yes i tried with and without and external dehaloer.

Also, my goal is a little different: Make a mask from the lines, do several stages of filtering (not just dehalos) and "paste" the black lines after all that:
For example: Playing with sat mask some times make the lines around saturated objects dark red or dark blue. I suposse this is for the lower resolution chroma, but im not going to stretch this to 4:4:4

Thanks

for the only mask EMask_dhh()

only mode=3 do mask for black lines, others just normal edges mask and with anime=true they maybe make the halo past from the source

and seems you dehalo after upscale, it's better to do it before that, and why you didn't post video sample?

edit: with simple code using your upscale pic

ImageSource("e84ll4_jpg.png")
Spline36Resize(720,480)
ConvertToYV12()
DeHaloHmod(smooth=true)
nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=1, etype=0, pscrn=1, threads=0, opt=0, fapprox=15,cshift="spline36resize",fwidth=1440,fheight=1080)

and of course you can use your dehalo in exdehalo