Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th April 2024, 00:21   #1  |  Link
simple_simon
Registered User
 
Join Date: Feb 2003
Posts: 124
Fix distorted aliasing from bad upscales

Is there any way to reverse this?: https://ibb.co/G5kXwBz
It looks like the video was badly upsized from 480p to 1080p without correcting the existing line aliasing so now the aliasing is stretched and impossible to fix with normal dealiasing filters. I came up with this filter chain:

Code:
propSet("_FieldBased",0).deep_resize(720,240,grain=0,qual=2,gpuid=-1,show=false)
maa2()
deep_resize(720,480,grain=0,qual=2,gpuid=-1,show=false)
maa2()
lsfplus(strength=40,preset="slow",preblur="neo_fft3d(sigma=4,u=2,v=2)")
deep_resize(960,720,grain=0,qual=2,gpuid=-1,show=false)
which works decently but figured I'd crowdsource and see if there a better solution I'm unaware of.
simple_simon is offline   Reply With Quote
Old 17th April 2024, 02:08   #2  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,416
Downscale, perhaps using an inverse kernel / desampler, then use 1x_AnimeUndeint_Compact_130k_net_g, then upscale. Adjust to taste.

I fiddled a bit with a height below 480 which seemed helped better with some aliasing artifacts, you can play with that and other kernels as well. FMTC has inverse kernels too

https://imgsli.com/MjU2Mjcx

Code:
ImageSource("snapshot-02-39-411.png")
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ncnn(network_path="PATH\1x_AnimeUndeint_Compact_130k_net_g.onnx", builtin=false)
mlrt_ncnn(network_path="PATH\2X_DigitalFilmV5_Lite.onnx", builtin=false)
z_convertformat(width=1440, height=1080, use_props=0)
The pavement is slightly less red from one of the models. When you convert it back to YUV, smoothtweak(hue2=1) will make it more like the original
poisondeathray is offline   Reply With Quote
Old 17th April 2024, 03:52   #3  |  Link
simple_simon
Registered User
 
Join Date: Feb 2003
Posts: 124
Those are a bunch of tools I'm not familiar with so it'll take me some time to play around with that (I don't even have a need to convert colorspace enough to know offhand how to convert back to YUV). But the results are damn impressive. Thanks for the suggestions.
simple_simon is offline   Reply With Quote
Old 17th April 2024, 20:21   #4  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,628
Wow PDR, that is impressive.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 17th April 2024, 21:02   #5  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,416
Quote:
Originally Posted by simple_simon View Post
Those are a bunch of tools I'm not familiar with so it'll take me some time to play around with that (I don't even have a need to convert colorspace enough to know offhand how to convert back to YUV). But the results are damn impressive. Thanks for the suggestions.


What I mean is the models work in 32bit RGB float (the mlrt_ncnn steps). But usually people encode to 8 or 10bit 4:2:0 YUV (but some people use 4:4:4 when models are used because some of them sharpen the chroma significantly)

The output of the script as-is before the last line is 1440x800 because the downscale was 720x400, then a 2x model was applied

The last line resamples to 1440x1080 (to compare to same as input dimensions) . If you wanted 8bit 4:2:0 , you can add pixel_type="YV12" or whatever format you're encoding to .

Models are slow - you you could use deep_resize instead like you did before instead of 2X_DigitalFilmV5_Lite - whatever you feel like . Most of the problem fixing for the aliasing is from 1x_AnimeUndeint_Compact_130k_net_g on a downscaled input . It's really good model for anime or cartoon type aliasing issues

Before commiting to the Smoothtweak adjustment, I would examine other frames with other colors as well. Look at histogram("levels") and compare the U,V channels

Code:
z_convertformat(width=1440, height=1080, pixel_type="YV12", use_props=0)
Smoothtweak(hue2=1)

If you're stuck and can't find the answers just ask and someone will help out
poisondeathray is offline   Reply With Quote
Old 17th April 2024, 23:04   #6  |  Link
lollo2
Registered User
 
lollo2's Avatar
 
Join Date: Aug 2017
Location: Italy
Posts: 115
Quote:
Originally Posted by poisondeathray View Post
Downscale, perhaps using an inverse kernel / desampler, then use 1x_AnimeUndeint_Compact_130k_net_g, then upscale. Adjust to taste.

I fiddled a bit with a height below 480 which seemed helped better with some aliasing artifacts, you can play with that and other kernels as well. FMTC has inverse kernels too

https://imgsli.com/MjU2Mjcx
Excellent result, master!
__________________
A channel on S-VHS / VHS capture and AviSynth restoration https://www.youtube.com/channel/UCMs...h1MmNAs7I8nu4g
lollo2 is offline   Reply With Quote
Old 18th April 2024, 10:24   #7  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,625
Quote:
Originally Posted by poisondeathray View Post
ImageSource("snapshot-02-39-411.png")
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ncnn(network_path="PATH\1x_AnimeUndeint_Compact_130k_net_g.onnx", builtin=false)
mlrt_ncnn(network_path="PATH\2X_DigitalFilmV5_Lite.onnx", builtin=false)
z_convertformat(width=1440, height=1080, use_props=0)
Can AVSLibPlacebo or FMTC replace the 2 z_convertformat instructions?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 18th April 2024, 12:17   #8  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,175
You can even try internal and eveready ConvertBits(32) to go into 32bits float RGB.
DTL is offline   Reply With Quote
Old 18th April 2024, 13:14   #9  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,625
Quote:
Originally Posted by dtl View Post
you can even try internal and eveready convertbits(32) to go into 32bits float rgb.
rgb=rgbps?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 18th April 2024, 14:48   #10  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,175
RGB with 32bit (float) per sample is RGBPS. AVS do not support integer 32bit samples so ConvertBits(32) can only convert to float.

So you first convert to Planar RGB and next convert any sample format to float32. Typically RGB float32 is the only common format accepted by 'neural-networks' libraries as most fail safe and most dynamic range capable. It is not optimized for performance but easy to implement by current very poor residuals of opensource developers.

The last z_convertformat(width=1440, height=1080, use_props=0) do not change sample format and only resize so you again can use any internal resize (or any other installed).

To go back to some usable by encoders you anyway need to add something like ConvertToYV12() or if it can not make direct convert from float32 - ConvertBits(8).ConvertToYV12()

See example how to feed that NN-plugins with RGBPS format with internal conversions and back - https://forum.doom9.org/showthread.p...55#post1983255

Last edited by DTL; 18th April 2024 at 14:55.
DTL is offline   Reply With Quote
Old 20th April 2024, 05:57   #11  |  Link
simple_simon
Registered User
 
Join Date: Feb 2003
Posts: 124
Quote:
Originally Posted by poisondeathray View Post
The last line resamples to 1440x1080 (to compare to same as input dimensions) . If you wanted 8bit 4:2:0 , you can add pixel_type="YV12" or whatever format you're encoding to .

Models are slow - you you could use deep_resize instead like you did before instead of 2X_DigitalFilmV5_Lite - whatever you feel like . Most of the problem fixing for the aliasing is from 1x_AnimeUndeint_Compact_130k_net_g on a downscaled input . It's really good model for anime or cartoon type aliasing issues

Before commiting to the Smoothtweak adjustment, I would examine other frames with other colors as well. Look at histogram("levels") and compare the U,V channels
I've got this setup and have been playing around with it. The results seem like some sort of wizardry. It's really incredible. I can see these and other AI models being incredibly useful in restoring a whole stack of badly encoded dvd's and old avi files that no longer have original sources that I've never been able to fix in any sufficiently satisfying way.

As to this particular clip I didn't end up using the smoothtweak because looking at it in AvsPmod with Prefetch RGB Display Conversion turned on I didn't see any color difference before and after filtering. I made a few minor modifications to your script but can you verify whether I'm making any mistakes converting back to YUV?

Code:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()

Last edited by simple_simon; 20th April 2024 at 07:17.
simple_simon is offline   Reply With Quote
Old 20th April 2024, 06:13   #12  |  Link
StvG
Registered User
 
Join Date: Jul 2018
Posts: 487
The line z_convertformat(pixel_type="RGBPS", use_props=0) means colorspace_op="601=>rgb". If you want 709->rgb do z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left").
StvG is offline   Reply With Quote
Old 20th April 2024, 07:15   #13  |  Link
simple_simon
Registered User
 
Join Date: Feb 2003
Posts: 124
Quote:
Originally Posted by StvG View Post
The line z_convertformat(pixel_type="RGBPS", use_props=0) means colorspace_op="601=>rgb". If you want 709->rgb do z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left").
Since the source is 1080p isn't it already in full colorspace? I wouldn't want to force limited-->full. Shouldn't it be full-->full?
simple_simon is offline   Reply With Quote
Old 20th April 2024, 09:56   #14  |  Link
StvG
Registered User
 
Join Date: Jul 2018
Posts: 487
The resolution doesn't define if the video is full or limited. If it's YUV (especially DVD or BD, or UHD) then it's usually (read 99% of the cases) limited.
StvG is offline   Reply With Quote
Old 20th April 2024, 11:21   #15  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,175
Quote:
Originally Posted by simple_simon View Post
Since the source is 1080p isn't it already in full colorspace? I wouldn't want to force limited-->full. Shouldn't it be full-->full?
It was a note like "Never use complex plugins with lots of hidden defaults left unchanged. The internal defaults may not match your source and no metadata magic may helps."

The main issue is not with range mapping but with system matrix. Your upscaled source may or may not use 709 hd-typical matrix (depending on previous history of conversion from sd source with sd-typical 601 matrix). And z_convertformat if options left unchanged will use 601 matrix.

"z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full","

For moving pictures typically used limited/narrow range mapping in YUV too. The RGBPS allows to keep all required data. So better to use z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:limited", . I also not know where do you going to use possible YUV420P8 format with full range mapping. It may be unknown to many YUV decoders.

There is unfortunately no any warnings output in AVS yet in some non-stopping of processing way so Avsresize do not emit stop-exception on the YUV+full format request and expect user understand how it will be looks like. In a more happy way it may simply ignore full-flag for RGB->YUV conversions.

Quote:
Originally Posted by StvG View Post
The line z_convertformat(pixel_type="RGBPS", use_props=0) means colorspace_op="601=>rgb". If you want 709->rgb do z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left").
In the script of

ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)

the z_convertformat is used as RGBPx -> RGBP 32bit float conversion engine only. No de-matrix YUV to RGB. So it should not need matrix operation param ?

So user must provide correct matrix param to ConvertToPlanarRGB() instead of default left to have correct conversion.

Last edited by DTL; 20th April 2024 at 12:01.
DTL is offline   Reply With Quote
Old 20th April 2024, 14:14   #16  |  Link
StvG
Registered User
 
Join Date: Jul 2018
Posts: 487
Quote:
Originally Posted by DTL View Post
... In the script of

ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", use_props=0)

the z_convertformat is used as RGBPx -> RGBP 32bit float conversion engine only. No de-matrix YUV to RGB. So it should not need matrix operation param ?

So user must provide correct matrix param to ConvertToPlanarRGB() instead of default left to have correct conversion.
Correct.

@poisondeathray's script is using ConvertToPlanarRGB() because his source is image (png - packed rgb).

@simple_simon's script doesn't need ConvertToPlanarRGB() and I have been thinking of:
Code:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()
StvG is offline   Reply With Quote
Old 20th April 2024, 15:00   #17  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,175
Quote:
Originally Posted by StvG View Post
The resolution doesn't define if the video is full or limited. If it's YUV (especially DVD or BD, or UHD) then it's usually (read 99% of the cases) limited.
Because current AVS interface do not support non-stop warnings - there is an idea to add bool param like 'treat_warnings_as_errors=(default)true' and throw stop-error if user provide any very highly looking like error combinations of params like YUV+full. And if user insist on the conversion - it is required manually force it by setting treat_warnings_as_errors=false.
DTL is offline   Reply With Quote
Old 20th April 2024, 18:41   #18  |  Link
simple_simon
Registered User
 
Join Date: Feb 2003
Posts: 124
Quote:
Originally Posted by StvG View Post
@poisondeathray's script is using ConvertToPlanarRGB() because his source is image (png - packed rgb).

@simple_simon's script doesn't need ConvertToPlanarRGB() and I have been thinking of:
Code:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()
So are you now saying that the "z_convertformat(pixel_type="RGBPS", use_props=0)" line is correct but just need to remove "ConvertToPlanarRGB()"?

I already noticed I got an error when using:
Code:
ConvertToPlanarRGB()
z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left")
so had to remove "ConvertToPlanarRGB()" already anyway.
simple_simon is offline   Reply With Quote
Old 20th April 2024, 20:09   #19  |  Link
StvG
Registered User
 
Join Date: Jul 2018
Posts: 487
Quote:
Originally Posted by simple_simon View Post
So are you now saying that the "z_convertformat(pixel_type="RGBPS", use_props=0)" line is correct but just need to remove "ConvertToPlanarRGB()"?
No.
If the source is 709:
Code:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f", chromaloc_op="left=>left")
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()
If the source is 601:
Code:
propSet("_FieldBased",0)
DeBicubicResizeMT(720,400, accuracy=2)
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_ort(network_path="1x_AnimeUndeint_Compact_130k_net_g.onnx", provider="cuda")
mlrt_ort(network_path="2x-AniScale.onnx", provider="cuda")
z_convertformat(pixel_type="YUV420P8", dither_type="error_diffusion").propSet("_ColorRange",1)
deep_resize(960,720,qual=2)
filmgrainplus()
You can check with mediainfo what's the source matrix or you can with avs:
Code:
sourcefilter
propshow()
StvG is offline   Reply With Quote
Old 20th April 2024, 21:30   #20  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,175
"If the source is 709:
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

If the source is 601:
z_convertformat(pixel_type="YUV420P8", dither_type="error_diffusion").propSet("_ColorRange",1)
"

Why we have such big difference in z_convertformat arguments only with input matrix difference (and also this difference eliminated with conversion to RGB before NN-processing) ?

Also documentation http://avisynth.nl/index.php/Interna...ons#ColorRange says
_ColorRange
int _ColorRange
Full or limited range (PC/TV range). Primarily used with YUV formats.
0=full range, 1=limited range.

So with arguments
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:full", dither_type="error_diffusion").propSet("_ColorRange",1)

the plugin simply ignores "=>709:709:709:full" and always outputs limited/narrow (for requested YUV420P8) so forcing of .propSet("_ColorRange",1) is valid ?

Documentation for ConvertBits() http://avisynth.nl/index.php/ConvertBits makes hints about (possible ?) valid YUV full-range for AVS+:
bool fulls = (auto)
Use the default value unless you know what you are doing.
Default value can come from _ChromaRange frame property
If true (RGB default), scale by multiplication: 0-255 → 0-65535;
Note: full scale U and V chroma is specially handled if false (YUV default), scale by bit-shifting.

Documentation on Avsresize http://avisynth.nl/index.php/Avsresize do not lists 'auto-ignoring of full range for YUV' and lists example of RGB to YUV420P8 as:
RGB to 8-bit YV12 (YUV 4:2:0 Rec. 709):
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:limited")

Also if user later make resize to 720p hd-like size it may be better to make RGB to rec.709 conversion to be more compatible with decode/display devices ignores metadata ? So backward conversion to YUV420P8 may be better
z_convertformat(pixel_type="YUV420P8", colorspace_op="rgb:601:601:full=>709:709:709:limited", dither_type="error_diffusion").propSet("_ColorRange",1) for rec 601 input matrix. If followed by resize to 960,720.

Last edited by DTL; 20th April 2024 at 22:12.
DTL is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 17:27.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.