Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 15th July 2015, 01:18   #1  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
Problem with a hand crafted script

Just some hours ago i updated my denoise function to allow the recalculate parameter.

The script seems to works just fine, but if i use it with the vectors creation of my script the entire script crash.

The script used to reproduce the issues is this one
Code:
import vapoursynth as vs
import MFunc as mon

core = vs.get_core()

file = "c:/Script/address.vpy"

src = core.std.BlankClip(width=32, height=32, format=vs.YUV420P8, length=10, fpsnum=25)

den = mon.Denoise2(src, 200, prefix = file, recalculate=8, fast=True)

den.set_output()
For creating the vector file, simply use a cmd like that
Code:
vspipe "c:/Script/address.vpy" "c:/Script/address.vpy.vec" -p
I'm quite sure is not a bug neither of the recalculate part nor in the vector creation part because
1)without 'create' the script works fine
2)an encoded version of a real film with and without vectors, but without recalculate, gave me the SAME file [the same MD5]

Debugging a bit the script [my way to debug a python script is putting line like a = x/0 where x is the instance of the breakpoint all over the code in a bisect way] it seems to crash when reaching the final iteration inside the
Code:
while blksize > recalculate:
loop, before the first call to Recalculate

This would suggest some problem when reaching the blksize=8 iteration, but lowering the recalculate parameter don't solve the problem

You can find MFunc here https://github.com/MonoS/MonoS-VS-Fu...aster/MFunc.py

Thanks for the attention
MonoS is offline   Reply With Quote
Old 15th July 2015, 06:43   #2  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
It sounds like a bug in Recalculate. What are overlap and blksize when it crashes?
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 15th July 2015, 11:02   #3  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
It crash with the pairs: 8,2;8,4;16,4;16,8

Changing the code from a loop to
Code:
        if blksize > recalculate:
		blksize = int(blksize / 2)
		if fast:
			overlap = int(overlap / 4)
		else:
			overlap = int(overlap / 2)
		
		bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
	
	if blksize > recalculate:
		blksize = int(blksize / 2)
		if fast:
			overlap = int(overlap / 4)
		else:
			overlap = int(overlap / 2)
		
		bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
	
	if blksize > recalculate:
		blksize = int(blksize / 2)
		if fast:
			overlap = int(overlap / 4)
		else:
			overlap = int(overlap / 2)
			
		bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
The code seems to crash right before the first recalculate in the second if statement with blksize and overlap of 16,8 or 18,4

Last edited by MonoS; 15th July 2015 at 11:10.
MonoS is offline   Reply With Quote
Old 15th July 2015, 11:48   #4  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
Does it crash if you use Analyse to create the initial vector clip?

Also, remove all lines that aren't necessary to reproduce the crash. (This goes for any bug you encounter, in any software.)
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 15th July 2015, 12:13   #5  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Recalculate works without any problem here, I used (blksize/overlap) 16/8 8/4 4/2 in Placebo.py and everything is just nice
feisty2 is offline   Reply With Quote
Old 15th July 2015, 12:14   #6  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
Following your suggestion i reworked mon.Denoise2 to the bare minimum
Code:
import os.path

def Denoise2(src, denoise=400, blur=None, lsb=True, truemotion=True, chroma=True, fast=False, blksize=None, prefix=None, recalculate=None, thSAD=None):
	core = vs.get_core()
	
	src16 = mon.Up16(src, lsb)
	
	if fast:
		if blksize is None:
			blksize = 32
		
		overlap = int(blksize/4)
	else:
		if blksize is None:
			blksize = 8
		
		overlap = int(blksize/2)
	
	if recalculate is None:
		recalculate = blksize
	
	if thSAD is None:
		thSAD = int(denoise * 1.25)
	
	super = core.mv.Super(src16, chroma=chroma)
	
	superRep = super
	
	bvec1 = mon.ReadVecs(0, prefix, 4)
	bvec2 = mon.ReadVecs(1, prefix, 4)
	fvec1 = mon.ReadVecs(2, prefix, 4)
	fvec2 = mon.ReadVecs(3, prefix, 4)
		
	while blksize > recalculate:
		blksize = int(blksize / 2)
		if fast:
			overlap = int(overlap / 4)
		else:
			overlap = int(overlap / 2)
			
		bvec1 = core.mv.Recalculate(superRep, bvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		bvec2 = core.mv.Recalculate(superRep, bvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec1 = core.mv.Recalculate(superRep, fvec1, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
		fvec2 = core.mv.Recalculate(superRep, fvec2, thSAD, blksize=blksize, chroma=chroma, truemotion=truemotion, overlap=overlap)
	
	fin = core.mv.Degrain2(src16, super, bvec1,fvec1,bvec2,fvec2, denoise, plane = 4 if chroma else 0)
	
	return fin
The crash still occurs

I didn't understand what you mean by "Does it crash if you use Analyse to create the initial vector clip?"
If i understood correctly, without using the vector file no it don't crash, it only crash in conjunction between vector file and recalculate

Also i first opened the script into VirtualDub, because usually it's crash reports are "very useful" and the crash originated from VD not from any of the script
MonoS is offline   Reply With Quote
Old 15th July 2015, 12:23   #7  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
Quote:
Originally Posted by MonoS View Post
If i understood correctly, without using the vector file no it don't crash, it only crash in conjunction between vector file and recalculate
Then maybe don't do that. (Sure, Recalculate should fail more gracefully when you give it a vector clip it doesn't like.)
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 15th July 2015, 12:30   #8  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
Quote:
Originally Posted by jackoneill View Post
Then maybe don't do that. (Sure, Recalculate should fail more gracefully when you give it a vector clip it doesn't like.)
the problem is exactly that, the vector file is too much useful [saved me days of time encoding]

Also, why recalculate should fail if the vector file is an exact copy of the vector produced by analysis [as i said doing an encode in x264 with and without vector file gave me the same file] and why VD don't report the crash in recalculate but in the vapoursynth dll??

It's not much of a problem not using it but i'd like to understand what's happening
MonoS is offline   Reply With Quote
Old 15th July 2015, 12:40   #9  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
solution: write the "recalculated vectors" as an external file, that's what I been doing
feisty2 is offline   Reply With Quote
Old 15th July 2015, 12:46   #10  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by MonoS View Post
an exact copy of the vector produced by analysis
not the "exact" copy, it comes with additional resolution tags, the exact copy should have "unknown dimensions", maybe remove those 2 tags and try again
feisty2 is offline   Reply With Quote
Old 15th July 2015, 12:49   #11  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
Quote:
Originally Posted by feisty2 View Post
solution: write the "recalculated vectors" as an external file, that's what I been doing
This is not feasible because the dimension of the vector file will become too big [talking about >100GB file] and i can't spare that much space

Quote:
Originally Posted by feisty2 View Post
not the "exact" copy, it comes with additional resolution tags, the exact copy should have "unknown dimensions", maybe remove those 2 tags and try again
YEAH, you're right O_O, i'll try that
MonoS is offline   Reply With Quote
Old 15th July 2015, 17:23   #12  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
@feisty: Yes, you were right, making the vector as fixed dimensions recreated the problem
Now i should find a way to remove the dimensions tags
MonoS is offline   Reply With Quote
Old 16th July 2015, 03:23   #13  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
now I get what's all that "varying dimensions" stuff on vector frames for,
it's for "recalculate"
the width of vector frames grows as you recalculate them, cuz smaller blocks = more blocks = more vectors, obviously
you don't have to care about the growing width at all if the width is already "variable"

as for dimension tag removal, guess Myrsloik wrote something about stripping frame property on the ModifyFrame page
feisty2 is offline   Reply With Quote
Old 16th July 2015, 12:56   #14  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
Ah, hmm. VapourSynth filters aren't allowed to return frames with dimensions that don't match those declared in the video info. (Recalculate should reject vector clips that have known dimensions.)

I don't think you can create clips with unknown dimensions using the Python module. It's not a matter of removing frame properties. The information is stored in the VSVideoInfo struct associated with the video node.
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 16th July 2015, 15:33   #15  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by jackoneill View Post
Ah, hmm. VapourSynth filters aren't allowed to return frames with dimensions that don't match those declared in the video info. (Recalculate should reject vector clips that have known dimensions.)

I don't think you can create clips with unknown dimensions using the Python module. It's not a matter of removing frame properties. The information is stored in the VSVideoInfo struct associated with the video node.
If you want an unknown dimension clip you can do something like this:

Splice([BlankClip(width=200), BlankClip(width=100)], mismatch=True)

Not the most obvious way but there's rarely a good reason to do this...
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 16th July 2015, 16:55   #16  |  Link
MonoS
Registered User
 
Join Date: Aug 2012
Posts: 203
Quote:
Originally Posted by Myrsloik View Post
If you want an unknown dimension clip you can do something like this:

Splice([BlankClip(width=200), BlankClip(width=100)], mismatch=True)

Not the most obvious way but there's rarely a good reason to do this...
Thanks Myrsloik, this seems to solve the problem

Code:
          if blksize > recalculate:
		bvec1 = core.std.Splice([core.std.BlankClip(bvec1, width=1, length=1), bvec1], mismatch=True).std.Trim(1)
		bvec2 = core.std.Splice([core.std.BlankClip(bvec2, width=1, length=1), bvec2], mismatch=True).std.Trim(1)
		fvec1 = core.std.Splice([core.std.BlankClip(fvec1, width=1, length=1), fvec1], mismatch=True).std.Trim(1)
		fvec2 = core.std.Splice([core.std.BlankClip(fvec2, width=1, length=1), fvec2], mismatch=True).std.Trim(1)
Now i'll try to make some real encode and see if i get the same result with and without the vectors file

Last edited by MonoS; 16th July 2015 at 17:04.
MonoS is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:13.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.