Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
15th September 2018, 16:50 | #21 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Franc62,
This should work perfectly for entire clip (mehopes). Use original ClipClop script for final render. I've used correlation, rather than frame diff. Need prev posted lib funcs. Code:
### CONFIG ### AVISource("D:\UVSwap.mp4.AVI") ClipClopName= "D:\ClipClopCmd.txt" DB="D:\UVSwap.DB" DEBUG = False # False only write Detections, not metrics CLIPCLOP_SHOW = True STACK = True CONT_U = 0 # ColorYUV U Sat adjust for bad range (0=no adjust) CONT_V = 0 # ColorYUV V Sat adjust for bad range (0=no adjust) ###### END CONFIG ### ORG=Last Crop_L=10 CROP_T=36 CROP_R=-20 CROP_B=-4 MID = Crop(Crop_L,CROP_T,CROP_R,CROP_B) # Crop off all border crap UC = MID.UToY8 VC = MID.VToY8 SWAPUV_C = MID.SwapUV() SWAPUV_UC = SWAPUV_C.UToY8 SWAPUV_VC = SWAPUV_C.VToY8 CN_Chroma = MID.MergeChroma(MID.SelectEvery(1,1)) # Copy Next Chroma into current, Fix End Bad + 1 half bad frame CN_Chroma_UC = CN_Chroma.UToY8 CN_Chroma_VC = CN_Chroma.VToY8 OUT_V1 = (CONT_U!=0 || CONT_V!=0)?ORG.SwapUV.ColorYUV(cont_u=CONT_U,cont_v=CONT_V):ORG.SwapUV OUT_V2 = ORG.MergeChroma(ORG.SelectEvery(1,1)) ### ClipClopName = (ClipClopName!="") ? RT_GetFullPathName(ClipClopName) : "" (ClipClopName!="") ? RT_Filedelete(ClipClopName) : NOP RT_DBaseAlloc(DB,0,"ii") # Alloc two int fields for SOS & EOS Records = DBase_DetectScenes(MID,DB) # Use Defaults for fields, Offset, and thSCD1 and thSCD2 RT_DebugF("DEMO: Detected %d Scenes, added to DBase",Records) # See output in DebugView (Google) BadSeqs=0 HalfBad=0 Records=RT_DBaseRecords(DB) For(scene=0,Records-1) { SFrm = RT_DBaseGetField(DB,scene,0) # Get scene start from field 0 (SOS) EFrm = RT_DBaseGetField(DB,scene,1) # Get scene end from field 1 (EOS) Prev_Start=-1 IN=False Prev_In=False for(n=SFrm+1,EFrm) { # Scan Frames in Scene, (start at 1st frame +1) U = RT_LumaCorrelation(UC,UC,n=n-1,n2=n) # U prev to current [correlation, higher is better match, -ve is invert) V = RT_LumaCorrelation(VC,VC,n=n-1,n2=n) # V prev to current USW = RT_LumaCorrelation(UC,SWAPUV_UC,n=n-1,n2=n) # U prev to current U SWAP VSW = RT_LumaCorrelation(VC,SWAPUV_VC,n=n-1,n2=n) # V prev to current V SWAP (DEBUG) ? RT_DebugF("%d] U=%f V=%f USW=%f VSW=%f",n,U,V,USW,VSW) : NOP T1 = (USW > U) T2 = (VSW > V) Bingo = T1 && T2 In = (BINGO) ? !In : IN if(In) { # Start if(!Prev_In) { Prev_start=n (DEBUG) ? RT_DebugF("%d] START: = %d",n,Prev_Start) : NOP } } Else { if(Prev_In) { # Stop BadSeqs = BadSeqs + 1 RT_DebugF("%d] STOP: Writing ClipClop replacment with Index 1 clip, range %d,%d",n,Prev_Start,n-1) RT_WriteFile(ClipClopName,"1 %d,%d",Prev_Start,n-1,Append=True) # Write ClipClop Range to replace with clip index 1 SWP_CN_U = RT_LumaCorrelation(SWAPUV_UC, CN_Chroma_UC, n=n-1,n2=n) # U SWAP prev to current CN_Chroma_U SWP_CN_V = RT_LumaCorrelation(SWAPUV_VC, CN_Chroma_VC, n=n-1,n2=n) # V SWAP prev to current CN_Chroma_V SWP_U = RT_LumaCorrelation(SWAPUV_UC, UC, n=n-1,n2=n) # U SWAP prev to current U SWP_V = RT_LumaCorrelation(SWAPUV_VC, VC, n=n-1,n2=n) # V SWAP prev to current V (DEBUG) ? RT_DebugF("%d] SWP_CN_U=%f SWP_CN_V=%f SWP_U=%f SWP_V=%f",n,SWP_CN_U,SWP_CN_V,SWP_U,SWP_V) : NOP T1 = (SWP_CN_U > SWP_U) T2 = (SWP_CN_V > SWP_V) # Correlation to Next chroma better ? Bingo = T1 && T2 if(Bingo) { # Half Bad Frame, next Chroma Better match HalfBad = HalfBad + 1 RT_DebugF("%d] HALFBAD: Writing ClipClop replacment with Index 2 clip, Frame %d",n,n) RT_WriteFile(ClipClopName,"2 %d",n,Append=True) # replace End Bad + 1 frame with clip index 2 frame } Prev_Start=-1 } } Prev_In=In } } (DEBUG) ? RT_DebugF("BadSeqs=%d HalfBad=%d",BadSeqs,HalfBad) : NOP NickNames =""" # Psuedonyms for clips (clip index number) SwapUV = 1 # Fixed Bad range clip (SwapUV with optional Chroma Sat adjustment) CN_Chroma = 2 # Fixed half bad end bad + 1 frame (Copy Chroma from next) """ ClipClop(ORG,OUT_V1,OUT_V2,Cmd=ClipClopName,nickname=NickNames,show=CLIPCLOP_SHOW) (STACK)?StackVertical(Last,ORG):Last For any mishaps (dont expect any), then can edit ClipClopCmd.txt file. EDIT: There are no thresholds, either better match or not, where in initial comparison, 1 pair is expected to be -ve, and other +ve. EDIT: If 1st frame of scene is swapped, then aint no way we can detect, with likely error, not likely to be able to detect using any other method either. EDIT: Quote:
but should still work OK.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 16th September 2018 at 12:12. |
|
15th September 2018, 18:42 | #22 | Link |
Registered User
Join Date: Mar 2017
Location: Germany
Posts: 234
|
Thanks, @StainlessS!
Will I need any drivers or other database files to run this, or is the whole database functionality integrated in your RT library, so that the code generates the DB-file and simply works? I have no time at all to test and try out at the moment, that's a real pity. But I will, by the beginning of next week, that's for sure. |
15th September 2018, 20:21 | #23 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Just need RT_Stats, old v1.43 should be OK.
+ Library func DBase_DetectScenes() previousy posted in post #13 [DB_FindTrim() not necessary]. + MvTools2, Masktools2, Avs+. Nothing else I think, should 'simply work'. EDIT: Quote:
but should still work OK.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 15th September 2018 at 20:40. |
|
20th September 2018, 11:46 | #24 | Link |
Registered User
Join Date: Mar 2017
Location: Germany
Posts: 234
|
Hi @StainlessS
series done. There were a lot of other issues, and also a small error: The scene we took for testing could be fixed ACCIDENTALLY by swapping U and V. There were a lot of different bad colours, all blue, all green, interlaced mixed, etc. So the only chance was to fix everything by hand, taking colour from nearest correct frames, motion compensated as good as possible. Looks quite ok now, though not perfect. again for help. As soon as I will find the time I will try all this - will be useful in the future, I am sure. |
20th September 2018, 11:54 | #25 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Ah, sorry to hear that, Having now completed the odyssey, I'm sure that you feel quite relieved.
EDIT: I guess that the DBase scene scanning stuff should be re-usable for something, was an interesting excercise anyway.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
20th September 2018, 20:00 | #27 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Propose something like below to avoid massive string memory allocation where function may be called hundreds of thousands of times in a
single script (where script can be a lot bigger than below one). Code:
# ############ # DB_FindTrim.avs [Replaced with RT_DBaseFindSeq() plugin function in RT_Stats v2.0] # DB_FindTrim Returns clip (or record number) containing movie frame Frame. # Search DBase for the record (clip) that contains the arg Frame, using Binary Search. # S_Field is the DB field that contains the first frame of the record clip. # E_Field is the DB field that contains the Last frame of the record clip. # Return of -1 = NOT FOUND. # ############ Function DB_FindTrim(String DB,Int S_Field,Int E_Field,Int Frame) { IsAvsPlus=(FindStr(UCase(versionString),"AVISYNTH+")!=0) HasGScript=RT_FunctionExist("GScript") Assert(IsAvsPlus || HasGScript,RT_String("DB_FindTrim: Need either GScript or AVS+")) result = -1 # Init NOT FOUND low = 0 high = RT_DBaseRecords(DB) - 1 Global DB_FindTrim_GLOBAL_STRING=RT_VarExist("DB_FindTrim_GLOBAL_STRING")?DB_FindTrim_GLOBAL_STRING:""" while(low <= high) { mid = (low + high) / 2 if(RT_DBaseGetField(DB,mid,E_Field) < Frame) { low = mid + 1 } Else If (RT_DBaseGetField(DB,mid,S_Field) > Frame) { high = mid - 1 } Else { low = high + 1 # Force exit Result = mid } } """ HasGScript ? GScript(DB_FindTrim_GLOBAL_STRING) : Eval(DB_FindTrim_GLOBAL_STRING) # Use GSCript if installed (loaded plugs override builtin) return result } EDIT: There is no string memory garbage collection in avsynth, strings are allocated/reallocated and never released until avs closedown. EDIT: And should above work OK (or perhaps even makes things worse, maybe I have to check). EDIT: Nah, dont seem to make a damn bit of difference (except being slower). Code:
# DB_FindTrim (original) AVSMeter 2.8.1 (x86) - Copyright (c) 2012-2018, Groucho2004 AviSynth+ 0.1 (r2728, MT, i386) (0.1.0.0) Number of frames: 73730 Length (hh:mm:ss.ms): 00:41:00.124 Frame width: 490 Frame height: 360 Framerate: 29.970 (30000/1001) Colorspace: YV12 Audio channels: 2 Audio bits/sample: 16 Audio sample rate: 44100 Audio samples: 108491612 Frames processed: 73730 (0 - 73729) FPS (min | max | average): 211.1 | 420.5 | 290.6 Memory usage (phys | virt): 122 | 148 MiB Thread count: 15 CPU usage (average): 24% Time (elapsed): 00:04:13.727 ########################################## # DB_FindTrim_MOD1 D:\>avsmeter _fix.avs AVSMeter 2.8.1 (x86) - Copyright (c) 2012-2018, Groucho2004 AviSynth+ 0.1 (r2728, MT, i386) (0.1.0.0) Number of frames: 73730 Length (hh:mm:ss.ms): 00:41:00.124 Frame width: 490 Frame height: 360 Framerate: 29.970 (30000/1001) Colorspace: YV12 Audio channels: 2 Audio bits/sample: 16 Audio sample rate: 44100 Audio samples: 108491612 Frames processed: 73730 (0 - 73729) FPS (min | max | average): 36.98 | 397.1 | 277.6 Memory usage (phys | virt): 122 | 148 MiB Thread count: 15 CPU usage (average): 24% Time (elapsed): 00:04:25.597 ##########################################
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 20th September 2018 at 21:10. |
20th September 2018, 23:26 | #28 | Link |
Registered User
Join Date: Mar 2017
Location: Germany
Posts: 234
|
Can you explain this a little bit?
-What do you search? A frame of a clip inside dbase? How? Do you have blobs or binary fields for each frame? Do I get this right? -And why then do you perform a binary search by hand? Why not use database logic (which works binary searching) and fire some SQL-command? I admit, I don't get the point at all. |
21st September 2018, 03:52 | #29 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
The DataBase is specific to RT_Stats plugin and it dont understand SQL (tis a little basic).
However Quote:
how to make a script function that is likely called at every frame, to be compatible with both GScript and AVS+ version of GScript extensions (and without using massive amounts of string memory, DB_FindTrim uses only a small script string but some funcs could be quite big). RT_Stats DBase can hold up to 1024 fields, Bool, Int, Float, String, and BIN(8 bit unsigned int), in v2.0 Beta, up to just below 1TB in size. In the DBase_DetectScenes() demo thing, all we did was detect scene changes using MVTools2, and create a record for each scene, with start and end frame of each scene. The DB_FindTrim() thing, at each frame searches the DBase using binary search to determine which record holds the current frame, and then client can just retrieve start and end frame of each scene from the DBase record. EDIT: Code:
RT_Stats DBASE functions allow fast access to a file based DataBase where each record can have up to 1024 fields of variable type, Bool, Int, Float, String, or BIN. BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in the DBase field as an unsigned 8 bit Int, upper 24 bits ignored. there is also a type double used in DBase, this is only of use to internal RT_Stats functions, setting or getting type double takes or returns type Float (as Avisynth cannot handle double precision float). The maximum possible DBase file size is about 1TB, and the number of records is restricted to $7FFFFFFE, about 2 Billion. Field types Bool and BIN require 1 byte, Int and Float 4 bytes, and String is of fixed maximum length (user chosen) up to 256KB ((256 * 1024), no nul terminating character is stored in a DBase String). The private type Double requires 8 bytes. A DBase file allows up to 1024 Attributes of types Int or Float, which can be used for whatever a user desires, these attributes are stored in the DBase file header which is 32768 bytes, so the actual data starts at $8000 hex. In addition to user attributes, a DBase has currently 10 user String Attributes that can be be used for whatever you want, the string attributes are currently a maximum of 1024 characters in length. In addition to user attributes and user string attributes, there are currently 128 ID's (Int or Float) that are intended to be used by either RT_Stats itself, or script developers, intended function of ID's are to in some way eg link a DBase to a particular clip, by storing maybe Width, Height, Framecount and perhaps other values, the stored ID's might be used to Validate a DBase against a clip. You could use the DBase functions to return multiple results from a script function to the caller via a caller supplied DBase filename. The DBase functions also make it more likely that script functions could be written to eg find where edits between two clips have been made, or previously impossible/implausible tasks performed. Code:
RT_Stats ARRAY functions allow fast access to a file based Array of up to 3 dimensions of fixed element type, Bool, Int, Float, String, or BIN. BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in the Array as an unsigned 8 bit Int, upper 24 bits ignored. The maximum possible file size is about 1TB, and depends on number and size of dimensions and element type and size, also depends upon avilable disk space. Element types Bool and BIN require 1 byte, Int and Float 4 bytes, and String is of fixed maximum length (user chosen) up to 256KB (256 * 1024, no nul terminating character is stored in an Array String). Maximum possible size of dimension 1 is $7FFFFFFE but depends upon available disk space and also limited to max 1TB. RT_ArrayAlloc, allows you to create an array file with a fixed number of dimensions and where the 2nd and/or 3rd dimensions of a multi-dimensional array are fixed in size but the size of the first dimension can 'grow', single dimension arrays can 'grow' too. You can create an array where the first dimension is 0 (any other dimensions fixed), ie no elements pre-allocated in array file, in this case you can use RT_ArrayExtend to 'grow' the first dimension of the array so that it can be used. In addition, single dimension arrays can also use RT_ArrayAppend to add elements to the end of the array. An Array file allows up to 1024 Attributes of types Int or Float, which can be used for whatever a user desires, these attributes are stored in the Array file header which is 32768 bytes, so the actual data starts at $8000 hex. In addition to user attributes, an Array has currently 10 user String Attributes that can be be used for whatever you want, the string attributes are currently a maximum of 1024 characters in length. In addition to user attributes and user string attributes, there are currently 128 ID's (Int of Float) that are intended to be used by either RT_Stats itself, or script developers, intended function of ID's are to in some way eg link an array to a particular clip, by storing maybe Width, Height, Framecount and perhaps other values, the stored ID's might be used to Validate an array against a clip. You could use the Array functions to return multiple results from a script function to the caller via a caller supplied Array filename, perhaps with some kind of status variable returned directly.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 21st September 2018 at 04:01. |
|
Thread Tools | Search this Thread |
Display Modes | |
|
|