Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th April 2019, 19:40   #21  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by _Al_ View Post
QTGMC can work with 422, so there can be just one color space conversion:
Code:
clip = core.ffms2.Source(source=INVID)
clip = core.resize.Bicubic(clip, format=vs.YUV422P8, matrix_in_s="170m", matrix_s="709")
clip = haf.QTGMC(clip, Preset='Medium', TFF=False)
clip = edi.nnedi3_rpow2(clip=clip, rfactor=2)
clip = core.resize.Spline36(clip, 1440, 1080)
clip.set_output()
Thanks for the suggestion Al - I'm curious about your QTGMC line, where you went with Medium. Is that intentionally not using "Slower"?
sunshine is offline   Reply With Quote
Old 18th April 2019, 19:42   #22  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,181
Quote:
Originally Posted by sunshine View Post
I did the following tests to compare PSNR of x264 and dnxhr vs. raw, and CRF 0 scored "inf" which may confirm it's lossless, if I understand correctly.. but again I'm a noob.
Yes it does confirm, but you'd see it had switched to encoding qp=0 anyway in the encode parameters - check with MediaInfo > Text view.
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 18th April 2019, 19:50   #23  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 81
Quote:
Do the processing steps prior to that final resize affect the color matrix? Is there a way for VapourSynth to tell me the color matrix before and after a processing step?
clip = core.text.FrameProps(clip)
that line would show you Matrix value, if 2, it means unknown, 1 is 709, 5 is 470bg (PAL DV), 6 is 170m (NTSC DV),

before encoding don't forget to comment out that line

or right after you load clip you could make sure and and set props correctly so Vapoursynth knows what the input is:
clip = core.std.SetFrameProp(clip, prop="_Matrix", intval=6)
clip = core.resize.Bicubic(clip, format=vs.YUV422P8, matrix_s="709")

btw, QTGMC takes YUV422P10 or YUV444P10 also

Quote:
I'm curious about your QTGMC line, where you went with Medium. Is that intentionally not using "Slower"?
no , it is just faster

Last edited by _Al_; 18th April 2019 at 20:15.
_Al_ is offline   Reply With Quote
Old 18th April 2019, 19:52   #24  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by WorBry View Post
Yes it does confirm, but you'd see it had switched to encoding qp=0 anyway in the encode parameters - check with MediaInfo > Text view.
I'm not sure about Text View as I don't see that in mediainfo output on Linux, but here's what I see in Encoding settings:

Code:
Encoding settings                        : cabac=0 / ref=1 / deblock=0:0:0 / analyse=0x1:0x131 / me=umh / subme=9 / psy=0 / mixed_ref=0 / me_range=24 / 
chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=0 / threads=18 / lookahead_threads=1 / sliced_threads=0 / 
nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / 
rc=cqp / mbtree=0 / qp=0
sunshine is offline   Reply With Quote
Old 18th April 2019, 20:04   #25  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 207
Quote:
Originally Posted by sunshine View Post
Do the processing steps prior to that final resize affect the color matrix? Is there a way for VapourSynth to tell me the color matrix before and after a processing step?
Resize has nothing to do with it.

The resize command:
Code:
clip = core.resize.Bicubic(clip, format=vs.YUV422P8, matrix_in_s="170m", matrix_s="709")
does not do a resize, it is simply used to change the colorspace.
Cary Knoop is offline   Reply With Quote
Old 18th April 2019, 20:08   #26  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,181
Quote:
Originally Posted by sunshine View Post
I'm not sure about Text View as I don't see that in mediainfo output on Linux, but here's what I see in Encoding settings:

Code:
Encoding settings                        : cabac=0 / ref=1 / deblock=0:0:0 / analyse=0x1:0x131 / me=umh / subme=9 / psy=0 / mixed_ref=0 / me_range=24 / 
chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=0 / threads=18 / lookahead_threads=1 / sliced_threads=0 / 
nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / 
rc=cqp / mbtree=0 / qp=0

There you go, qp=0.

Yes, I think with the linux version of MediaInfo you have to look in 'Show All Formats' or something. I'm not on linux just now to check.
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 18th April 2019, 20:24   #27  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,181
Quote:
Originally Posted by poisondeathray View Post
Quote:
Originally Posted by WorBry View Post
With 8bit sources, libx264 switches -crf 0 to -qp 0 automatically, but for 10bit you need to set -qp 0.
Yes, and I'm not sure why they changed the behaviour (inconsistent)
Actually, to qualify that - with 8bit sources libx264 swiches -crf 0 to -qp 0 automatically when encoding to 8bit (by default or set -pix_fmt). But for 10bit sources and 10bit x264 encoding, you need to set -qp 0.
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Old 18th April 2019, 22:57   #28  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,155
Quote:
Originally Posted by WorBry View Post
Does this actually work with Resolve ? Resolve has native support for import and export of a very limited range of VFW codecs (Cineform, Grass Valley, Uncompressed) and tends to be very picky about the AVI formatting - won't import uncompressed 4:2:0 in any form and you need the licensed (paid) Studio version to import lossless x264 (Hi444Pred). No ffmpeg support either, although that has oft been requested.
Just output the video to rgb30, it works
lansing is offline   Reply With Quote
Old 19th April 2019, 00:26   #29  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by poisondeathray View Post
You can look at the clip props, but sometimes videos are flagged incorrectly and you will get the wrong results

In general SD is 601, HD is 709

Roughly , yes. I you might get 1-3 % space savings for 2-3x slower processing . You can do some mini tests, but it's really not worth it .


I mean bob deinterlacing (ie. double rate) with QTGMC . Not using "bob" algorithm to deinterlace
I'm taking your recommended approach (deinterlace the SD DV, then do my edits, color correction, etc. in resolve, then resize to 1080 height) but I've encountered a small problem.

The aspect ratio is incorrect, and the video looks off.

AVI input file to VapourSynth:
Code:
Width                                    : 720 pixels
Height                                   : 480 pixels
Pixel aspect ratio                       : 0.889
MOV output from ffmpeg:
Code:
Width                                    : 720 pixels
Height                                   : 480 pixels
Pixel aspect ratio                       : 1.000

To correct this, I've updated my ffmpeg command with "setsar=sar=8/9,setdar=dar=4/3" as follows.. Did I do this correctly?!

Code:
vspipe --y4m ./Deinterlace.py -a "INVID=$INVID" - | ffmpeg -nostdin -i pipe: -i "$INVID" -pix_fmt yuv422p -c:v libx264  -crf 0 -tune fastdecode -x264opts keyint=1 -c:a copy -map 0:0 -map 1:1 -vf setsar=sar=8/9,setdar=dar=4/3 "$OUTVID"
After this adjustment, the MOV output from ffmpeg:
Code:
Width                                    : 720 pixels
Height                                   : 480 pixels
Pixel aspect ratio                       : 0.889
Thanks!
sunshine is offline   Reply With Quote
Old 19th April 2019, 01:51   #30  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,976
Quote:
Originally Posted by sunshine View Post

The aspect ratio is incorrect, and the video looks off.

AVI input file to VapourSynth:
Code:
Width                                    : 720 pixels
Height                                   : 480 pixels
Pixel aspect ratio                       : 0.889
MOV output from ffmpeg:
Code:
Width                                    : 720 pixels
Height                                   : 480 pixels
Pixel aspect ratio                       : 1.000

To correct this, I've updated my ffmpeg command with "setsar=sar=8/9,setdar=dar=4/3" as follows.. Did I do this correctly?!

Code:
vspipe --y4m ./Deinterlace.py -a "INVID=$INVID" - | ffmpeg -nostdin -i pipe: -i "$INVID" -pix_fmt yuv422p -c:v libx264  -crf 0 -tune fastdecode -x264opts keyint=1 -c:a copy -map 0:0 -map 1:1 -vf setsar=sar=8/9,setdar=dar=4/3 "$OUTVID"
After this adjustment, the MOV output from ffmpeg:
Code:
Width                                    : 720 pixels
Height                                   : 480 pixels
Pixel aspect ratio                       : 0.889
Thanks!




Without getting into a lengthy AR interpretation discussion, that's one way of getting 4:3 DAR . Typically I only set the SAR, because setting the DAR will only give you non ITU aspect ratio options

But for analog sources like VHS, technically the SAR is actually 10:11 . DV too. And those are what you are using according to the 1st post. The central portion or active image area is 702 width (rounded to 704), and pillar boxed

You can verify with some objects, such as a round clock or car tire shot straight on. 99% of the time 10:11 will give you the perfect circle, but 8:9 will give you slightly oval. It also depends on the lineage and what steps or types of transfers were done inbetween. It also depends on the software, you might have to interpret the AR in some software, but resolve can distinguish between the two (at least with the SAR flag)

It's the same with upscaling to HD . The AR will be slightly off if you use the wrong AR interpretation.

(But many people don't care about tiny AR error . I guess it depends on how picky you are, or if driving on "oval" tires is ok )
poisondeathray is offline   Reply With Quote
Old 19th April 2019, 04:43   #31  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by poisondeathray View Post
Without getting into a lengthy AR interpretation discussion, that's one way of getting 4:3 DAR . Typically I only set the SAR, because setting the DAR will only give you non ITU aspect ratio options
Is there a better way you'd recommend? I want to do this as "properly" as possible.


Quote:
Originally Posted by poisondeathray View Post
But for analog sources like VHS, technically the SAR is actually 10:11 . DV too. And those are what you are using according to the 1st post. The central portion or active image area is 702 width (rounded to 704), and pillar boxed

According to "ffprobe -i", my AVIs are:

Code:
Stream #0:0: Video: dvvideo (dvsd / 0x64737664), yuv411p, 720x480 [SAR 8:9 DAR 4:3], 28772 kb/s, 29.97 fps, 29.97 tbr, 29.97 tbn, 29.97 tbc
Is it possible this is incorrect? I used this to inform my choice to "setsar=sar=8/9,setdar=dar=4/3", maybe I'm misunderstanding but I assumed that would suggest my deinterlaced output should also target that.

Quote:
Originally Posted by poisondeathray View Post
You can verify with some objects, such as a round clock or car tire shot straight on. 99% of the time 10:11 will give you the perfect circle, but 8:9 will give you slightly oval. It also depends on the lineage and what steps or types of transfers were done inbetween. It also depends on the software, you might have to interpret the AR in some software, but resolve can distinguish between the two (at least with the SAR flag)

It's the same with upscaling to HD . The AR will be slightly off if you use the wrong AR interpretation.

(But many people don't care about tiny AR error . I guess it depends on how picky you are, or if driving on "oval" tires is ok )
I'm VERY picky, as I'm trying to produce "archival" grade footage of my family's old home movies. I'm OK spending the time and effort to get it right!

When it comes time to process in Resolve, I'll likely scale up 2x, crop those pillarboxes and the fuzzy bit at the bottom of the image out (not sure what it's called, but it's that distorted portion at the bottom of old video recordings).

Do you have a suggestion for dimensions to output from Resolve? Would you avoid cropping? I want to do this properly, but am unsure what the "right way" would be.

Last edited by sunshine; 19th April 2019 at 04:56.
sunshine is offline   Reply With Quote
Old 19th April 2019, 05:08   #32  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,976
Quote:
Originally Posted by sunshine View Post
Is there a better way you'd recommend? I want to do this as "properly" as possible.

According to "ffprobe -i", my AVIs are:

Code:
Stream #0:0: Video: dvvideo (dvsd / 0x64737664), yuv411p, 720x480 [SAR 8:9 DAR 4:3], 28772 kb/s, 29.97 fps, 29.97 tbr, 29.97 tbn, 29.97 tbc
Is it possible this is incorrect? I used this to inform my choice to "setsar=sar=8/9,setdar=dar=4/3", maybe I'm misunderstanding but I assumed that would suggest my deinterlaced output should also target that.


ffprobe/ffmpeg , like mediainfo cannot tell you things such as what the actual content AR is. It will only tell you what the flags or how it was encoded. The content SAR is probably 10:11 . If you use 8:9 likely you will get ovals instead of circles for things like DV, VHS . 99.9% of those types of sources will NOT use 8:9. They will use the ITU aspect ratio rules

So I am suggesting -vf setsar=sar=10/11 , but you have to check the actual footage with known objects or references because all sorts of things might distort the AR (studio screw up before broadcast, bad cap etc...) But a native DV camcorder will use ITU rules 99.9% of the time



Quote:

When it comes time to process in Resolve, I'll likely scale up 2x, crop those pillarboxes and the fuzzy bit at the bottom of the image out (not sure what it's called, but it's that distorted portion at the bottom

Do you have a suggestion for dimensions to output from Resolve? Would you avoid cropping? I want to do this properly, but am unsure what the "right way" would be.
You're probably referring to the VHS head switching noise. It's personal preference, but some people would cover them up with black overlays / black borders. Others prefer to crop . But that can distort the AR slightly if the areas are not proportional and it's not done properly

Personally, I wouldn't use resolve to do this part . It's too limited of an editor (it's getting better... but... can't handle simple things like interlace properly). I would actually keep it as DV interlaced and edit it that way in a proper editor

Resolve is fantastic for some things, not so much for others. Pros/cons
poisondeathray is offline   Reply With Quote
Old 19th April 2019, 06:18   #33  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by poisondeathray View Post
So I am suggesting -vf setsar=sar=10/11 , but you have to check the actual footage with known objects or references because all sorts of things might distort the AR (studio screw up before broadcast, bad cap etc...) But a native DV camcorder will use ITU rules 99.9% of the time
Great suggestion - I'll find myself a perfect circle, make a video of it and examine the two options. Back sometime next week with the results!
sunshine is offline   Reply With Quote
Old 19th April 2019, 06:22   #34  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 207
Quote:
Originally Posted by sunshine View Post
I'm VERY picky, as I'm trying to produce "archival" grade footage of my family's old home movies. I'm OK spending the time and effort to get it right!
Archival grade means you do not alter the video at all but capture the original as best as possible. Once you deinterlace and scale (unless it is lossless) you 'cook' the video.
Cary Knoop is offline   Reply With Quote
Old 19th April 2019, 07:14   #35  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,976
Quote:
Originally Posted by sunshine View Post
Great suggestion - I'll find myself a perfect circle, make a video of it and examine the two options. Back sometime next week with the results!
Maybe for the VHS footage , because of generational and possible transfer issues;

But the native DV camera, don't waste your time. Or do it for fun, because 99.9% is really underestimating and conservative. It's really 100% for 4:3 NTSC DV.

https://en.wikipedia.org/wiki/Pixel_aspect_ratio

PAR (pixel aspect ratio) is known as "SAR" (sample aspect ratio) in MPEG4 terminology. They mean the same thing, they just decided to change names. You can think of it as the w:h of the pixels . Eitherway 10:11 is correct . It's been confirmed many, many times over the years, and it's official in all the documentation and ISO standards

Here is another recent example from another forum from a DV camera
https://forum.videohelp.com/threads/...e5#post2547654

4/3 = 704/480 * 10/11



And definitely, "archive" is the original DV footage . Keep that safe. Something better might come along (definitely for scaling; but nothing on the horizon in the near future for deinterlacing compared to QTGMC, but you never know, some parts of it can be improved such as the NNEDI3 routines)
poisondeathray is offline   Reply With Quote
Old 19th April 2019, 18:44   #36  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by Cary Knoop View Post
Archival grade means you do not alter the video at all but capture the original as best as possible. Once you deinterlace and scale (unless it is lossless) you 'cook' the video.
A fair point, and I'm addressing that by:

1. Archiving the original MiniDV and Hi8 cassettes, though they're becoming effectively inaccessible due to the obsolescence of the hardware needed to use them.

2. Archiving the AVI transfers from the MiniDV camera. I'll keep these unedited as-is. They're not very "watchable" though, as they're interlaced and most playback devices my family/ friends will be using will not automatically deinterlace, and when they do, they do so relatively poorly compared to QTGMC for example.

3. From the AVIs, I'm producing deinterlaced, edited, color corrected, and likely scaled version for consumption on modern devices.

If someone wants to view the original AVIs, they'll remain accessible for years to come. If they want to view the cassettes, it'll be a challenge but they'll be here. For day to day viewing, I feel the deinterlaced, edited, color corrected versions will be the most enjoyable.

I think I'm covering all my bases with this approach, but I could be wrong. Thoughts?
sunshine is offline   Reply With Quote
Old 19th April 2019, 18:49   #37  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by poisondeathray View Post
Maybe for the VHS footage , because of generational and possible transfer issues;

But the native DV camera, don't waste your time. Or do it for fun, because 99.9% is really underestimating and conservative. It's really 100% for 4:3 NTSC DV.

https://en.wikipedia.org/wiki/Pixel_aspect_ratio

PAR (pixel aspect ratio) is known as "SAR" (sample aspect ratio) in MPEG4 terminology. They mean the same thing, they just decided to change names. You can think of it as the w:h of the pixels . Eitherway 10:11 is correct . It's been confirmed many, many times over the years, and it's official in all the documentation and ISO standards

Here is another recent example from another forum from a DV camera
https://forum.videohelp.com/threads/...e5#post2547654

4/3 = 704/480 * 10/11
I don't doubt your expertise and knowledge, but I definitely doubt mine. The video I'm working with is coming from so many sources that I may have miscommunicated, misunderstood, or just plain goofed something in the process. For example - much of it is from the 80's, shot on analog Hi8. Much is Digital8. Some is VHS, shot on god knows what. All of it was passed through a "modern" Sony Digital8, including the VHS (from a TBC VHS to the the SVHS input of the camera, out its firewire to WinDV).

So it's a mixed bag.. and ideally I'd like to at least eliminate any damage I might do to it by goofing the AR. So a quick test (shot using analog Hi8, and some shot on Digital8, both sent to the PC via the same path described previously) may help me to identify any issues I'm introducing through this approach.

Quote:
And definitely, "archive" is the original DV footage . Keep that safe. Something better might come along (definitely for scaling; but nothing on the horizon in the near future for deinterlacing compared to QTGMC, but you never know, some parts of it can be improved such as the NNEDI3 routines)
Absolutely. See my other reply, I have a plan to store physical and digital media at every stage for the reasons you describe.
sunshine is offline   Reply With Quote
Old 19th April 2019, 19:21   #38  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 207
Quote:
Originally Posted by sunshine View Post
A fair point, and I'm addressing that by:

1. Archiving the original MiniDV and Hi8 cassettes, though they're becoming effectively inaccessible due to the obsolescence of the hardware needed to use them.

2. Archiving the AVI transfers from the MiniDV camera. I'll keep these unedited as-is. They're not very "watchable" though, as they're interlaced and most playback devices my family/ friends will be using will not automatically deinterlace, and when they do, they do so relatively poorly compared to QTGMC for example.

3. From the AVIs, I'm producing deinterlaced, edited, color corrected, and likely scaled version for consumption on modern devices.

If someone wants to view the original AVIs, they'll remain accessible for years to come. If they want to view the cassettes, it'll be a challenge but they'll be here. For day to day viewing, I feel the deinterlaced, edited, color corrected versions will be the most enjoyable.

I think I'm covering all my bases with this approach, but I could be wrong. Thoughts?
I would stay away from AVI's as a container and use something more flexible.
Cary Knoop is offline   Reply With Quote
Old 19th April 2019, 22:37   #39  |  Link
sunshine
Registered User
 
Join Date: Apr 2019
Posts: 17
Quote:
Originally Posted by Cary Knoop View Post
I would stay away from AVI's as a container and use something more flexible.
For archival, Iím keeping the AVIs produced during WinDV transfer. For intermediary output Iím using x264 lossless in MOV, and for final output from Resolve Iím looking for a good lossless option for archival, and will use x264 in mp4 or mov for local use and pushing to YouTube.
sunshine is offline   Reply With Quote
Old 19th April 2019, 23:12   #40  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,181
Quote:
Originally Posted by sunshine View Post
and for final output from Resolve I’m looking for a good lossless option for archival.
I see you brought that up on the BMD forum:

https://forum.blackmagicdesign.com/v...s+TIFF#p494354
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 19th April 2019 at 23:29.
WorBry is offline   Reply With Quote
Reply

Tags
deinterlace, qtgmc, resize, resolve, vapoursynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:17.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.