ChatGPT vs. Claude on ColorMatrix conversion - which answer is correct?

[orion44]

I've asked GPT-4o and Claude 3.5 Sonnet the same question, but they gave different answers.

Which answer do you think is the correct one?

Question: "When resizing 1080p video to SD resolution (640x360), should I apply ColorMatrix ('ColorMatrix(mode="Rec.709->Rec.601")')
before or after resizing the video, to convert from Rec.709 to Rec.601?"

GPT-4o:

When converting a 1080p video to SD resolution, you should apply the `ColorMatrix("Rec.709->Rec.601")` conversion **before resizing** the video. Here’s why:

1. **Color Primaries and Matrix**:
- HD content (like 1080p) is typically encoded with Rec.709 color space, while SD content (such as 640x360) is usually in Rec.601 color space. The color matrix conversion from Rec.709 to Rec.601 ensures that the colors are accurate for the SD standard.

2. **Avoiding Interpolation Artifacts**:
- Applying `ColorMatrix` before resizing preserves color accuracy by converting colors in their original, unscaled form. Resizing the video first and then applying the color conversion can lead to minor interpolation artifacts, as the color matrix adjustment may not apply as cleanly after scaling.

3. **Performance**:
- Applying `ColorMatrix` before resizing is generally faster because it processes fewer pixels in the original HD frame than it would after the image is scaled down.

### Correct Usage Example

```avs
# Convert Rec.709 to Rec.601 for SD resolution
ColorMatrix(mode="Rec.709->Rec.601")

# Resize to SD resolution
BicubicResize(640, 360)
```

By doing it in this order, you ensure that the color conversion is done on the full HD image, preserving color fidelity and achieving accurate color reproduction in your SD output.

Claude 3.5 Sonnet:

The ColorMatrix conversion from Rec.709 to Rec.601 should be applied AFTER resizing. Here's why:

1. The resize operation should be done in the original colorspace (Rec.709 for HD content) to maintain the best quality during the scaling process
2. Only after you've resized to SD resolution should you convert to Rec.601, since that's the standard colorspace for SD content

Correct order:
```
video = LSMASHSource("your_video.mp4")
video = Spline36Resize(video, 640, 360)
video = ColorMatrix(video, "Rec.709->Rec.601")
```

Incorrect order:
```
video = LSMASHSource("your_video.mp4")
video = ColorMatrix(video, "Rec.709->Rec.601")
video = Spline36Resize(video, 640, 360) # Don't do this
```

This order ensures:
1. The resize operation works with the full color information in the original colorspace
2. The colorspace conversion happens only once, at the final resolution
3. Better preservation of color accuracy during the entire process

[Z2697]

Applying `ColorMatrix` before resizing is generally faster because it processes fewer pixels in the original HD frame than it would after the image is scaled down.

Really??

[Z2697]

I'm convinced by the second point by ChatGPT

[FranceBB]

Please don't use AI for encoding related questions as they generally hallucinate a lot and especially in Avisynth they even tend to give non working outputs.
I'd like not to see them anywhere on Doom9 as it's literally one of the last few places that is preserving human knowledge without being polluted by artificially generated stuff
Ask directly on Doom9 after looking at other threads with similar questions.
Now, doing it before or after will not realistically have an impact on the overall quality and this is because of how Colormatrix() works, hence why both AI are hallucinating here.

FFMpegSource2("whatever.mxf", atrack=-1)

ColorMatrix(mode="Rec.709->Rec.601")

SinPowerResize(640, 360)

Now the AI is clearly thinking "oh, this way Colormatrix() is gonna work on the full resolution of the video so that it first upscales the chroma from 4:2:0 planar to RGB, then it applies the matrix conversion and then it scales back to 4:2:0, so we're actually performing the conversion at a higher quality 'cause we're preserving the chroma before we resize."

FFMpegSource2("whatever.mxf", atrack=-1)

SinPowerResize(640, 360)

ColorMatrix(mode="Rec.709->Rec.601")

Here the AI will probably think "see, if we perform the colormatrix conversion after the downscale, then the chroma, which in the 1920x1080 resolution was 960x540, has been further downscaled to 320x180 'cause we're starting from 640x360 in 4:2:0, so when we upscale the chroma to match the luma resolution in RGB and apply the colormatrix conversion we're working with fewer pixels and therefore the result is worse".

Now, all of this would make sense if we were using avsresize, in other words:

z_ConvertFormat(colorspace_op="709:709:709=>470bg:601:470bg")

This is because avsresize is based on zlib and such a library will ALWAYS internally convert to RGB when it performs the matrix conversion (in this case BT709 to BT601 PAL), so by doing it in 1920x1080 4:2:0 it would be upscaling the chroma from 960x540 to 1920x1080 to go to RGB, then perform the conversion, then downscale it back to 960x540 to get a nice 1920x1080 4:2:0 and then you would call SinPowerResize to finally go to 640x360. That's how it's supposed to be done to preserve quality and achieve the best conversion, everybody is happy, job done.

BUT... we're using Colormatrix() here and Colormatrix() DOES NOT internally convert to RGB as all the operations are done in YUV.
So... does it make a difference to perform the conversion before downscaling with Colormatrix()? Maybe.
Is it noticeable at all? No.
Can you do it once you've downscaled to save CPU cycles? Absolutely.

[lollo2]

Originally Posted by FranceBB

Please don't use AI for encoding related questions as they generally hallucinate a lot and especially in Avisynth they even tend to give non working outputs.
I'd like not to see them anywhere on Doom9 as it's literally one of the last few places that is preserving human knowledge without being polluted by artificially generated stuff

+ 1

Well said!

[kolak]

ChatGPT doesn't know things or create them.
It repeats what have been already said when it comes to "knowledge".
Its answers will be based also on doom9

[orion44]

Originally Posted by kolak

ChatGPT doesn't know things or create them.
It repeats what have been already said when it comes to "knowledge".
Its answers will be based also on doom9

I've noticed this also.

Newer versions of ChatGPT seem to have the entirety of doom9 forum in its database.

[VoodooFX]

Originally Posted by kolak

ChatGPT doesn't know things or create them.
It repeats what have been already said when it comes to "knowledge".

That's not quite true. Same you could say about humans.

[kolak]

No. Human can do own tests and come to new conclusions. ChatGPT knowledge is purely based on internet (what already been said/tested/proven etc.).
It's more advanced only in language related tasks.

[GeoffreyA]

Originally Posted by kolak

No. Human can do own tests and come to new conclusions. ChatGPT knowledge is purely based on internet (what already been said/tested/proven etc.).
It's more advanced only in language related tasks.

I think these are early versions of part of the human mind, perhaps the language faculty, lacking the sense modalities, persistent state, perpetual training in real-time, and using a lot of power.

Though by Microsoft researchers, this paper had some interesting findings:

https://arxiv.org/abs/2303.12712

[Z2697]

Originally Posted by GeoffreyA

I think these are early versions of part of the human mind, perhaps the language faculty, lacking the sense modalities, persistent state, perpetual training in real-time, and using a lot of power.

Though by Microsoft researchers, this paper had some interesting findings:

https://arxiv.org/abs/2303.12712

Oh yeah, research by the big money behind the AI model being studied, I totally belive it.

[GeoffreyA]

Microsoft is a joke, no doubt about that. In general, though, I don't see why neural networks can't match the human brain. Right now, they're missing a lot of parts, and power is a barrier. Perhaps the turning point will be biological circuitry. As for the hard problem of consciousness, we haven't even explained how it works in the brain, but it is tied to certain parts (the anterior cingulate cortex, I believe), so conceivably, might be some trick of recursive circuitry and a working temporal window. Having said all that, I'm actually one who's job is being threatened by AI, and am not cheering the laughable bandwagon at all, where every company is hungry to plunge their hand into that pot of dollars. My interest is more from a science-fiction, philosophical point of view

[VoodooFX]

Originally Posted by kolak

No. Human can do own tests and come to new conclusions.

Take out all preexisting knowledge and your most advanced tests will be counting the fingers on a left hand.

[kolak]

I didn't say you want to abandon existing knowledge, but to come up with something new based on it.
Ask ChatGPT to prove his method is better than others.
All what it does when it comes to "avisynth" is search and repeat what others said.
It's just a fancy google search which better understands your questions and can give nicer answers (due to its ability to understand language very well).

[VoodooFX]

Originally Posted by kolak

...repeat what others said.

No, it can synthesize of what it knows, so it can come up with answers never said in previous (training) knowledge.

[kolak]

This is still rather limited.
Typically it won't tell you anything more than what you can find on the net.
Beauty is that ChatGPT will do it quickly and present it in nice form ready to digest

[Z2697]

Let's go back to original track a bit.

BUT... we're using Colormatrix() here and Colormatrix() DOES NOT internally convert to RGB as all the operations are done in YUV.

But even we're not going a roundtrip to RGB, that still means it needs to somehow make the subsampled planes "unsubsampled", right?
I don't know how this filter works, but I assume the simplest way is to use the equivalent of PointResize.
Either it creates a upsampled copy in memory, or just reference the same pixel 4 times, the effect should be identical, right?

Now here comes the second question: when you do this roundtrip to RGB or 4:4:4, what resampler to use?
I again assume PointResize might be a good choice because you avoid "actual" interpolation, if not doing any other operation, PointResize shoule be just perfectly reversible by itself.
But is the "blocky" chroma detrimental to the matrix conversion?

Since we are talking about AI: most image / video processing NNs nowadays take RGB as input. For the roundtrip, what resampler to use then? (we probably should take different types into account: super resolution, restoration, frame interpolation, etc.)


EDIT:
(I still can help but go off track)

No, it can synthesize of what it knows, so it can come up with answers never said in previous (training) knowledge.

Beauty is that ChatGPT will do it quickly and present it in nice form ready to digest

Also don't forget it sometimes lies. It generates wrong answer with unwavering tone, not the actual lying of course, that's a human (or post-human skynet "AGI") thing.
It's generating. It's not answering. It generates something resembles an answer.
It's lying, not because it want to lie. The machine and data determines it will lie.

You can still argue that we humans will give wrong answers, we will lie, we don't even know if it's really us that "want" to lie.
We don't know what Intelligence is. We don't understand conciousness.
But this is a bottomless rabbit hole as of now.

[GeoffreyA]

Indeed, till we crack the riddle of consciousness, we can't answer some of these questions. My feeling, though, purely speculative, is that this is the primitive version of the technology in our brains. If the result is eventually the same, is there really a difference? As time goes by, people are going to deny that these things match the human mind, or when sentience is implemented (or emerges, which I doubt; simple creatures have awareness; thus, it is likely some circuit or architecture), deny that too because of feeling threatened or human pretensions to superiority. But this is rather for the realm of machine ethics.

Back to the topic a bit, another point left out by the models' answers is downscaling in the linear domain, though Claude seems to be vaguely scraping the surface of that.

[Frank62]

Why does the single-celled organism move forward? It has no stomach that growls, no brain that can think about the fact that it might find something to eat after moving forward, or at least that the probability of doing so increases. A simple mathematical odyssey? Hardly. God? Possibly at the very end (I abstain), but the path before that is still completely unclear!

My opinion, translated with DeepL (to write good English once).

[GeoffreyA]

Originally Posted by Frank62

1. why does the single-celled organism move forward? It has no stomach that growls, no brain that can think about the fact that it might find something to eat after moving forward, or at least that the probability of doing so increases. A simple mathematical odyssey? Hardly. God? Possibly at the very end (I abstain), but the path before that is still completely unclear!

I think it could be some low-level mechanism based on chemicals that drives a cell to X, Y, Z. Without it, life would flounder.