I'd be very surprised if LSmash converted the colorspace. It has an option for doing so but you have to specify it. It's far more likely DirectShow was converting it. If you uploaded a small sample somewhere it'd speed up the process of finding the problem considerably. It's just a guessing game otherwise.
Could I be so bold as to condense your script a bit? It shouldn't have anything to do with the problem displaying the overlayed video. I don't know why that's happening.
Code:
vid1 = LSmashVideoSource("Y:\test.MOV")
vid2 = Vid1.GradFun3().ReduceBy2()
Overlay(vid1, vid2, x=0, y=0, mode="blend", opacity=100)
Although to compare the two, it might work better to stack them.
Code:
vid1 = LSmashVideoSource("Y:\test.MOV").Spline36Resize(640,360) # any resizing optional
vid2 = Vid1.GradFun3()
StackVertical(vid1, vid2) # or StackHorizontal