You are likely experiencing
madVR issue #205 which only seems to occur with mod2 video when using madVR >= 0.87.0 with certain settings. Please check the madVR setting workarounds posted in that issue (disabling optimize subtitle quality for performance, or enabling debanding), and also EVR-CP. This isn't a bug with XySubFilter itself.