This is not the fault of xy-vsfilter/XySubFilter but of the splitter/container format. The subtitle in e.g. mkv is stored at 2:00 and if you skip to 2:03 it might not be read by the splitter and consequently never forwarded to xy-vsfilter/XySubFilter. New additions to mkv could be used to fix this but
a patch for LAV splitter with support has not been accepted.