So, after much suffering (because nothing helped against the crashes, not even running it on 64bit avs+) I figured out to take a look at the intermediate avs scripts generated by the optimizer to find what has brought it to its knees.
https://pastebin.com/bgUELWu6 this one crashes 100% of the time, can you tell what the problem is?
I'll save you time, it's the combination of pel 4, blocksize 8 with overlap 4, divide > 0 with large search radius and (surprise!) removal of ConvertToYV24. What does it have to do with MT? Well I don't know, ask the mvtools maker about their error reporting style.
Further testing revealed that for pel 4 it is enough to have search radius of 4 to cause error, with pel 2 it takes around 12 and I couldn't reproduce it for pel 1. While I admit that having a search radius larger than block size seems strange it doesn't cause an error for YV24 or divide 0 / pel 1. What's more weird is that unlike the chroma subsampling violation it does not necessarily raise an error right away, sometimes it happens in the middle or at the end of the script, sometimes it's not the MT error but some random access violation. Moreso, setting chroma to false doesn't help (but converting to YV24 still does even then). Nothing in the readme has prepared me for this.
Worst thing I don't even know how to report it, the thread has been abandoned for months.
I guess more filters/minmaxes are in order, but with the current notation it's hard to figure out how to write them. Set divide to 0 when blocksize is 8 and overlap 4 and pel > 1.
In a way, Optimizer can be used as an automated plugin testing tool, since it tries out so many parameter combinations and reveals all kinds of bugs and readme inconsistencies.
Also, zorr, you might want to link you large explanative posts from this thread in the first post, now that the discussion took off it'll be more difficult to find them later.