Yeah, that's a known problem with Qt 5.12.0 on macOS. Qt 5.12 has other issues, too, but it does solve the very real issue of 5.11 not being usable at all on Mojave in dark mode. Meaning I'll stick with 5.12. I'll definitely update Qt as soon as new versions come out.
Your options are, obviously:
- Stay with a version before v30.
- Stay with a version after v30 and live with the bad font rendering.
- Compile MKVToolNix v30.1.0 yourself against Qt 5.11.x (MKVToolNix is fully compatible with earlier Qt releases down to 5.3 or so).