That custom resolution feature seems to work for me right now. I played two different videos for over an hour each and tested two optimized pixel clocks with a compatibility of 50. I got two times a hit.
There is an Extension blocks section on the main window of the Custom Resolution Utility and I had to delete the resolution I currently try to optimize from two different Data Blocks called TV resolutions and HDMI support (Screenshot).
I don't use CRU to add those resolutions to my system. madvr is doing all this work and now I wonder if it will work a third time.
EDIT: It worked for a third time.