If you have x264 with ffms2 built in, you can just use the fraps .avi directly and specify "--colormatrix bt709 --fullrange on". With this, all supporting players should show correct colors. Of course, not all players support it
As far as I know there is no way to convert colorspaces with only x264. When using Avisynth, you should use ColorMatrix rather than ConvertToRGB as it'll be faster and should retain better quality.