I searched within the Ripbot thread and saw a few people having success (namely
this post and
this one), and it seemed like the key thing might be to ensure that Audio track 1 was AAC. I didn't have any success getting this done with MP4muxer though.
I ultimately had success using Handbrake (64-bit on W7) just using the Apple TV preset. It produced an Apple TV compatible M4v file with both an AAC and AC3 passthru channel, and upon playing this file, my receiver began outputting 5.1 sound.