Normally, mkvmerge takes way less memory; there are basically two possibilities why this could happen:
a) Your file might contain only few frames that are detected as keyframes by mkvmerge so that mkvmerge has to buffer a lot of material before it can write it again. If so, then your source material might simply not contain a lot of keyframes; or they aren't detected by mkvmerge. The latter case would be a bug.
b) There is a memory leak going on. This would also be a bug.
If you want to help, you should upload the source file (not the remuxed file) to
the server described here and open an issue
here.