Sneaker is right.
Just changing the frame rate doesn't change the frame count.
If you increase the frame count the encoding time goes up, if you decrease the frame count encoding time down.
For the encoder the fps shouldn't really matter (aside from calculating an average bitrate or similar).
So if you use SVP or ffmpegs
framerate,
minterpolate or simple frame repetition you create new frames which increases the frame count thus encoding takes longer.