Looks like this was broken in all of 2.x. Could you give this version a try?
Some fast-forwarding is to be expected and cannot be avoided because of how the mod works but there should be at most one world change (the initial join). Especially for minigame servers, that should make a huge difference.
Well, I did fix a rendering related issue between the first and the second version I've linked.
But that issue should be unrelated to which encoding you're using and be purely dependent on whether the mod has to rewind the replay to start rendering.
How often have you tried rendering MP4 vs not MP4? i.e. is it possible that you mistook the above issue as being caused by the MP4 setting?
To test that, just try using an old version (i.e. 2.1.1) and render without MP4 but make sure to jump to the beginning of the replay before rendering. If it's been the above issue all along, then that'll work. If it's really something with the MP4 vs not MP4 setting, then it's another issue.