Here's a ReplayMod version which includes this Quick Mode feature: https://minio.johni0702.de/replaymod/replaymod-1.12.2-2.1.4-8-gef36825.jar
You can change the Minecraft version in the URL to any of the supported ones. The Quick Mode will only be available on 1.9.4+ though due to the additional development effort being required to port it to 1.8.9 or older not being worth it (at least for now).
You can enable it while in a replay by pressing Q (iirc) or clicking on the arrow in the bottom left of the screen and then selecting "Quick Mode".
The inital setup of the Quick Mode might take some time, any subsequent loading should be faster (similar to the entity position tracker). Also be warned that it'll require quite a bit of RAM, don't expect to load a multiple hour minigame replay with less than 4GBs of it.
Also make sure to disable the Quick Mode before rendering. That doesn't yet happen automatically.