The ReplayMod should, during recording, not influence performance at all. If I'm not mistaken, literally the only thing which could impact performance during recording under normal operation (given you have more than one CPU core) is the recording indicator in the top left and that's practically nothing.
Please post the contents of the fml-client-latest.log
file inside the logs
folder inside your .minecraft
folder to some pastebin site and link to the post here.
Note that this log will only contain information on the last time you started Minecraft, not any previous runs.