I suppose the question is, putting ReplayMod aside, (and forgive me if the answer is, "of course I do"), but do you have shaders running to start with. I did and when I added ReplayMod it just recorded with shaders running.
I'm not sure I can tell you what to do other than tell you my spec which records fine with shaders running:
I'll highlight some bits I think are important:
• Windows 7-64bit fully patched with latest Nvidia graphic driver.
• Launching Minecraft 1.11.2 via MultiMC 5.0-945
• Java version 1.8.2_131
o Min mem: 1024MB
o Max Mem: 2048MB
• Resource Pack:
o ChromaHills-128x1.11-v1
• Forge: 13.20.1.2386
• Mods:
o OptiFine_1.11.2_HD_U_C1
• Replaymod-1.11.2-2.0.0
• XRay-48 (I know, but then the real world has seismic detection)
• ShaderPack: SEUS-v11.0.zip
Good luck.