Be warned that I don't use OSX nor have I ever used it.
FFmpeg should not have any undesirable side effects on any other part of your system. That doesn't mean however that it can't. You might already have some program installed that supports ffmpeg if it finds it and then uses it once you install it. Even though unlikely, this could also lead to said program no longer working correctly. In short, answering whether FFmpeg will affect anything at all on your computer, is not possible without knowing every tiny bit of it.
I've never used homebrew either. However, from how I understand things, you can't remove homebrew after installing ffmpeg without loosing the ability to uninstall ffmpeg itself.
If you want to be on the safe side, you can download a static build of ffmpeg (should be a .7z file), unzip that to some arbitrary place of your choosing and point the Replay Mod at the contained ffmpeg executable in the Render Settings. Using the static build should have an abysmally small chance of affecting your computer in any other way (except used disk space).