As you noted, the RM records what a single player sees and that is no coincident. It's actually rather integral as to how the mod works and cannot really be changed.
The RM works (roughly anyway) by recording every packet the server sends to your client and during replay it pretends to be a server and just replay those packets.
The main reason the RM works this way is because it generalizes pretty damn well as opposed to the other option which would be to manually record every action in the game. That is every entity move, block change, entity action, scoreboard change, time change, etc., etc.. There are lots of things which one would have to write specifc code for and once you add mods, you can basically forget about doing it that way. The way it works right now, we had to special-case one or two forge things and thousands of mods just workedâ„¢.
So, to record everything the way the RM does it, requires recording every player's connection. Now to address your individual questions:
"Replay Files are pretty small [...]"
Resource consumption without doing any additional optimizations will be huge since you're duplicating lots of data when separately recording every player's connection.
"Based on what I've seen from ReplayMod [...]"
You are correct in your observation. However, the server would not have to restart every hour. If you want replays no longer than an hour, you'll have to disconnect every player at most an hour after they connected though. Not too great.
"Current ReplayMod records what a player sees [...]"
As explained above, the way the RM works this cannot happen.
"I know there are a lot of limitations [...]"
Yes, kind of, unfortunately. At least it won't be as nice as one would hope (i.e. just recording the usual, limited recordings on the server side) or require tons of work (i.e. going the manually recording everything route).
Having said that, iirc there are/were two larger German servers which did go there.
One allowed you to download replays of your games. IIRC the game map was small enough as to where it was sufficient to record it from the perspective of a fake player placed centrally on the map.
The other had a system completely independent from the RM, no idea how it worked on their backend.
I don't remember either of their names but given they were big, for-profit servers, I doubt they'd share.
There's also this which does the one-replay-per-player-recorded-on-the-server thing but never actually ended up being usable.