Yes, I know exactly what you are doing :)
When you hit "V" to synchronise the camera timeline with the master timeline it is dependent upon the playback speed setting.
For example:
You set a time and camera keyframe then start playback to watch the action.
You set the playback to x8 to get through it faster so you can set your next set of keyframes.
You find the spot and hit "V", the cursor jumps down the camera timeline and you set your next set of keyframes.
BUT if you stop here, set playback speed to x1 (even though it is paused) and now hit "V" the cursor jumps eight times further down the camera timeline than it did before. And it is this time on the camera path that Replay will attempt to fit the footage on the master timeline into.
Whatever you have playback speed set as when you hit "V" basically sets the speed that the footage will play back at in the period covered on the camera timeline.
Does that make sense?