OK! I assumed the flickering edge issue would still be an issue, but it's not.
The problem I was having, is that I'd be watching my replay as I'm editing it, and I would see the whole scene. But then I would render, and it wouldn't render out those extra chunks.
Now I know why. The rendering system (and rightly so) only starts rendering chunks from the starting point of your first time keyframe. So any chunk that would have been rendered prior to that point, wont show up on the replay render.
So basically, if I want those chunks to show up in a replay render, I need to include the part of my recording where I walk around to get those chunks to load, as part of the replay render, then edit that beginning part out in post.
Thank you for your help. I'm not sure if there's an efficient way for your system to make sure to render those extra chunks prior to starting the clip... it would likely add a significant amount to the render time. I'll just have to add my work around to my work flow :)