Have you monitored your CPU/GPU utilization while playing in these conditions?
I don't have these chips or a headset of this res, but my hardware is no slouch... I find that the game regularly dips into lower frame rates without utilizing even close to all my hardware's power. This is true for flat and VR, and the common factor I can find is NPCs. Large numbers of NPC ships or NPC persons cause the game to slow down.
To make it easier to view (and to take VR reprojection out of the equation) I tried some of these scenes on flat mode and found that I could get it to dip below 100hz (the refresh of my screen) into the 70s and 80s inside concourses and see that my CPU utilization is in the low 20%s and GPU was around 60% with 1-2% variance. Temps all nominal, nothing throttling, etc.
I see similar things in VR, of course, but it's a bit harder to measure.
Clear evidence that stronger hardware wouldn't solve it, right? The game isn't using what's there already.
To confirm it practically, I fly down to a settlement in VR and on approach the performance is really terrible. Lower than the half refresh rate reprojection in the headset.
I land and murder every NPC, then return to my ship and take off.
Performance is much, much better now. Easily maintaining half-refresh rate reprojection and much smoother frame time variance --the stuff you really feel.
So what changed here is that the NPCs aren't doing any AI. The models are still there laying on the ground, so I think the problem can't (entirely) be up to not enough LODs for VR. And this happens in space also. and frankly it happens in horizons also, though that's so much easier to run that it doesn't stick out to us as much.
I've been doing a lot of AX conflict zones lately and the performance in there can be described as "fine, with periodic massive slowdowns". The FPS will be nominal and then spike for a few seconds of slideshow, and then return to normal. This happens in horizons or Odyssey; they feel pretty much the same. And if I check my performance monitors for those moments, they aren't spiking up to 100%. Sometimes I can even see dips where the GPU goes idle...
Something the game is doing with AI/pathing/whatever we want to call it is causing the process of drawing frames to... I don't know... "wait".
Edit since it's not entirely irrelevant: My machine is 5800x / 3080. They're both overclocked but nothing crazy and they stay cool and maintain their boost speeds. Headset is Rift S at 1.5 super sample, monitor is 3440x1440 100hz when flat.