The PC is not the only factor, but it is the major factor.
The lower the hardware the more significant the delay is. For example, I run Horizons on both my PC & my Xbox, on the PC there is very little lag at 1080p (though there is a little) on the XB1 at the same resolution there is much much more. As I'm currently forced to play at 720p (due to a monitor "accident") the lag on my PC is almost unnoticeable, the xbox hasn't really improved however, presumably because it is still processing at 1080p & downscaling to 720p.
Given that the XB1 hardware is a low end PC at heart (very low end now) trying to replicate the settings that a higher end PC is producing, it isn't suprising that there is a delay for some actions, the PS4, even the Neo is going to suffer from the same issues as it is essentially the same hardware. The same is true if I try to up the graphics settings on my PC, upgrading the quality or resolution will have a corresponding fall in frame rate & overall responsiveness it's more prominent in Elite because it is especially punishing on both CPU & GPU, bottlenecks can & will be prevalent in both, especially in busier areas.
In other words, turn the graphics wick down (especially supersampling & shadows) and you will see vastly improved menu responsiveness (except on xbox).