I used to upgrade mine all the time and never had any issues with it. If you're running SLI it can really be bad. But lately I haven't been doing it so much as VR can be a bit touchy and my card is struggling to work with it.
Suppose everyone's set up is slightly different, but I upgraded the Nvidia driver (for Oculus Dash). No problems at all.
As to my settings - running SS at 1.2 and HMD at 2.0 ATM (with some of the other graphics settings turned sown). Very happy with the results, though have a 1080ti, so has the overhead.