Games with first-person fighter-like combat, like Elite: Dangerous, become a hell of a lot less fun when maneuverability and speed are lessened.
I think having to account for module placement/mass distribution, structural reinforcement trade-offs, cargo, passengers, etc, when deciding how hard to push those manuvering thrusters, would be a hell of a lot more fun. Indeed, the best first-person fighter-like combat games I've played took at least some of this sort of stuff into account, and you could readily back/redout or rip the wings off your aircraft by pushing things too far for the situation or loadout at hand.
Do you kill all those imperial slaves in your hold by trying to keep your Anaconda pointed at the Viper orbiting you? Do you armor all of your ship, or do you leave modules placed far from the center of rotation more exposed, so you can concentrate mass and protection near the middle for better manuverability? Can your ship, shot up as it is, even hold together if you do? Are you willing to take that risk? Maybe if you jettison some fuel/cargo/modules to lighten things up and/or keep them from crashing around...
All decisions I'd like to be able to make.
We don't need real-world constraints...it's fine if our ships and our CMDRs are considerably tougher than modern materials and people. However, the utter lack of limits deprives us of much gameplay potential.