Being in space has very little to do with it.
Well, it has a number of things to do with it. As you mention later, space maneuvering and being able to ever land/takeoff from high-gravity worlds are very different beasts. You can spin a ship around it's axis pretty quick with the right thrust at the right moments in free-fall. Far easier to spin around than to arrest it's previous motion and move in a new direction though which is probably where I see the most issue with ED flight. Ships stop and change vectors on a dime. But without numbers it's hard to say. I kind of want to give SE a try too.
Even if you did have infinite thrust, then you're dealing with insane G forces, not just on the pilot, but the superstructure. Try flying a 747 like you fly an F-16, and you'll likely snap the wings off or crack the fuselage.
True, but here again atmospheric drag plays a big part in that too. Outside that you'd kill the pilot before the ship folded up. But yes, being able to take off from 2,3 and more G's is a bit comical. The energy required to reach escape velocity would be insane. Even 1G would be a struggle for most ships. It takes a skyscraper full of chemical energy to get a literal tin-foil ship up from 1G.
Rule of cool I guess.