It just swings back and forth super sensitive and always over compensating.
Hate to point it out, but it's you effectively over-compensating.
FA-Off means your ship stops firing thrusters to maintain your ship's heading and instead you're now flying with real Newtonian physics. That means when you push left on your stick, your ship will continue to spin left equal to the thrust you've given it, until equal force is applied in the opposite direction. If you then apply too much thrust to the right, it will spin to the right.
Basically you have can't just pull right to turn right. Instead, you push right briefly and let go.
To counter that and come to a stop you (very gently) push left and let go, and keep doing it until you've built enough thrust to stabilise - if you over do it, you then have to push quickly in the opposite direction.
As others have said it's certainly not impossible, but once you get it into your head that you don't just pull on the stick, and you have to be real gentle with the force you apply, it does get far easier.