Any nearby planets or stars will slow you down and the closer you are to them the greater this effect becomes.
No they won't slow you down! You would accelerate as you get drawn towards the gravity well, and using this effect you can sling-shot yourself around something with heavy gravity to increase your speed.
This is a common way to increase speed of probes we are sending out through the solar system right now.
I was actually disappointed the first time I played this game and I couldn't sling-shot myself around the different planets to boost my speed