Look down to your SRV info screen, you'll see an option for "Ship Recall"
And just FYI, if this hasn't changed recently, if you ever deploy your SRV while your ship is docked at a starport, you'll have to travel at least 4km away from the spaceport before the option to recall it is available, the other option being getting back to the starport and requesting 'docking' clearance from your SRV, get in the indicated hangar and board your ship.
LOL. The first (and only) time I tried out the SRV, I deployed while docked, had a quick drive around and then couldn't board my ship. Had to drive away from the port for 4km and recall the ship. Didn't realise I needed to request docking from the SRV to re-board a docked ship. Oops.
@Steve_T: to be honest, I did exactly the same thing, found out you could request docking clearance from the SRV afterwards from somebody else on this forum
Never had a chance to fiqure this out so I drove to the spaceport, got wanted for trespassing, drove mad around, then crashed my SRV till I died as I had no idea how to dock. Will never leave ship in a starport again