The farther away from an "object", the faster the movement - it is a "law of Physics" in Elite Dangerous.
Try in orbital cruise, to choose the destination point (settlement) into the target and take advantage of the "compass".
If the problem is not the search for settlements, in that case you should change the description of the subject matter.