It happens from time to time, regardless of Game Mode.
Unless it happens frequently, no reason to tear the own networking apart.
In case of frequent issues though, there's indeed a few things to check.
- WLAN reliable/unreliable
- Firewall/AntiVirus/Personal Security Center Software intervening
If no solution is apparent, in the past disabling UPNP and setting up Manual Port Forwarding in the Router has helped alot (!) of people with such or similar issues (i.e. distinct lags and instancing problems, Mission board timeouts etc.).
A quick guide on how to set up port forwarding for Elite Dangerous.
support.frontier.co.uk