We're constantly trying to improve the underlying systems code in the game, as well as the gameplay, but sometimes it can be difficult to diagnose and fix problems when you can't reproduce them in-house. In order to help understand the causes of instancing and connection problems, we have been working recently with the Fuel Rats, to collect network logs of any rescue attempts that didn't go as smoothly as they should.
Some of the issues we have seen from these reports have already been fixed in the live game, with hot-fixes to the servers. If you're already in a wing with another player, and you're trying to meet up, then you should be assigned to the same server when jumping into the system (even is one player is un USA and the other is in Europe.)
We have a number of fixes to the networking code which we're testing in this new beta, but in order to explain the changes I'll first need to explain about 'Turn'. When we're trying to set up a connection between two player machines, it's sometimes the case that due to the way the routers or firewalls are configured, it's not possible to establish a direct connection. In this case, we follow an internet standard called TURN (rfc5766) to relay the packets from one player to the Turn server, then back to the other player.
Bug no 1: Prematurely Skipping to Turn
Because of the timeouts and retries, it normally takes around 15 seconds to decide that a direct connection isn't working, so we should switch to using Turn. Now we know that we're never going to be able to set up a direct link between certain types of routers, and we're exchanging info on the router type along with the connection addresses, so in those cases where we know we're not going to succeed with a direct link, there's an optimisation to go straight to Turn: however this wasn't taking into account those cases where one of the players had set up manual port forwarding on his router (in which case a direct connection should be possible.)
In the latest beta, if you have configured manual port forwarding, this info is also passed to the other player, so we don't skip straight to Turn when a direct connection should be possible.
Bug no 2: Incorrect Letter Fragmentation
The networking code exchanges packets from one machine to another; each packet contains one or more letters, but a packet cannot be more than 1500 bytes (maybe less, depending on the MTU.) One of the network logs from the FuelRats showed an error where a large letter (over 4k bytes) had been broken into smaller letters for transmission, but then one of those fragment letters was still too big to fit into the packet. This bug would eventually result is a p2p disconnection.
What was happening was at the time the letter was being broken into fragments, it was using the theoretical maximum packet size for the connection; however when it came to put the second or subsequent fragments into a packet, the buffer size for the packet was actually smaller than expected (because it was communicating over Turn!) This bug is also fixed in the current beta.
Bug no 3: Initialisation Race Condition
One of the things we need to do at startup is to identify the type of router: this can sometimes take several seconds. In some cases, we were connecting to the server before this process was complete, and passing incomplete connection details to the server (in particular, this left out the Turn details) - these incomplete connection details would then be passed on to other players, and if a direct connection proved to be impossible, it would not then be able to fall back to using Turn. We have a fix for this in the pipeline for beta3.
Bug no 4: Handling Port Forwarding
As mentioned above, some players set up a manual port forwarding rule on their router, so that (for example) any packets coming in on the router's external port 5100 should be mapped to their PC's local port 5100. They would then set port="5100" in their appconfig.xml. However this port forwarding usually only applies for incoming packets: when the PC sends a packet out, the router may select a direct random external port to transmit from. This means that when our server receives the packet, it thinks that random port number is the one to reply to (which works, because the router can see it's a reply), and it also uses it when telling other players about how to connect to the machine (which typically will not work).
Back in summer 2015, we added another appconfig setting, eg. routerport="5100" which means the game will tell the server that manual port forwarding is in use, and the server should reply to that port 5100. However this new setting was not adequately communicated to the players, and relatively few have set this option.
In beta3, the game will assume that if you have set port="5100" in your appconfig.xml, this means that you have set up port forwarding in your router, and the routerport option should no longer be necessary (unless you're using a different port number, I can't see why you would want to do that, but I'm not going to prohibit it)
For most players using a domestic broadband router, manual port forwarding should not be necessary - if the router supports UPNP the game can tell the router what ports to use. In the current beta, only around 1.5% of the connections are from players with manual port forwarding.
I'd like to thanks the Fuel rats (especially Cmdr Absolver, Cmdr Termite Altair and Cmdr Curbinbabies) for their help in investigating these problems, along with Cmdr Jan Solo for his log files with evidence of the race condition bug. We will continue to look into bug reports: if you think there's a networking issue, please submit a support ticket, and supply network logs if possible, but I hope this fixes will make a noticeable improvement to network stability.
