Page 1 of 8 1235 Last
Results 1 to 15 of 116

Click here to go to the first staff post in this thread.
Thread: Networking Changes in v2.2.03

  1. Click here to go to the next staff post in this thread. #1

    Networking Changes in v2.2.03

    We're constantly trying to improve the underlying systems code in the game, as well as the gameplay, but sometimes it can be difficult to diagnose and fix problems when you can't reproduce them in-house. In order to help understand the causes of instancing and connection problems, we have been working recently with the Fuel Rats, to collect network logs of any rescue attempts that didn't go as smoothly as they should.

    Some of the issues we have seen from these reports have already been fixed in the live game, with hot-fixes to the servers. If you're already in a wing with another player, and you're trying to meet up, then you should be assigned to the same server when jumping into the system (even is one player is un USA and the other is in Europe.)

    We have a number of fixes to the networking code which we're testing in this new beta, but in order to explain the changes I'll first need to explain about 'Turn'. When we're trying to set up a connection between two player machines, it's sometimes the case that due to the way the routers or firewalls are configured, it's not possible to establish a direct connection. In this case, we follow an internet standard called TURN (rfc5766) to relay the packets from one player to the Turn server, then back to the other player.


    Bug no 1: Prematurely Skipping to Turn

    Because of the timeouts and retries, it normally takes around 15 seconds to decide that a direct connection isn't working, so we should switch to using Turn. Now we know that we're never going to be able to set up a direct link between certain types of routers, and we're exchanging info on the router type along with the connection addresses, so in those cases where we know we're not going to succeed with a direct link, there's an optimisation to go straight to Turn: however this wasn't taking into account those cases where one of the players had set up manual port forwarding on his router (in which case a direct connection should be possible.)

    In the latest beta, if you have configured manual port forwarding, this info is also passed to the other player, so we don't skip straight to Turn when a direct connection should be possible.


    Bug no 2: Incorrect Letter Fragmentation

    The networking code exchanges packets from one machine to another; each packet contains one or more letters, but a packet cannot be more than 1500 bytes (maybe less, depending on the MTU.) One of the network logs from the FuelRats showed an error where a large letter (over 4k bytes) had been broken into smaller letters for transmission, but then one of those fragment letters was still too big to fit into the packet. This bug would eventually result is a p2p disconnection.

    What was happening was at the time the letter was being broken into fragments, it was using the theoretical maximum packet size for the connection; however when it came to put the second or subsequent fragments into a packet, the buffer size for the packet was actually smaller than expected (because it was communicating over Turn!) This bug is also fixed in the current beta.


    Bug no 3: Initialisation Race Condition

    One of the things we need to do at startup is to identify the type of router: this can sometimes take several seconds. In some cases, we were connecting to the server before this process was complete, and passing incomplete connection details to the server (in particular, this left out the Turn details) - these incomplete connection details would then be passed on to other players, and if a direct connection proved to be impossible, it would not then be able to fall back to using Turn. We have a fix for this in the pipeline for beta3.


    Bug no 4: Handling Port Forwarding

    As mentioned above, some players set up a manual port forwarding rule on their router, so that (for example) any packets coming in on the router's external port 5100 should be mapped to their PC's local port 5100. They would then set port="5100" in their appconfig.xml. However this port forwarding usually only applies for incoming packets: when the PC sends a packet out, the router may select a direct random external port to transmit from. This means that when our server receives the packet, it thinks that random port number is the one to reply to (which works, because the router can see it's a reply), and it also uses it when telling other players about how to connect to the machine (which typically will not work).

    Back in summer 2015, we added another appconfig setting, eg. routerport="5100" which means the game will tell the server that manual port forwarding is in use, and the server should reply to that port 5100. However this new setting was not adequately communicated to the players, and relatively few have set this option.

    In beta3, the game will assume that if you have set port="5100" in your appconfig.xml, this means that you have set up port forwarding in your router, and the routerport option should no longer be necessary (unless you're using a different port number, I can't see why you would want to do that, but I'm not going to prohibit it)

    For most players using a domestic broadband router, manual port forwarding should not be necessary - if the router supports UPNP the game can tell the router what ports to use. In the current beta, only around 1.5% of the connections are from players with manual port forwarding.


    I'd like to thanks the Fuel rats (especially Cmdr Absolver, Cmdr Termite Altair and Cmdr Curbinbabies) for their help in investigating these problems, along with Cmdr Jan Solo for his log files with evidence of the race condition bug. We will continue to look into bug reports: if you think there's a networking issue, please submit a support ticket, and supply network logs if possible, but I hope this fixes will make a noticeable improvement to network stability.

  2. #2
    I don't suppose you have a list of which bugs this fixes from the player's perspective? Like "everyone in a wing except the leader has their game freeze at once" or "when docking at a station with a wingmate, the interior desynchs" etc? It's great that you're fixing stuff but I've no idea how any of what you're saying relates to my actual game experience.

  3. #3
    By letter, I'm assuming you mean game events that are sent via a formatted/serialized message that needs to be reassembled from fragments to be properly read.

  4. #4
    It is excellent update long overdue Howards, huge thank you for keeping us in loop what's going on with improvements for networking and it is nice to hear you have employed players to help you in specific bug cashing (y)!

  5. #5
    Originally Posted by Pilchenstein View Post (Source)
    I don't suppose you have a list of which bugs this fixes from the player's perspective? Like "everyone in a wing except the leader has their game freeze at once" or "when docking at a station with a wingmate, the interior desynchs" etc? It's great that you're fixing stuff but I've no idea how any of what you're saying relates to my actual game experience.
    I concur with this, some of the symptoms should be spelled out for non-technical players for them to report whether your changes have had any effect.

  6. #6
    Deleted to prevent people disabling their registry.

  7. #7
    Hmm. In a way this is the best news of the complete beta for me yet. Or at least it is, in case i can persuade my friends who already quit to return and give it another try.
    .

  8. #8
    Fixes are always good, but it might be helpful to identify which routers present the most problems so players can consider upgrading their equipment.

    This still won't fix people deliberately "massaging" traffic to cause problems though.

  9. #9
    Howards, while you at it - maybe FD QA and you guys might benefit or having some sort of special "known/reported issues" list for networking? Posts like these gives good indication where we are regarding known issues and are there ack from you guys and if there's fix in works. Would give lot of people a bit of peace of mind.

  10. #10
    .
    Originally Posted by Pilchenstein View Post (Source)
    I don't suppose you have a list of which bugs this fixes from the player's perspective? Like "everyone in a wing except the leader has their game freeze at once" or "when docking at a station with a wingmate, the interior desynchs" etc? It's great that you're fixing stuff but I've no idea how any of what you're saying relates to my actual game experience.
    .
    Definitely interesting is bug #2. That one very much describes what i experienced many times, that connections were desynced and some people in an instance were not able to see other people there any more. My prefered example is when i was observing a training fight and both suddenly claimed that the other one combat logged, while i still was able to see both of them flying around and complain about each other... they also did not get each others chat messages any more and thought the other one went offline.
    .

  11. #11
    Based on the config file(s) being mention, these fixes appear to be for PC. Will these changes also fix issues encountered by XBox players?

  12. #12
    So… uh, how about IPv6? For a game like this that relies heavily on P2P networking, that should be a very interesting thing to support since it would just make most of those problems except fragmentation non-issues between IPv6-capable systems. From a conceptual point of view, "dual-stack" clients don't seem like a very hard issue?

    Originally Posted by CMDR Dreamstate View Post (Source)
    Disable Nagle’s Algorithm
    There's no Nagling in UDP.

  13. #13
    Originally Posted by Shadowdancer View Post (Source)
    There's no Nagling in UDP.
    Not all ED connections are UDP though.

  14. #14
    Originally Posted by Cmdr Eagleboy View Post (Source)
    Not all ED connections are UDP though.
    The P2P ones that are the problem would be though.

  15. #15
    Originally Posted by Shadowdancer View Post (Source)
    The P2P ones that are the problem would be though.
    Just a reminder, there are regular issues with HTTP connections using REST when connecting to gameplay servers too. But I understand your point though - this OP is mostly abou peer to peer issues, which is separate subgroup in ED case.

Page 1 of 8 1235 Last