Almost a year later, Multiplayer is still a mess

Status
Thread Closed: Not open for further replies.
Star Citizen will be better no doubt! :p

But seriously: We have a mostly-offline game at the moment, with players often not noticing when the matchmaking fails because you can play just as well alone. People asking where everyone else is on the forums... it's not that they're far out, it's that for some the P2P just doesn't work.

THIS IS NOT A DEBATE ABOUT OFFLINE MODE.
This is about the issues of P2P networking and the limitations thereof, as well as my suggestion on how to improve it.



I've warned repeatedly about the problems and challenges of P2P in a multiplayer game like this. I've warned the Nornware guys (Space Beast Terror Fright) but they too said that "they know what they're doing".
Turns out that their main dev almost burned out of the problems of P2P. Why? Because you can't have consistency, availability, and partition tolerance all at once. You can trick a bit with eventual consistency, but the problem remains that P2P inherently brings with it the question of how to define truth. You can't establish facts when two equal-level participants talk to each other. You start giving one or the other authority over things in the game world - but then you get the problem of partitioning, which is when participants drop out and perhaps even back in again. If this sounds complex, that's because it is. I've spent the past 15 years of my life learning about this. It's *really* difficult. But don't get me wrong, I totally understand why Frontier chose P2P for Elite Dangerous, and their argument for that is sound. I do have a suggestion, but first let me state the current issues.

So what problems do we see in Elite right now:

1) Low player numbers per instance.
That's because P2P degrades for everyone when one participant with a bad connection joins. This is terrible for Elite because that's the reason why you often can't see your wingmate. Either he or you just happened to be put into an instance with an already low health, and it won't take both of you. There's no solution for this within P2P. This is the biggest issue with Elite at the moment.

2) Bad performance.
Look around in supercruise. Often times you can see ship trails jump and shudder around. Worse, interdictions get unfair because of the shudders and jerking around. One bad client ruins the entire instance for everyone in it. Dogfights work perfectly fine, but then everything goes to hell if one other player with a bad connection joins. Proper matchmaking can only help to a certain degree here.

3) Inconsistent world state.
Allegedly, Jesus said to Pontius Pilate: "What is truth?"
I am certain that he was a programmer of distributed systems, because that question is fundamental for a working multiplayer game. Elite currently "mostly" works. Mostly. It utterly fails when the situation isn't perfect. A few examples: You can duplicate mining fragments if there's more than one client in the instance. You might see cargo that your wingman can't. You can have visible cargo that you can't scoop up (also only when more than 1 player in that instance). You can see different elements in asteroids than your wingman. A wingman shooting a mining laser which you have placed prospector drone on will find that it will spew fragments indefinitely, even after the asteroid has been depleted.


I think P2P is here to stay for the general universe, and most of these issues can be resolved by server-moderation, just like they did with the landing pad assignments and the refinery. The point of truth is no longer one of the peers but a central server. Using a lightweight backend, for example nodejs with redis as storage, will support tens of thousands of clients per server, and you can scale that out if you're choosing the right way to do the business transactions (idempotent operations, optimistic locking, etc)

But for central areas of the game, we absolutely need client/server. This doesn't mean Frontier should host thousands of game servers - just a couple dozen would suffice. These servers could host client/server sessions for warzones, Sol, Eravate, and other well-frequented systems and places. These client/server sessions could be used for all parts of the game: Supercruise, normal flight, you name it. They could be assigned dynamically and support a large number of players (128, 256, maybe even 500 or more). If a server is full or not available, the game could just fall back to P2P.

This would allow for...


1) Giant warzones that really do feel like there's a war going on

2) Systems that feel alive and buzzing with human traffic

3) Much more interaction and communication between commanders, something Elite more or less completely lacks atm.

Best of all, Frontier could scale the available servers up and down depending on current budget and playerbase. They wouldn't lose the flexibility of P2P, but they'd still have the ability to make community goals and warzones actually a giant thing that everyone wants to participate because it's something you experience anywhere else at that time.


I'm posting this in the hopes that this catches the eye of someone at Frontier. But also for everyone to discuss. I know the game works "most of the time" and I'm not saying that they did a bad job. However, before you disagree saying that we don't "need" any of this, I implore you to think about this. For you nothing would change. You don't have to participate in a big warzone. But people who WANT that sort of thing, would then have the option to. At the moment, they don't.

And I really want Elite to become a little... bigger in it's multiplayer ambition :)


Agreed? Disagreed? Bananas?
I'm not disagreeing with the points you raise (and it's good to see a well-written and argued post). A hybrid system would be an interesting idea, if it were necessary.

However, I don't think Elite is, at its core, a multiplayer PVP game, with all the demands for improved connection performance that brings. People want it to be, but it's not. It's a co-op game. All of the patches and tweaks have been focused around that - I would even argue that this is why CQC exists: It's a low-cost PVP arena for straight-up guns-out "interaction" between CMDRs. In the base game, with the three network flags, the only constant point of interaction between CMDRs is the abstract background simulation. A great many players ignore the BGS, but perhaps it's actually more important in terms of the design philosophy of the game than people realise. It seems to me that the game is built around the BGS, and PVP is a secondary concern. The co-op tools in the game - friendslists and wings influencing instancing, for example - work pretty well for me. I've only seen one issue in the last few months, and it was solved by the player logging out and logging in again. Perhaps our Princess is in another castle?

Incidentally, the interdiction tunnel jumps around even when you're the instance owner alone in a system.
 
I'd reckon it's not quite dead yet and for a game that's supposedly has many more years of upgrades through expansions, it being a "massive effort" shouldn't dissuade FD from investing more in their network.

As I noted: On top of that, it works "good enough" for most people for most of the time. There is little incentive to actually change.

And as i also mentioned: Server costs - are you willing to pay a subscription fee?

And as i also mentioned: C/S brings its own problems. Effectively swapping one set of problems for another.
 
P2p in my humble opinion has effectively nerfed proper multiplayer, period. I honestly believe the only people who don't see that have never played a proper multiplayer game before......

Then i'm afraid you are wrong. I have played many proper (whatever that means) multiplayer games, and on top of that, ED works fine for me as a multiplayer game. I can meet other players, i can fight other players, I can work together with other players. Seems to qualify as a proper multiplayer game for me.
 
I have had, for a while now, similar thoughts to the OP. In all honesty, the very thought makes me giddy. I mean, dropping into Lave station and seeing the hundreds of player ships incoming and outgoing? Or a combat Community Goal that has a CZ-turned warzone of countless hollow triangles in a mess of a struggle against an equal amount of enemy ships? Actual threats of blockades (and blockade-breaking) that don't get sniped by instancing shenanigans? Hell yes.

I will admit that I don't understand all of FD's networking tools that make the game run. Even more so after seeing their presentation the other day, they've got so many protocols and servers running data in circles all over the place. In my limited understanding, I have an idea of how patching in a server system could work. That is (in theory), servers could be set up like super-clients, and the P2P would simply connect you to the super client when you're within instance range instead of making a new instance. Presumably, then it's just a matter of assigning servers to in-game locations and tweaking the super-client instance to allow more than 32 players (and probably some sort of prioritization of data, since clients probably don't need constant 100% real-time data about other clients more than double their sensor range away).

In the real-world? Well they'd need enough CPU time to cover the scope. Since the P2P system works great for small incidents in random space like interdictions, I'll pretend I know what I'm talking about and say scope could be Points of Interest (Stations, CZs, Nav beacons, and etc), and Supercruise. If we're talking just the bubble (plus an exception for any CG that falls outside it) and leave outer systems to P2P, that's still a large amount of Virtual Machines spread across a good number of real servers. They could probably shut off super-clients for systems that don't need it, by having a few "empty" VMs as a buffer ready to take over when a super-client instance is needed and turn on VMs if the buffer gets low, and they'd still need a good amount of servers. Plus, since latency is going to be an issue otherwise, they'd need the same setup done once somewhere in the US, another somewhere in Europe, and third in Asia, at the absolute minimum. I assume that's potentially a lot of servers to pay upkeep on, especially since they'd require a high-end data plan for each location.

Could E:D as it is cover those costs? Is any of it even possible anyway? Even if so, is there any chance of FD switching gears that hard? I have no idea. But I still hold out hope, because I would love so very much for it to happen.

--

As an aside;
Star Citizen will be better no doubt! :p
I dislike when people do this. As far as I know, SC is about as related to E:D as Call of Duty is to Team Fortress 2. The only real attributes that both games share is that they feature space piloting, and were both on kickstarter. In my opinion, trying to leverage for change by bringing up SC is at best an unrelated comment, and at worst disrespectful to both developers. Just my two cents.
 
Relax, the SC thing was a joke. There's a smiley next to it too. Neither game is a religion so it's ok to joke about things, right?
 
I stopped reading after

I stopped reading after "i stopped reading after".


About server costs: colocation is super cheap. Hosting two blade racks cost me 300$/mo for network and power. Hardware cost comes on top but it's around 10k for my stuff. Ymmv but just to show it's not millions.

AWS is also possible. We are not talking tens of thousands per month. And as i said, a hybrid system means FD could scale it up and down according to budget and demand. They could even stop at any time and p2p would still be there.

Also they already have servers for adjucation, matchmaking and other things.
 
Last edited:
I have found the P2P occasionally frustrating, but on the whole not too bad. (Compared to general experience of a wide range of other multiplayer online games.)

I do agree with the OP on limited C/S model. It would be ace to have some systems running as massive instances in AWS with a load of players in.

For instance you could have it tied into the galnet stories. Perhaps for an occasional system that can actually be blockaded for a couple of days while the event takes place. (IE no solo for that system)
 
I saw a thread yesterday where someone was complaining he couldn't go anywhere where there were no people. Can't make everyone happy.

-Pv-
 
I saw a thread yesterday where someone was complaining he couldn't go anywhere where there were no people. Can't make everyone happy.

-Pv-
if it's the guy I think you are talking about, he was annoyed that he couldn't find a system with 0 player influence which he determined by using the traffic report which reports however many ships visited (also counts repeat visits)the system. He wasn't talking about player instancing but rather the background simulator which is affected by solo and open.
 
Lots of people here cutting FD a lot of slack and saying C/S would require a monthly fee.

Thats kinda laughable. Living under an online rock last 15 years are we? The only thing it would require is some competence and a lower h&b budget.
 
Stopped reading when I got to the part where you said 'mostly-offline game' in reference to a game that requires an internet connection to play; I assume this level of accuracy and objective assessment is the bar for the rest of the text and therefore not interested.. Thank you for playing. NEXT!
Stopped reading when I got to the part where you said 'stopped reading when I got to the part...' in reference to a well reasoned, thoughtful post; I assume this level of laziness and arrogant discourtesy is the bar for the rest of the text and therefore not interested.. Thank you for posting. NEXT!
 
Good OP.
:
Two points.
:
The first is that although P2P is iffy now, as internet connections generally improve, P2P will probably become better over time. Admittedly, there will always be the issue of a single bad connection dragging the instance down, but over time it may be possible to "grade" the player base so it players with known bad connections aren't matched into otherwise good instances. FD have mentioned their strategy of building games that can be "up-settinged" to run well in 10 years time. I think they mentioned that coaster tycoon is still an earner today, it just runs on maxed out settings that would have required a super gaming rig at launch but is now within the ability of a budget laptop.
:
Secondly, (and I think zxabre alluded to something similar) would it be possible to modify the current pure P2P structure of each client talking each client, for one where each client gets updates directly or gets them from another client (which i'll call the chairman). Essentially any instance can have a 'chairman' client that is the "arbiter of truth" for that instance. Any missing packets from a bad client are essentially substituted with information from the "chairman'. The 'chairman' client (like any other client) gets information from every other client but if any client data is missing (due to bad connections) it can "fill in the blanks" for itself i.e. this client was flying straight, so i'll assume it is still flying straight. If all connections are perfect, then the version in the 'chairman' memory matches reality. However if one player has a poor connection the other players will mainly see the 'chairman' version of the bad client and the bad client will see the chairman's version of the other players (which will be largely correct as they have good connections). The net effect would be the experience would only degrade slightly for players with good connections, whilst the player with the bad connection might get a worse experience. This is an advance on the current situation where one bad client can ruin an instance.
:
I think this would only be a smallish architecture change (note might or might not be a small code change), to go from "poll each client, then update with their data" to "poll each client, if one is unresponsive, request and use chairman substitute data for them". The 'chairman' would also be useful for arbitrating any variations such as disputed weapon hits or near collisions
:
The problem with having a 'chairman' client is that it makes you venerable to exploits as you have to trust a client's data. To get around this the 'chairman' clients would not actually have a presence in the instance i.e. they would be a ghost client and only FD servers could be 'chairman' clients. Essentially FD would provide "ghost" clients to improve the performance of NY instances they deemed required it. These rules could be simply locational (e.g. around a particular station) or could be triggers by events or simply by poor experience metrics from a particular island.
​​
 
Nothing will drive me from open faster than a switch to a central server approach (except maybe hackers I suppose). Fine if you live relatively close to the server location, but unless Frontier would be prepared to host multiple servers in varied locations in multiple countries, a fully server-based approach will disadvantage those who live furthest away, due to latency. Against NPCs that ain't so bad, but terrible in multiplayer. From where I sit, P2P matchmaking is the better option as I'm in Australa - better than the latency I'd have to put up with connecting to a central server in the UK (sheesh, playing on a US-based multiplayer server is bad enough in most games, particularly with high player numbers). Some people need to realise it's not all about them, but rather trying to compromise so that everyone has a chance of good connectivity with the players they meet. P2P ain't perfect but I'll take it over a UK server-based approach thanks (unless Frontier put a server in Sydney lol).
 
Last edited:
Nothing will drive me from open faster than a switch to a central server approach (except maybe hackers I suppose). Fine if you live relatively close to the server location, but unless Frontier would be prepared to host multiple servers in varied locations in multiple countries, a fully server-based approach will disadvantage those who live furthest away, due to latency. Against NPCs that ain't so bad, but terrible in multiplayer. From where I sit, P2P matchmaking is the better option as I'm in Australa - better than the latency I'd have to put up with connecting to a central server in the UK (sheesh, playing on a US-based multiplayer server is bad enough in most games, particularly with high player numbers). Some people need to realise it's not all about them, but rather trying to compromise so that everyone has a chance of good connectivity with the players they meet. P2P ain't perfect but I'll take it over a UK server-based approach thanks (unless Frontier put a server in Sydney lol).

This, exactly.

And, specifically, having to choose between...

1. World of Tanks "US East" server (better population, 230ms latency)
2. WoT "US West" server (long matchmaking waits, 150ms latency)
3. WoT "South East Asia" server (one-way permanent migration, full-potato teams, difficulties encountered communicating with players in English, 70ms latency)


Frontier's solution ain't so bad. Works well for me.
 
Lots of people here cutting FD a lot of slack and saying C/S would require a monthly fee.

Thats kinda laughable. Living under an online rock last 15 years are we? The only thing it would require is some competence and a lower h&b budget.

Rather than question posters' knowlege of online gaming, why not detail how FD should generate the revenue for for servers placed around the world and high speed connections between them? We can't have time dialiation like in EvE, so they have to work great even with high load. Remember, with C/S every laser shot, every adjustment of thrust and direction has to be arbitrated by the server and communicated to every client in the instance. There would be no "lower h&b budget".

Hmm.. so, no subscription? Then i guess its either increased prices of expansions, or, pay to win store. Unless you can explain how FD can do this on the cheap.

PS: Please be sure to send FD your technical and business plan for how they can do it. I'm sure they will be interested.
 
Last edited:
one major flaw about this P2P system is the ease of cheating, keeping important stuff serverside on a client/server structure just eliminates this almost entirely, now people just have to circumvent the watchdog (which isnt hard) and they can basically alter whatever they want, which nobody will see them do so if they remain solo, they can go open as well if they just turn off the scripts and nobody will suspect them of cheating
 
Agreed? Disagreed? Bananas?

Totally agree. That's also my biggest negative point about Elite. There's no player interaction. The entire game feels empty and it's pretty boring to play against NPCs. It's just not the same as playing online with real players and interact with them.
 
one major flaw about this P2P system is the ease of cheating, keeping important stuff serverside on a client/server structure just eliminates this almost entirely, now people just have to circumvent the watchdog (which isnt hard) and they can basically alter whatever they want, which nobody will see them do so if they remain solo, they can go open as well if they just turn off the scripts and nobody will suspect them of cheating

And apparently a new version of the cheat will be out today. But its taken them a couple of weeks to get it out. And we have a new patch coming next week. Let's hope FD changes something again so it borkes the script again.

They can't win against the hackers, but they can make their life annoying.

- - - Updated - - -

There's no player interaction.

:O

Are you playing in Solo by any chance?
 
Status
Thread Closed: Not open for further replies.
Back
Top Bottom