Proposal Discussion Concern about cheating and P2P handling of NPC simulation

[Dejay]

I was curious about the network architecture and who determines the behavior of the NPCs. Someone recently wrote that the "Zombie AI" bug in the alpha is due to individual PCs crashing and the NPC AI is left hanging.

Is the NPC AI running on a player client computer?
If so, it would open the door to all kinds of cheating.

It would be rather easy to hack the client or the network traffic and cause all sorts of cheats in the MMO universe. I realize that running the AI on a client computer is a great way to reduce the CPU load on the ED servers, since then the server only has to handle sessions and not the NPC simulation. But with a client hack someone could boost his elite rating, have NPC just drop their cargo or even attack other player ships without any cause. Or even cheat in a private group and bring the winnings to the all group. A couple of questions:

Will the network traffic and NPC behavior be monitored by the server to make sure people aren't cheating? Some kind of spot checks of the server?
How will you catch a cheater if he only cheats if he is alone in a star system?
Will FD ban clients quickly from playing online if they are suspected of cheating?
Can the client be made sufficiently "cheating proof" like other games have tried? I know that bots in WoW are undetectable in e.g. OSX.
Could all AI simulations be "mirrored" and checked on another random player client so that cheating will be spotted? (Similar to HAL in 2001, this would not cause more server load)
What other anti-cheat systems could be used?

Here is the link where someone replied to a questions about the Zombie AI and mentioned that Micheal explained the bug. If the support teams feel it's better to discuss this in the closed alpha forum, feel free to move it even though I don't have access. But I'd guess that "professional gold farmers" will pick up on this pretty quick anyway, especially since ED allows selling credits. Of course there are other ways to cheat with bots anyways to create money (e.g. automated exploring) but hacking the NPC AI would open the door to annoy other players a great deal.

 

[Razorwire]

...since ED allows selling credits...

E-D sell credits, not sure they allow selling credits by others.
That would leave them open to undercutting and revenue loss.

 

[Cmdr Eagleboy]

I don't have exact quotes at my fingertips (have to dig into Alpha forum archive), but sounded that FD is quite aware about potential exploits, however they really want to hear new variants how game could be hacked so they can plug it before general release

 

[Cmdr Snuffler]

I don't have exact quotes at my fingertips (have to dig into Alpha forum archive), but sounded that FD is quite aware about potential exploits, however they really want to hear new variants how game could be hacked so they can plug it before general release

I suggest that if anyone does find any exploits/cheats then they quietly raise a ticket with ED and do not post it in public forums. I have already raised a tiny exploit.

 

[Dejay]

I'm not sure if it's that easy to plug since it's a rather fundamental vulnerability. It's like a bank allowing the client to edit his own banking account A hacker planning to cheat would also not reveal his exploit now or in the beta.

The only idea I have is to have a "referee" to check if the output of a player is correct or is cheating. In most MMO's the server is doing that, checking if the player position doesn't jump around or is moving too fast etc. If they want to keep server load light, a random player client could be assigned as a "referee" to check and then signal the server if he detects or suspects cheating. But it's a very complicated systems

 

[KING5TON]

Isn't it a consequence of having a Peer to Peer system?
If it's true Peer to Peer and there isn't one true host computer and a number of clients but everyone is kinda the host then there are consequences when one player disconnects.
If it's Peer to Peer as I've seen with Console games last gen then one person is the host and everyone else are clients then you'd only get this type of issue if the host disconnects.

Regards cheating it's up to FD to put the controls in place that either prevent or detect cheating. That could be stopping actions that shouldn't be possible in game (e.g. going from one end of the galaxy to the other in just one jump) or monitoring the interchanges between connected players and banning people that cheat in some way.

 

[psykokow]

Regards cheating it's up to FD to put the controls in place that either prevent or detect cheating. That could be stopping actions that shouldn't be possible in game (e.g. going from one end of the galaxy to the other in just one jump) or monitoring the interchanges between connected players and banning people that cheat in some way.

Punish them by making them play Wing commander 3 Or I-War (sorry Colin)

 

[Dejay]

What wasn't raised in your post was the very basic issue of players knowing one another's IP addresses and so being able to use out-of-game attack vectors directly against other players (would assume most easy way would be to spam out of date real data - causing confusion over position of ships etc in addition to the potential DOS/DDOS).

I think this cannot be helped. Many online games with p2p udp traffic have this, is it a big problem there?
Generally only the port used by ED would be accessible from the outside. DOS would always be an option, but the ED network code could simply disregard packets from someone the client has "banned". That still enables a slow down or clogging up the users internet connection by DDOS but that costs resources and is potentially even involves cyber crimes.

What I mean is that detection of hacking / cheating is important. As soon as your client knows he can report abuse and as soon as a central server has verified it he can be banned from online play.

The root of the vulnerability of such systems is that clients report the results of their actions rather submitting a request to carry out an activity. So the key stage of an authoritative server determining whether an action is valid is missing.

Double authority is a bad idea - what happens if 2 conflicting sets of data are received (and most data-set pairs wouldn't exactly match anyway)? 3 is the minimum for such a system to work if there's no actual unbiased (and reliable) authoritative source. And how does that work for people playing single-player online? Right now they're effectively their own server (same as any instance with only 1 player in). Should they also be running duplicate servers for 1 or 2 other players? If so then that removes (and maybe even worse) any speed benefit caused by the use of PTP rather than a central server. i.e. from a player perspective it no longer offers any of the benefits of PTP - as they can't do anything without all actions being processed by someone else. Not to mention the point of how do you ensure a client that is trying to cheat actually listens to another player acting as a check on them (bear in mind they can just IP-block that other player at their firewall - and cheating would be done by sending results to the central server anyway, so what actually happens in their client isn't that important.).

Well the first step would be detecting shenanigans. Then someone would have to verify. After that a client can simply be banned.

I don't know how crypto-currency works exactly, but I guess the same principles could be applicable. You have transactions like movement, shots, lots of player actions. The client normally just simulates his instance and executes all actions, but those actions need to be verified somehow. A kind of "post action verification". Instead of a central server any random player client could do it, and if shenanigans are detected then it is reported and the transaction log is reported as abuse and rechecked by a central FD server. The presences of a central authority should make it easier to implement that the fully distributed bitcoin.

So you fly around, mind your own business and send a transaction log to one or more "verifies" asyncronously. If they yell shenanigans they report cheating and the server investigates, banning if true. If not, they send you a cryptographic signed "clean bill of anti-cheating" that your actions haven't violated any rules and you can go on.


If you google for "distributed game server verify" you find some papers but I haven't read them. Hopefully someone at FD did read them though

Of course all speculation but I like to speculate Somewhat related to this, I also thought about how persistence could be implemented distributed and wrote some stuff in the mining DDF.


BTW, How would you have titled the post?

 

[Jenner]

Punish them by making them play Wing commander 3 Or I-War (sorry Colin)

Hey, Wing 3 was good!

 

[Asp Explorer]

The problem with banning clients based on network data is fraught with issues. Traffic can be spoofed, changed en-route, or a machine compromised and set to send crap without the users knowledge.

There are ways around it though, and I'm interested to see how FD implement this

 

[huwthomas]

I was curious about the network architecture and who determines the behavior of the NPCs. Someone recently wrote that the "Zombie AI" bug in the alpha is due to individual PCs crashing and the NPC AI is left hanging.

Is the NPC AI running on a player client computer?
If so, it would open the door to all kinds of cheating.

Maybe the same NPCs will run on multiple peers?

As long as the peers have the same random seeds, and can see the same actions from other peers, they should be able to create duplicate NPC actions, allowing validation to be done.

Of course the communication of player actions needs to be very robust for that to work. So in practice ... who knows?

 

[Ian Doncaster]

As long as the peers have the same random seeds, and can see the same actions from other peers, they should be able to create duplicate NPC actions, allowing validation to be done.

I don't think that can work here - because the simulation tick has to run considerably faster than the network can transfer data for smooth visuals, the simulation of the NPC will inevitably be working from different inputs on each player's computer, especially if there's above-average network delays or some packet loss, so you'd expect divergence.

You could do some validation on "is this change in state plausible" e.g. if a client reported at one point that a ship was in a particular position, and its next report placed it at a position too far away for it to have reached, something would be up.

 

[wstephenson]

[ Average Speed Check ]

You could do some validation on "is this change in state plausible" e.g. if a client reported at one point that a ship was in a particular position, and its next report placed it at a position too far away for it to have reached, something would be up.

Maybe float big 'Average speed check' signs just outside every Coriolis and apply automatic penalty points and fines to naughty Commanders too.

 

[neilisdead]

Is there anything we can read about their network implementation... I work in secure public safety telecoms, sure I can get some of our architects to skim over a spec and look at exploits.

 

[Dejay]

Traffic can be spoofed, changed en-route, or a machine compromised and set to send crap without the users knowledge.

That's actually easy to fix, you just sign / encrypt with standard SSL stuff. It's easy to do with standard library.

Maybe the same NPCs will run on multiple peers?

Yeah, that's what I meant. But you'd have to send much more actually. You'd have to transmit your current session state (basically your random seed plus persistent changes you've made, kinda like upload your savegame to other client) and then the other client can verify if your actions sound bogus.

Uhoh... is this why the upload rate is so high atm? If the NPC logic runs on the clients, all updates to the NPC's needs to be transmitted to other players even without cross checking. Of course they don't update that often and are more predictable.

I don't think that can work here - because the simulation tick has to run considerably faster than the network can transfer data for smooth visuals, the simulation of the NPC will inevitably be working from different inputs on each player's computer, especially if there's above-average network delays or some packet loss, so you'd expect divergence.

You could do some validation on "is this change in state plausible" e.g. if a client reported at one point that a ship was in a particular position, and its next report placed it at a position too far away for it to have reached, something would be up.

The other client checking if you're cheating would basically have another instance of your game session running, just delayed. So it you can pack and compress larger packets and send them delayed via TCP (no packet loss), just for validation. Of course I don't know if that will work either though I just now that simulating NPC behaviour on clients sounds like a pretty hard to fix security whole for cheaters to abuse.

 

[KickahasTier]

Is there any more information than this?

Someone recently wrote that the "Zombie AI" bug in the alpha is due to individual PCs crashing and the NPC AI is left hanging.

Does this really mean instances are peer to peer based and not server managed?

 

[Cmdr Eagleboy]

Is there any more information than this?



Does this really mean instances are peer to peer based and not server managed?

Instaces are peer to peer based. Zombie NPCs *likely* (speculation warning) are due of not transferred "authority" to other hosts when someone crashes or leaves session improperly. So other hosts know there's NPC there, but no one takes care of it - t.i. doesn't calculate it's behavior.

 

[KickahasTier]

Instaces are peer to peer based. Zombie NPCs *likely* (speculation warning) are due of not transferred "authority" to other hosts when someone crashes or leaves session improperly. So other hosts know there's NPC there, but no one takes care of it - t.i. doesn't calculate it's behavior.

Instances are peer to peer based!

Well ok then, there must be some pretty interesting distributed processing going on.

If we are speculating that a single master does the NPC processing causing this zombie. Maybe they should consider a shared mastering over multiple clients.

They could be doing something like state machine replication using Paxos


Now that would be cool

 

[pompomJuice]

If only you guys knew..

That is all I am going to say about it.

Cheers.

 

[pompomJuice]

If only you guys knew..

That is all I am going to say about it.

Cheers.

Actually that is a bit harsh.

The game is in early alpha. We know this. But we don't think about the consequences. Some of the issues raised here are still to be addressed by FD.

That being said. From what I can tell 90% of what has been delivered works and clearly shows these guys know what they are doing. Unfortunately the last 10% is deadly.

It's going to take more time.

Lots more time.