6m30s - Two schools of thought on live demos, and they're both correct:
- The real thing that's indistinguishable from magic is a rigged demo. (Riff on the Arthur C Clarke quote)
- Don't do live demos because they always fail. CIG seem to have good luck on this front. (Which may speak to how rigged they are...)
(7m35s - By 'live' here its running off his laptop / a server under the lectern / back in the production booth etc. Not 'live live' on the PU style etc.)
12m - Strength of graph database is that it's 'an atomic operation'. Indivisible. You cannot get stuck in an intermediate state to move the edge of a parent node to another parent node. (So is it poss that a starship could get stuck on both sides of a star gate? Hopefully not... Parent node of your ship is just one node - should be simple to swap the parent node from one node to another. In an ideal world).
- Interesting thing here is it's being used for arbitrary areas of space. (Generally it's unlikely that Microtech would interact with Hurston. Only way you can interact is to go, with a lot of delay). But here they're using it for small volumes, which share surface area etc.
16m - Notes the transition points, and the dog-leg shape of the demo map. Hard to go from red to purple quickly etc. suspects relevant due to server tick rate. (Think they've made this choice to limit how much things can interact if they're not in adjacent regions. Really have to spend some time in the middle region for a while before you can go to next region, he suspects. Very hard to move fast enough that you can get to third region before server is up to date etc).
18m15s - Global graph database. (Re base vid: He doesn't know anyone doing that for transactional / main source of truth database etc. He thought naively RL was just to store properties (key value pairs) of nodes etc.
19m20s - Graph database is designed and optimised for really fast queries of this type: How do I get from point A to point B? What's the shortest path from point A to point B? How many inbound edges do this node share with the outbound edges of this node. How many outbound edges do these two nodes share? (Can have loops in the graph, which complicates).
20m45s - Realises now the RL is a 'write replica' of the main graph database. Really just letting them have multiple copies of the graph database, cached closer to the game servers. It's a way they solve problems too. Can also have 'read replicas'. Depends if read or write heavy workload etc. As per
his talk with Zak Johnson of Blizzard - this is a write-heavy circumstance. Doing all these writes to a graph database local to the game servers (RL), which will then get eventually persisted back to the global database (entity graph). RL has the full database, or at least the bits relevant to this game server. Related players, ships, cities, planets etc. Again, do this with table databases,
but still doesn't know anyone using a graph database this way. 'That's a brave choice'.
24m15s - Doesn't think the client is running neo4j (the graph database)
. When he says replicated on the client, he means replicated in the graph structure it holds in memory. Has no persistent storage then. (As far as he understands it game engines 'think' in graph structure. Would make sense that then there'd be little 'translation'. But graphs are really hard for computers - cache locality being an issue. CIG have chosen not to use tables to work around. Again, nobody has tried to make this choice before.)
26m05s - Game spawns Picos, they go into server memory (in this case the RL of the server memory)
- As he understands it if/when this instance of the game server gets destroyed, the server will persist its state back to the Entity Graph.
He sees this as a bold move 
. If the game server crash corrupts the RL, then the info can't be persisted back, then the plushies that you've spawned are gone etc. [So 30ks etc still an issue potentially etc]
27m30s - Zone transition with Picos. 'This is the hard part'. He correctly notes that this is all on one server at the mo though, so he's not clear on authority etc. But re the theory main issues would be: the fundamental issue of distributed systems is the 'the distributed lock problem'.
- Say you have 3 servers, a distributed file system, 'distributed lock service' ensures only one server can ever have the file open to write. The other servers check the DLS to see if they can or can't write. It's a hard problem. (It was the first third of his opening course. Foundational etc. At the time an algo called Paxos was used.)
- In the Blizzard vid discusses how distrib system broke so that no one could get a 'lock'. [IE no server could write to entity?]. Whole system ground to a halt. Which is preferable to the alternative, of all servers being able to write to same entities [?]. Grinds to a halt in a much more 'firework-y' way.
32m20s - His face as they list the number of entities in PU
. 'Not actually a large graph, but a large graph to be dealing with in real time in this kind of fashion'. The cadence of updates is what matters here.
34m - Streaming out. They're doing what you might call 'garbage collection' on the server memory. No idea how densely packed they intend the server zones to be. Important to be able to shut down servers with no clients though. Would be interested to know the memory 'pressure' on both the server (physics) & RL. (Can always toss 100g memory into service etc). Infers there's a limitation on memory though. (Very common generally on client side.)
36m - (He's unclear on where it re-stablished server info gets streamed back in from)
(37m+ the RL service now running. Client and server 'now not directly connected'.). This is the 'write cache' he was talking about.
38m40s - The '30k' killed server. Notes that the frozen player etc = not great in PvP. But still better than a 30k crash.
39m27s - The servers are responsible for 'updating the world state as represented by the RL'. [When the server comes back, the simulation continues]
41m30s - Just talking about the tool running the stack on this PC. It's as he expected re running off local machine etc. Tool takes a while.
'Just think how slow it'll be when they get a hundred people in it'
42m15s - 3 server demo. 'Very first version of working server mesh' etc. So he sees it as: Three game server prcosses. Each one has authority over a zone. Some process for handing over authority. He guesses that they're co-ordinating the shared / distributed physics simulation via the RL. So it's a 'centralised lock server'. Rather than a distributed one.
43m45s -
CIG start to explain: 'How the magic works. Because it really is magic'. OP intercedes: 'Or a faked demo '
44m30s+ - CIG: Anything with authority on green server is being simulated by green server, but just being replicated by RL on others. KR: Because everything is shared through the central RL. Presumably each of these servers is 'subscribing to updates' from those zones via the RL.
45m30s - Authority transfer: There's some mechanism in the server code to understand transitions. The RL is overseeing, but the server still has to understand 'at this tick, this entity is transitioning'. He wonders if they control when it's fully out, when it begins to leave etc.
46m35s - Authority transfer: Both game servers had the requisite data (via RL). Server 1 probably communicating / confirming the handover etc with Server 2. Sees it as a three phase handoff.
'By no means impossible, or particularly novel. But tricky. Important to get right. Interesting.'
47m45s -
Because this is all local, that's why it's smooth / seamless etc. If this was all running off servers & services in cloud, the latency for these handovers is important. Both the game server process and the RL processes need to be tightly co-located. Entirely possible that they'll be on a single physical machine [and that is the plan]. And so could achieve something similar to laptop-level latency. But 'this is one of the rigged parts of the rigged demo. The reason he doesn't experience substantial latency is because these things are not talking through a wire, y'know, and network cards, to physical computers. They're passing packets around inside the memory of a computer.'
50m - The 'shooting across servers' bit. This is what he found most impressive. [NB so def his second viewing].
Doesn't know if this works because 'bullets are physicalised (doesn't see an authority box around them etc - although poss not drawn etc)'. Can't see the graph database to be sure. (Spawning them all as entities would be 'challenging' at scale). Not impossible again. And hasn't looked at how the netcode works.But still, 'this is the magic'.
52m - Abil to get in buggy and be combo entity = strength of the graph system etc. The buggy moves to a new server, the player moves too. (The reason it's not 'completely ridiculous' for CIG to use graph databases
)
53m35s+ Wonders now if the dog-leg shape might be because the entities aren't loaded in, not server tick. But then remembers they should be - says scratch that.
54m40s - Flips back. Imagine if it was 3 servers in a straight line. A bullet from red server to purple server. Green server would have items streamed in, but would not be authoritative for the interaction between the bullet and the target object. Purple server doesn't have the object streamed in? Although the player can see it. [He's confused how that works]
56m30s - Thinks RL is re-streaming in the 'garbage collected' entities that have been streamed out etc - but that servers are memory constrained. So the RL will need to have a lot of memory. But the servers can be CPU heavy but don't need a ton of memory. That's probably the trade off.
58m35s - Latency between servers matters, but careful server design can help. (IE all running same code, only communicating via RL). Not dealing with scenario where people will be running older versions etc. They'll all make comparable changes to physics of an entity etc. Just need to make sure the authority is right. (And they don't have to wait for the info of the authority object as it transfers etc, they already have it via RL.
Although feels the 'atomics' / 'making that atomic' is not trivial.)
1h02m30s - It's a big deal. He's very impressed. [
Talks about fan's 'pathological willingness to continue believing in CR' all the wild eyed optimism and holding the line etc. AAA or smaller publisher wouldn't / couldn't have pursued etc
]
1h05m00s - Goes back to the shooting across boundaries:
- If you shoot someone, who decides if you got shot or not. 'The shooter calls the shot'. They determine. Blizzard has a presentation on this re Overwatch. IE as the sniper, you've got the perfect position, tank waltzes into your sights, you're zeroed on his head, you fire and no kill. That's a ty feeling. So in online gaming, the server is authorative, but it's designed so that unless the client is doing something unusual, the client gets to decide. So server agrees that shot connected. (Ideally the target client agrees).
- There's a distributed systems lesson there. Friend of his: Jake Beale [sp] was a MIT PhD candidate who LARPed. They played with the rule that 'the shooter calls the shot'.
- He's sure that the purple server, with the purple shooter, is authorative over whether the green buggy gets shot.
robertsspaceindustries.com
