I'd take a guess that Ci¬G had already done a fair bit of the donkey work before Turbulent took it over, I think they're closer to making it work than 18 months. I'd optimistically say they'll have a static server mesh up and running in some form by years end...full dynamic system by Q2 next year. It all depends on how cooperative Ci¬G are now that Turbulent have taken it on.
Game Discussions Star Citizen Discussion Thread v12
- Thread starter Yaffle
- Start date
-
- Tags
- star citizen
Continued.
- The presenter in glasses said "shards" and "instances" were different, claimed he would explain the difference, then failed to do it and segued to the second presenter *
He mentioned the graph database storing the state of a single shard that is seeded when a new shard is created. That would mean no communication between shards, apart from their version of background simulation (economy, reputation etc.). Just like it is done by ED, I think.
- It is not clear if players can hop shards. IMHO they cannot. Once in a shard, always in the shard, unless they are able to extract part of a shard data relevant to a player and transfer it to another graph database responsible for another shard. This could be tricky. Not technically, but game-logic wise. I won't speculate more on this topic, watching further.
- "Entity graph" as an "evolution of iCache". Not sure what it means. iCache was supposed to be a cashing layer speeding up queries, not the central store for the global state. Unless it begun as a small indexing structure and then overtook more and more information about the game world. Nobody knows what it was anyway.
- Graph databases responsible for shards are also "sharded". This has nothing to do with the sharding of the game world. In database systems "sharding" means a database itself is distributed between multiple nodes, but looks like a single entity from the outside. If they are using a distributed graph database system, the list of candidates could be googled by someone who knows more about this topic. I only know that they exist
- The second presenter seems to be right about a graph db being a good match for their use case. It makes sense to me, intuitively. What he says about multi-mutation transactions (all changes succeed or all fail) also makes sense but is very basic. Generally he explains those details very well, but at the same time what he is saying has nothing to do with the viability of the entire solution and its scalability. He explains common sense practices, nothing groundbreaking. Kudos for clean delivery.
- Another confirmation of Kafka (or something similar delivered by AWS) being used for the "replication layer" can be found when he is talking about a "persistent queue". This is exactly it - every consumer is "subscribed" to topics they are interested in and they can resume consumption at any point in the stream, even independently replaying parts of it to catch up in case of failure. Again, common sense in similar systems.
- More on seeding of a shard graph database. He mentioned that the seeding also is done by the replication layer. There must be a "facsimile" of a fresh shard data stored somewhere. Intuitively, this means that the facsimile is stored in cheaper permanent storage as, you guessed it, a series of ordered messages. Seeding means that these messages are "replayed" in order to the newly created shard database. If they want to modify the facsimile (because they added something to the world), they add more messages to it. A very clean pattern. And yeah, this is how it has been done for years.
- Correction on the shard assignment. There is logic to shard assignment and they have a way of transferring a player between shards, through the central uber-graph and "stowed" entities. Again, this raises interesting issues about the game's logic because it allows for items to be removed from one shard and appear in another. Say "bye bye" to the coffee cup you left on Magda only to return to it a week later. You will be assigned to another shard in which it will not exist anymore. Another player will be able to find it, though. Mission partially accomplished, I guess.
- All the issues of "dense" situations, server authority etc remain. Sharding will help when players are distributed over a larger space but will not do much for dense situations, like a close-distance space battle. There is no mention of the "old" dynamic server meshing (distributed octree as it is in the Aether Engine) that could help in those cases.
- I think it means that @WotGTheAgent was right. This is it, there won't be much more. Instead of trying to implement dreams.txt without realising they had no viable way of doing it, CIG finally reached for common sense and clean, well established patterns in distributed data systems. With all their limitations. They will push for larger shards as far as they can and call it a day.
The sad thing? They could have implemented what they are showing on this video years ago.
* Or not. He actually meant that "instance" was the same as an "AWS instance". It contains all the data for a game session and when it dies, its 50-player universe dies with it. This is not the same as what "instance" usually means in MMOs. Depending on the source, instances and shards will mean more or less the same - a separate copy of the game world, or part of it, that contains its own, independent state and a limited number of players who can interact with each other.
- The presenter in glasses said "shards" and "instances" were different, claimed he would explain the difference, then failed to do it and segued to the second presenter *
He mentioned the graph database storing the state of a single shard that is seeded when a new shard is created. That would mean no communication between shards, apart from their version of background simulation (economy, reputation etc.). Just like it is done by ED, I think.- It is not clear if players can hop shards. IMHO they cannot. Once in a shard, always in the shard, unless they are able to extract part of a shard data relevant to a player and transfer it to another graph database responsible for another shard. This could be tricky. Not technically, but game-logic wise. I won't speculate more on this topic, watching further.
- "Entity graph" as an "evolution of iCache". Not sure what it means. iCache was supposed to be a cashing layer speeding up queries, not the central store for the global state. Unless it begun as a small indexing structure and then overtook more and more information about the game world. Nobody knows what it was anyway.
- Graph databases responsible for shards are also "sharded". This has nothing to do with the sharding of the game world. In database systems "sharding" means a database itself is distributed between multiple nodes, but looks like a single entity from the outside. If they are using a distributed graph database system, the list of candidates could be googled by someone who knows more about this topic. I only know that they exist
- The second presenter seems to be right about a graph db being a good match for their use case. It makes sense to me, intuitively. What he says about multi-mutation transactions (all changes succeed or all fail) also makes sense but is very basic. Generally he explains those details very well, but at the same time what he is saying has nothing to do with the viability of the entire solution and its scalability. He explains common sense practices, nothing groundbreaking. Kudos for clean delivery.
- Another confirmation of Kafka (or something similar delivered by AWS) being used for the "replication layer" can be found when he is talking about a "persistent queue". This is exactly it - every consumer is "subscribed" to topics they are interested in and they can resume consumption at any point in the stream, even independently replaying parts of it to catch up in case of failure. Again, common sense in similar systems.
- More on seeding of a shard graph database. He mentioned that the seeding also is done by the replication layer. There must be a "facsimile" of a fresh shard data stored somewhere. Intuitively, this means that the facsimile is stored in cheaper permanent storage as, you guessed it, a series of ordered messages. Seeding means that these messages are "replayed" in order to the newly created shard database. If they want to modify the facsimile (because they added something to the world), they add more messages to it. A very clean pattern. And yeah, this is how it has been done for years.
- Correction on the shard assignment. There is logic to shard assignment and they have a way of transferring a player between shards, through the central uber-graph and "stowed" entities. Again, this raises interesting issues about the game's logic because it allows for items to be removed from one shard and appear in another. Say "bye bye" to the coffee cup you left on Magda only to return to it a week later. You will be assigned to another shard in which it will not exist anymore. Another player will be able to find it, though. Mission partially accomplished, I guess.
- All the issues of "dense" situations, server authority etc remain. Sharding will help when players are distributed over a larger space but will not do much for dense situations, like a close-distance space battle. There is no mention of the "old" dynamic server meshing (distributed octree as it is in the Aether Engine) that could help in those cases.
- I think it means that @WotGTheAgent was right. This is it, there won't be much more. Instead of trying to implement dreams.txt without realising they had no viable way of doing it, CIG finally reached for common sense and clean, well established patterns in distributed data systems. With all their limitations. They will push for larger shards as far as they can and call it a day.
The sad thing? They could have implemented what they are showing on this video years ago.
* Or not. He actually meant that "instance" was the same as an "AWS instance". It contains all the data for a game session and when it dies, its 50-player universe dies with it. This is not the same as what "instance" usually means in MMOs. Depending on the source, instances and shards will mean more or less the same - a separate copy of the game world, or part of it, that contains its own, independent state and a limited number of players who can interact with each other.
I'm optimistic that Turbulent can get this thing moving is all. They've been given the PU remit for a reason, so it's not just me that's being optimisticThat is definitely optimistic
JFC, Turbulent is doing locations and core tech now. Chris is being pushed out by the Calders. It is Freelancer all over again.I'm optimistic that Turbulent can get this thing moving is all. They've been given the PU remit for a reason, so it's not just me that's being optimistic
The coffee cup thing was only CR's silly dream...now that Ci¬G as a whole seem to be shedding themselves of his silliness, I'm reasonably confident that there may be some progress...with the PU...Sqn 42 doesn't concern me in the slightest, not only with Turbulent reverting to common sense and standard industry practises with regard to server architecture...but there seems to have been massive changes and attitude at Ci¬G over the last few months. It's almost like all the really talented people they do have have been given the freedom of agency to get stuff done with less micromanagement.
Don't quote me on it...it's just from what I'm seeing reading between the lines.
Last edited:
Clive always looked to me like he wasn't convinced by what he was saying. In one video I saw, he had the appearance of a hostage reading out his captors demands, whilst trying to pass a "help me" message by blinking in Morse code.
Waht my brain could understand from the server meshing presentarion:
- Single shard is gone
- Matchmaking is back (can I have my fking pvp slider back pls?)
- Replication layer sounds like an overly complicated session controller/load balancer
- they will use some kind of distributed tree-based database engine to hold all entities/objects that are active… and sent the inactive ones to a normal datastore (probably distributed as well)
LOL, i wish!
But one thing it opened my eyes up to.
1) Its actually quite tough work and not quick.
2) I'm a nobody and got a pretty packet for it. I can't imagine how much all those A list actors got paid per hour. They must have spent millions upon millions for them.
I didn't get much chance. Its hard to nip of for a pee when you are covered in sensors, wearing a helmet with cameras, and you can't even itch your nose because you might rub some dots off.
Kinda cute that the game seems to be drunk too around you...
(Bartenders merging, inevitable guy standing on seat, your av walking in mid-air for your mate
You're a real bore.
Read the first post and what came to my mind wasn't
But CIG put themselves into that situation.
Then i read the second post where they said
And i'd still say no. CIG put themselves into their situation and backers put themselves into their situation.
Everyone has to take personal responsibility for the situation they find themselves in.
Source: https://youtu.be/TSzUWl4r2rU
My Amateur Summary:
Old Server Meshing idea scrapped:
Because its crazy...
Replaced by the 'Replication Layer':.
A system which will magically connect a ton of clients, a ton of servers, and the persistence database etc. And which is not crazy at all. Oh no.
A single server will have authority for any given entity:.
Allocated on a first-come-first-served basis, and transmitting that simulation to other servers via the RL. Authority can migrate between servers depending on changing circumstances.
There will be multiple shards:
They've scrapped the single shard idea.
Bonus stuff:
- They'll start with 'static server meshing' (servers allocated to fixed locations, rather than mobile bubbles) because the authority transfer aspect is a nightmare.
- Many aspects of the game will need to be redesigned (missions, background services like 'Quantum' etc) to work with this new system, now that they've finally settled on a final approach. Kind of.
- Entity Graph is "an evolution of the original iCache".
- The broad beats all tally very nicely with claims by TheAgent's sources.
---
Introduction: [20s]
- OCS is a thing.
- Dynamic entities are nested. "Additionally to the static hierarchy of object containers there are also all the dynamic entities which bring the universe to life. NPCs, an interactive vending machine, and of course players and spaceships. Most of these entities are made of a hierarchy as well. For example a player has this body and undersuit and armor attached to it, and they are all child entities of the player. The streaming system treats these mini hierarchies as streaming groups to make sure that an object like a spaceship is always streamed in as one unit."
'Streaming and Replication': [2m50s]
- Streaming bubbles: "When a player connects we create a so-called streaming bubble around that player and object containers as well as streaming groups that are visible from the player's point of view are considered inside this bubble. Any object container that is inside the bubble will stream its content and any streaming group within the bubble will also be streamed in on the server and then replicated to the client."
- "Entities are considered inside the streaming bubble if their projected screen size on a virtual 1080p plane is larger than 5 pixels based on distance of the player. So while a large object like a moon will be considered inside the bubble from far away a small object like ship will only be considered inside when it's much closer to the player."
- "When the player starts moving across the universe entities that leave the streaming bubble will become unbound and the replication layer will remove these entities from the client. Entities that enter the streaming bubble will get bound to the client which caused the network layer to replicate these entities to the client, effectively streaming them in. We call this technique entity bind calling because streaming on the client is driven by the network layer binding and unbinding entities."
'Problems to Solve': [4m38s]
- "This model works quite well on the client however it doesn't scale well on the server... The more clients we try to match to given game server the likelihood of a player being at every single location increases and that basically nullifies the benefit of server-side streaming."
- "So how do we solve this? The answer should be simple. Allow multiple instances of the game server to work together so they can split up the work. Well it's not quite that simple."
'Architecture': [5m19s]
- Current dedicated game servers
'Replication Layer': [6m4s]
- "The goal of server meshing is to allow multiple DGS instances to work together and divide simulation cost between each server and the mesh. In the best case we can scale this to infinity by adding more nodes to the mesh."
- "If you want to mesh these servers together we need to find an efficient way to synchronize data between each server. With our current architecture, depending on the vision of the simulation, and the overlap this would require, a lot of synchronization points between each node. It's an exponential problem as in the worst case each node would need to talk to each other node in the mesh, severely limiting our ability to scale it."
[Otherwise known as the 'Chris is a Moron' slide
- "To solve this issue we are separating simulation and replication."
- "The replication layer has two major functions. It holds the state of every entity in memory, and replicates the state to clients but also to server nodes."
- "I said server nodes because in this setup the traditional dedicated game server becomes a game server node. This server node connects to the replication layer, very similar to a client, and only a subset of entities are replicated to that server node."
- "Replication to server nodes is controlled by the network bind culling algorithm that we saw earlier, and it's driven by streaming bubbles, and it works very similar to how it works on clients."
- "The server node has certain[?] streaming bubbles assigned to it which will cause the replication layer to replicate entities from these streaming bubbles to the server node."
- "Contrary to a player's client the server node has the additional responsibility to execute server-side authoritative code for those entities. Controlling AI, doing damage calculations, etc etc. The result of the simulation is then written back from the server node to the replication layer, and from there it is replicated to all connected clients and other server nodes."
- "Since streaming bubbles can overlap entities may be replicated to multiple server nodes, exactly the same way how they are currently replicated to multiple clients if players are at the same location. To avoid two server nodes trying to simulate the same entity only one server node can have authority over any given entity, and only that server is allowed to write entity state back to the replication layer. This is usually the first server node who replicated the entity and other server nodes will only run client code on those entities"
- "Authority can transfer between server nodes. For example if an entity leaves the streaming bubble of the current authoritative server it is then transferred to the next server node that has this entity currently streamed in.Further authority can be transferred between server nodes on demand in order to load balance the mesh."
'Shards & Persistent Streaming': [9m8s]
- "Since we now mesh multiple server instances together to simulate a shared state of the universe we no longer call this instance but instead we call it shard."
- "A shard is still a unique version of the universe,and we still have multiple shards running in parallel. However, the server mesh will lift our current hard limit of 50 players and it will enable us to steadily increase the number of players we can support within one shard."
- "There are some fundamental differences between a shard and an instance. And for this, we need to take a closer look at the replication layer and talk a little bit about persistent streaming." [Chin emote here...]
- NB there seems to be a replication layer for each shard, going by this graphic:
- "The entire state of the universe is stored in a graph database. We call this entity graph and it's an evolution of the original iCache."
'Persistence': [11m13s]
- Benoit on the graph format for storing data in the entity graph etc. Blah blah.
- "Our objective is to be able to save the state of the replication layer, which includes all entities in a given universe shard,in order to provide a truly persistent world, where actions you take as a player can influence environments in the game world, permanently."
- "Each of these entity nodes holds properties with regard to what the entity represents in the game. That class of object it is, the item type, its legal owner, orientation, and of course, the very precise physical location within the game world."
- Much blah about how the graph system super much better than columns. Easy to serialise & transfers, data won't be lost etc
- "The system retrieves a constant ordered streams of mutation from the replicant scribes that are part of the replication layer and are enqueued in durable queues, to ensure that no message is lost even if the service is unavailable or paused."
'Seeding': [19m10s]
- So the Replication Layer [RL] starts with every entity loaded into it in their default / 'initial' state. (Universe / solar system / planets / station etc. From unstreamable conceptual entities, to static stations, to dynamic items like doors, ship components, bottles etc)
- "As you play the game and go about with your ship,your playing character entity moves from location to location, getting attached to new zones as you travel. Your player aggregate is itself part of the entity graph, and your location and state are persisted by the replication layer scribes to the entity graph of your given territory."
- "When you interact with dynamic objects and their properties change, the state of that entity will not persist until this instance of the database is undeployed." [So dynamic objects only saved in place periodically etc?]
- "There are in fact multiple copies of the universe that are seeded at a given time. We call those shards. Each shard is a unique copy of the game world,complete with all of its entities and unique states. Think of it as an alternate universe." [Oh well, so much for Clive's single shard universe]
- "Dynamic entities that have been modified in each dimension will have different states. The bottle on the bar was moved or the door was destroyed, might not be in the same state between shards."
- A way to gain scalability as playerbase grows. "Even if each shard database is itself clustered and can grow substantially past a single machine there is a point where multiple clusters are needed." [IE one version can't hold all the entities, seems to be the admission , and/or handle all the info networked between multiple machines, once it grows beyond a certain capacity?]
- "In order to select the correct universe shard for you to play on using multiple data points like your friends location your active party your last game session and or which shards still have items on it that you own this is to ensure as much as possible that you end up on the same shard you expect to be as a player in order to provide a seamless game experience."-"It would be terrible if you lost items you used when you were in a given shard versus another or if your character was bound to a shard forever. To alleviate this the system includes the concept of stowed and unstowed entities."
'Benefits and Challenges' [25m39s]
- Synchronisation issue dumped: Don't have the issue of synchronisation between servers [in the old plan etc]."Each server node has one single connection to the replication layer which is used to push and get updates for entities of interest."
- Servers like current clients: "The second advantage is that the same streaming and replication logic that we already use for clients can be applied to servers and that server nodes will only stream in a small area which will greatly increase performance"
- 'Static to start...': "The first version of this technology will contain a static server mesh instead of the fully dynamic mesh that we saw earlier the static mesh assigns server nodes to predefined sections of the solar system this will reduce the amount of authority transfer that game code has to address in this first release." [GULPS]
- Loads of current stuff will need redesign: "So you can imagine that this will affect things like missions that currently are spawned locally on the game server. These now need to be spawned globally within the chart and also persist their state. So all services that are attached to missions where, whether it's the Quantum system in the back end, or the Quasar tools, need to know know about the concept of a shard. This also goes deep into like things that are mechanical, like you know getting global chat to work on a server. That concept now needs to be extended to the shard where this will probably push us to implement this as a location-based chat for example. And so many teams in the company now need to change their feature to take into account the meshing technology that's behind it."
---
SUMMARY OF SORTS:
So the Replication Layer...
- Is connected to every player client in that shard.
- Is connected to every single server in that shard.
- Is connected to the entity graph data store etc. (Of the precise location, orientation etc of all of the things...)
Conclusion: The replication layer is the new promised land.
Hold up...
LOL, they did it again didn't they? A magical tech didn't live up to the hype, so they create a new tech on which backers can pin their hopes.
I must admit, i admire their round thingys.... big, bold, brass, round thingys.
I'm having a back and forth with one of the faithful. But it didn't take long before their attempts at debate were given up in favour of direct insults.
All i did was say that there was no release date for SC, SQ42, or ToW.... and in return they accused me of trolling and gaslighting.
I'm flabbergasted, i mean, that's direct from CIG. They have no dates for any of those. How can it be trolling or gaslighting? Its fact.
I sometimes do wonder if these accounts are actually chatbots that are programmed to defend CIG. But then i remember, bots would run off some sort of logic, and these accounts seem bereft of it.
Can confirm, working hard on that right now. Don't even need Star Citizen for it..
Kotaku | Gaming Reviews, News, Tips and More.
Gaming Reviews, News, Tips and More.
www.kotaku.com.au
A more positive (or less negative) take on the con.
With special guest star Joe Blobers in the comments.
With special guest star Joe Blobers in the comments.
Yeah sounds like the coffee cup has to pick its forest...
This is my take away too. It sounds like they've settled on something that isn't designed to meet the absurder 'Server Meshing' targets of 1000+ player concurrency etc. It's just designed to work at a basic level. And have a chance of scaling up to slightly chunky numbers. (So they'll get some server performance & player cap improvements by handling smaller regions per server, and try and ramp up the number of clients they can get in per shard etc. But they're not expecting to get good performance in heavy scenarios via combined servers or anything. IE no real expectations of giant 200+ player battles where it's all running reliably).
Will be interesting to see if the tail end of the Agent gossip pans out too there
TheAgent said: