Someone care to explain why the server updating takes so long (whole day)?
- Thread starter The Law
- Start date
I'm sorry...all I can see is that man's giant balls.
Are your family looking at you like some weirdo they've not seen for a couple of years?? "Mummy, who's that?"
Last edited:
I really wish they taught programming and other technical skills in high school. There are so many people that have no idea how challenging making a game and updating it is.
Used to work at a site that had several petabytes of storage and one bright spark (read Muppet) tried to propose tape backup.
I almost died that day
If they did, no-one would go into IT.
Overtime!
And there are many that do.
Thanks for posting this. Great to get a view of the architecture that makes our gameplay possible. And anyone watching it will hopefully appreciate the complexities involved in performing a major update such as this release (not to mention the day to day challenges of keeping it all ticking over smoothly).
To my knowledge a lot of high schools offer programming and technology courses.
Personally, I learned Visual Basic and enterprise networking in high school. I pursued Physics at university but retreated back to IT once I realized there is no money in that.
Physics (a discipline that requires mathematical, logical, analytical skills) gets you into high finance, even better if you can program. Six figure salaries, seven figure bonuses. Not my personal cup of tea, but it's hardly a dead-end if you are motivated purely by money.
The Elite Dangerous database is HUGE. Updating data, schemas, and applying new tables, columns, and populating it with the proper data takes a lot of time. This is also a time when we generally generate database backups in the event of the update on the backend failing, we can roll back if needed. In addition, large updates requires QA to comb over these change to ensure the game is 1) stable, 2) won't show signs of data corruption, 3) no bugs that decided to make a return. I'm sure our server/engineering team can greatly expand on what i've said - as they know the nuts and bolts to the entire ordeal much more so than I.
I would love to hear some more details on this
From what we know so far, its a combination of MySQL, MongoDB, Redis and AWS DynamoDB. I would love to know some general tech info, like how big the DBs are (to the nearest TB), what are the bottlenecks in the backups etc. I *think* you are using self-hosted Mongo/Redis/MySQL (i.e. running on AWS instances)? Curious to know if you're using something like Google's CloudSQL behind the scenes for better scalability etc? I've done backups/migrations etc on CloudSQL and its super-fast.
I actually did go into finance but on the transaction and records management side. We did a lot of really neat big data analytics to see whether a 6 month old would be a good investor in 18 years based on their family location and opportunities. Crazy stuff.
But seeing the under belly of the system...and the kinds of assumptions and wild theories applied by other physics folks to essentially take advantage of the middle-class, I was definitely soured on all that.
Money is important, but I don't need a lot of it. My dream for physics was always going to be more lab-based and academic...which just doesn't pay much and I was pretty burned out on upper level math, anyway.
I'm pretty happy now using my problem solving skills in designing solutions for software....and it pays pretty well.
Then they'd be asking why it takes so long when they can do the same thing using Visual Basic in an afternoon.
Best that they continue to think that we perform magic. The biologists at work don't bat an eyelid now when I perform my incantations ...
the thing is, there are some people here, who learned programming around high school, like me. programming language was called basic, and email was just invented... so, this stuff is really interesting for me! (furthermore because i did jobwise things not at all connected to such things)
As a Senior Software Developer this is a good read. I've had to perform database maintenance but nothing on the level which FD are currently doing.*
*Hopefully for just another 10 minutes or so.
Heh, yeah, I think I have some experience with this. The company I used to work for managed the records for American Funds...that is $1.3 Trillion dollars in managed assets (plus the next three big dogs in the mutual funds industry). Down time had to be scheduled on Friday after 3PM EST so that maintenance could be allowed to run all through the weekend until the markets reopened on Monday @ 8AM. Further downtime cost us thousands per hour in SLA fees, so the pressure was always on to make these deployments efficient. Even still, it would often take 10 hours to install and refresh everything and then verify integrity.
Back in the 90s when I used to work for NASA, Dan Golden, then NASA head administrator, started the initiative: "Faster, Better, Cheaper". Whenever he gave a speech to us and used that slogan, the engineers would whisper to each other "pick two of the three". While that's a bit of a snide remark, the basis for it is true. It is near impossible to achieve all three at the same time. Sure, sometime you get lucky, like the original Mars rover program. But other times you end up with Mars Climate Orbiter.
Personally, I'm willing to give up a bit of time to make sure FD achieves "better" and "cheaper".
Personally, I'm willing to give up a bit of time to make sure FD achieves "better" and "cheaper".
That's not a faster, better, cheaper lesson. The lesson from the MCO was "Use metric exclusively. US customary units will come back to bite you (and, as per Murphy, always at the most expensive point)".
- - - - - Additional Content Posted / Auto Merge - - - - -
Used to work at a site that had several petabytes of storage and one bright spark (read Muppet) tried to propose tape backup.
I almost died that day
You laugh, we actually used to do that. Turn out tape is the cheapest solution available for bulk storage (especially if it's things that don't need instant access; think tax records, contract histories etc).
Of course, at that point it's not so much "tape backup" as "hierarchical storage with tape as the final tier".