Server Maintenance Downtime Over Christmas Period

[Agony_Aunt]

Fair comment there. If the timer still keeps ticking when we are unable to do anything about it that is not good at all although I suspect when the server is down the BIG DADDY program (sorry all I could think of ) is probably stopped as well... has anyone checked this, do timers change after server downtime ?

I believe devs have stated that when servers are down, timers stop. Easy enough for you to verify yourself if you take note of any timers running when the server goes down and take note of real time passage vs in-game mission timers when you log back in.

 

[dungeonseeker]

While I don't begrudge them the downtime, they set themselves up for these complaints by including missions that tick down in real time, and don't stop timers when the servers are down.

If you notice, the in game clock actually mimics the IRL clock (the minutes match up at least) so I'm guessing the timers are programmed to use the in game clock to count, in other words it was done that way on purpose. Fdev have already they're making the game they want to play, I don't see this design choice being reversed anytime soon.

 

[ollobrains]

Sever with the address ( and this is the only one doing it for me) keeps causoing graphics errors mind u my system is more than enough and it copes on other servers but heres the IP address 54.170.20.117.19364

 

[Nonya]

Looks like the server TIME is synced to GMT. Which would make things easier when troubleshooting something.

 

[tomPhoenix]

I like how there's daily maintenance... That's programmer code for "we have a memory leak and the best way to get rid of it is a scheduled reboot." Ah well the timing does help me decide when I need to put the game down and go to bed. =)

 

[ollobrains]

the server updates arent doing their thing with system influence changes and so forth gamesbroken

 

[tomPhoenix]

The downtime is probably to stop the services, rotate the logs, and take snapshot backups of the virtual servers this thing runs on so in the event of a major catastrophe they can quickly recover. I'm sure as time goes on and if they expand their server base they'll figure out a way to do all that without having to shut the entire thing down, possibly by moving players to other virtual servers when they jump from one system to another and then backing up the now-empty servers - who the heck knows how they've built this thing? The engineering behind these things alone are usually trade secrets.
This is no different from any other company that runs it's own servers and isn't Amazon, Microsoft, or Google with literally hundreds of thousands of servers each scattered all over the planet.

They shouldn't need to shut down servers or stop services to rotate logs. Logrotate does this automatically, but even their own program can choose to log to different files. S3 backup (Glacier even better for cost) of logs (if they even back them up). The server images would likely require an outage, but you can keep that a rolling outage by doing each server one at a time or just bringing new cloned images online and discarding old servers entirely. In fact it's likely better for security to discard servers and not let them get too old should one be compromised (not that the new ones also couldn't be compromised, but hopefully they'd come online patched).

Stateless. Is the name of the game here. 12factor.net.

The only reason to have an outage is if you have a memory leak or you aren't a good software/server architect. OR you only have one server =) and are on a budget.

 

[ollobrains]

changes in system statuses have occoured at downtimes to each day

 

[Deyn]

As others have said already, downtimes in a flexible, scaling cloud service is bad architecture or severe program deficiency. I do this architecture planning stuff as part of my job. If you plan and build it with downtimes 'required' you are either still in proof-of-concept-phase ironing out design flaws, that shouldn't be there in the first place as they should have been omitted in the design, or the software you use is not built for cloud environments. Both alternatives don't sound good to me. Either Frontier decided to go live far to early (as lot of testers during late beta, pointed out) or they patched and stitched the backend together from parts not fit for the task... so we will see regular downtimes or even longer outages in the future.
I hope it is a design issue that can be fixed and will move on to a seemless cloud environmemt with smooth transition between updates and maintenance as a rolling replacement of virtual instances, as in any modern cloud service.

 

[Deyn]

Now I'm beginning to see why people were in an uproar about the decision to cut out offline mode.

Yep! That was one of the reasons... and it is why people ask for mission timers to stop, when you are not playing, or can't play, as well as people love to come home have 1 or 2 hours before the family needs attention. So they start the game, only to see there is an 'forced' update, that will ruin 40-120 minutes of their playtime.
No offline means, you can't skip the update and just play a bit, postponing the update to the end of the session, so you can prepare dinner/watch a movie with your wife, while the update is running... yet another hour wasted, thanks to online only game dictating your life!

 

[Deyn]

Also, do you expect the server team to be up and in the office at 5 AM? Be reasonable.

Actually I expect the service to be available 24/7 as my customers expect from the services I design for datacenter implementation. Welcome to the 21st century. It is called 'operations', you know the guys in that global team, doing follow-the-sun work to keep business running. How would you react, if your shopsystem provider would tell you it couldn't process purchases through credit cards during 7:00 and 8:00 am each day due to maintenance downtime? Guess you would switch the provider quickly or have him make it happen?

Yes, with scrapping offline mode it is absolutely reasonable, that you guys do what is necessary to make the servers available at all times. Your decision, live with the consequences.

 

[RickDangerous]

Actually I expect the service to be available 24/7 as my customers expect from the services I design for datacenter implementation. Welcome to the 21st century. It is called 'operations', you know the guys in that global team, doing follow-the-sun work to keep business running. How would you react, if your shopsystem provider would tell you it couldn't process purchases through credit cards during 7:00 and 8:00 am each day due to maintenance downtime? Guess you would switch the provider quickly or have him make it happen?

Yes, with scrapping offline mode it is absolutely reasonable, that you guys do what is necessary to make the servers available at all times. Your decision, live with the consequences.

Agree. My company has staff in US, Europe, and Asia, so if there's a fire we can get on it ASAP. That's just how it is these days.

 

[nillian]

I like how there's daily maintenance... That's programmer code for "we have a memory leak and the best way to get rid of it is a scheduled reboot."

Haha, that brings back memories

 

[Assimilator1]

Actually I expect the service to be available 24/7 as my customers expect from the services I design for datacenter implementation. Welcome to the 21st century. It is called 'operations', you know the guys in that global team, doing follow-the-sun work to keep business running. How would you react, if your shopsystem provider would tell you it couldn't process purchases through credit cards during 7:00 and 8:00 am each day due to maintenance downtime? Guess you would switch the provider quickly or have him make it happen?

Yes, with scrapping offline mode it is absolutely reasonable, that you guys do what is necessary to make the servers available at all times. Your decision, live with the consequences.

Rubbish, invalid comparison, I bet your data centre involves businesses that make money? Obviously ED is a game & we don't make money from it (barring the 1 off competition). I suppose you'd argue that this down time will upset customers & they could leave, bearing in mind we are just talking about the xmas hols having any significant effect, those who like the game aren't going to leave only because their might be some downtime/server problems once a year over a week or so, & the few that might aren't going to have much affect, don't forget it costs more to have them manned 24/7, I don't see that a few leavers not leaving under those circumstances making up for it.

 

[epoch71]

Rubbish, invalid comparison, I bet your data centre involves businesses that make money? Obviously ED is a game & we don't make money from it (barring the 1 off competition). I suppose you'd argue that this down time will upset customers & they could leave, bearing in mind we are just talking about the xmas hols having any significant effect, those who like the game aren't going to leave only because their might be some downtime/server problems once a year over a week or so, & the few that might aren't going to have much affect, don't forget it costs more to have them manned 24/7, I don't see that a few leavers not leaving under those circumstances making up for it.

Frontier Development certainly make money, and for sure it's in their interest to ensure they have a solid, satisfied, customer base. They don't just have customers to keep happy. They have investors and shareholders too. And, money aside, reputation is quite important.

I'm just making a general point here, and not specifically in relation to the current problems which don't bother me in the slightest. I'm gonna blow the dust off my Commodore 64 and play the original for a bit. Or maybe some Jet Set Willy.

 

[andino]

This is annoying.

Server down again........

 

[andino]

This is annoying.

Server down again........

I paid $60 to wait on a server?????

 

[Deyn]

Rubbish, invalid comparison, I bet your data centre involves businesses that make money? Obviously ED is a game & we don't make money from it (barring the 1 off competition). I suppose you'd argue that this down time will upset customers & they could leave, bearing in mind we are just talking about the xmas hols having any significant effect, those who like the game aren't going to leave only because their might be some downtime/server problems once a year over a week or so, & the few that might aren't going to have much affect, don't forget it costs more to have them manned 24/7, I don't see that a few leavers not leaving under those circumstances making up for it.

Oh, sorry. Of course my "expertise" is rubbish, how could I know you are an expert on global cloud data center implementations? You may want to send me your resume, we might use somebody with your knowledge about scalable high availability solutions and service design. But what do I know...

The funny part is, if they need downtimes, to do manual maintenance, they even have more people to pay as if they would do it like professional operators with high level of automation. So your argument about the money is exactly, why they should have done it right in the first place.

I'm not talking about one Data center, nor is it "mine"... I have to take into account a design spanning all regions and multiple data centers when implementing services for a global company.
Our "customers" can't just leave, they are part of the company, rather if I make big mistakes, I have to fix them or face the consequences.

If Frontier hadn't bragged about their experience in game development and "that solo mode is as good as offline singleplayer", did advertise Elite: Dangerous without tricking people into buying with "Yes, there will be an offline singleplayer. You won't need a server."... but now it is just another point, why people wanted to have an oflline mode.

And BTW, where do they say it is just ONE week?

 

[s4bleknight]

Rubbish, invalid comparison, I bet your data centre involves businesses that make money? Obviously ED is a game & we don't make money from it (barring the 1 off competition). I suppose you'd argue that this down time will upset customers & they could leave, bearing in mind we are just talking about the xmas hols having any significant effect, those who like the game aren't going to leave only because their might be some downtime/server problems once a year over a week or so, & the few that might aren't going to have much affect, don't forget it costs more to have them manned 24/7, I don't see that a few leavers not leaving under those circumstances making up for it.

Aww damnit Elite was free? And I paid for it...

 

[ThatFuzzyTiger]

I'm not talking about one Data center, nor is it "mine"... I have to take into account a design spanning all regions and multiple data centers when implementing services for a global company.
Our "customers" can't just leave, they are part of the company, rather if I make big mistakes, I have to fix them or face the consequences.

Or worse, your clients have nasty contracts with very, very scary Service Level Agreements woven into them. Believe me, those things tend to focus your mind in a way that the only comparison I can draw would be a large man leaning over and resting his hand on your shoulder and saying in a low voice "You got five minutes. Fix it."

Moral of the story, if you're working in an environment where the words "Service Level Agreement", "Unplanned Downtime" and "Per minute or portion thereof" are placed in the same document, big mistakes equate to the powers that be making you feel very small indeed. Thankfully, I only had a near miss (I -only- managed to patch the PBX into one of the core routers and therefore blipped everything on the node for a few seconds), and said near miss was heralded by my then supervisor screaming "WHATEVER YOU DID, UNDO IT RIGHT NOW!", but I've seen the results of people who didn't near miss. It wasn't pretty.