[ANNOUNCEMENT] Additional Information Regarding Recent Server Disruption - 04/01/2015

[Andrew Barlow]

Hello,

I had previously posted a message similar to the following in the forums already (https://forums.frontier.co.uk/showth...=1#post1425751), but felt the players (you guys!) deserved a little more explanation and a plan of things to come!

Firstly, I'd like to apologise to all commanders affected by the recent server issue we suffered over New Year. On January 1st 2015 around 11:00pm we had a network issue with one of our databases. Our online engineering team spotted the issue and reacted to it immediately. During this period, a relatively small number of commanders were affected in various ways. Some having cargo stuck in their ship, others with missions that couldn't be completed/aborted and vanishing ships. Our customer services team have been working through tickets to resolve these issues on an individual basis since the event. We have also updated and improved the architecture and solidified our server side technology to avoid this happening again.

After shoring up the database, in the early hours of the 2nd of January, we automatically refunded all the affected players for their lost transactions and also granted them a 20% bonus for their troubles. However, during this refund process, a very few (lucky) commanders received more credits than perhaps they were entitled to (5 billion credits in some cases)!

To be clear - these players will not be banned. Over the next 24 hours we will be investigating every affected player and performing individual commander rollbacks where we feel it's fair to - which may include credits, assets and rank where appropriate. We may reach out to individual commanders where this situation may not be straight forward. None of these players will be eligible for the "Race to Elite" competition if we feel they have gained an unfair advantage over the competition.

On the morning of the 3rd of January, after reports that some players were still having issues, we ran a second pass over the database to remove any excess cargo commanders may have still been carrying, which should have unblocked the commodities markets for those remaining affected players. In the next 24 hours we shall also be issuing a server side fix to automatically issue a credit or asset refund on login after cases similar to these. Of course, if you are still having problems, please submit a support ticket at https://support.elitedangerous.com - our customer services team can help.

We are also aware of an exploit (I wont detail it here) which occurred due to this database error, where players have been abusing the broken state of their ship to "create credits". These players know they are exploiting the system and will also be investigated. Actions may be taken against players who are known to be exploiting the system and do stand out from the commanders who were simply given the credits during our credit refund process. These players will also not be eligible for the "Race to Elite" competition if we feel they have gained an unfair advantage over the competition.

Again, I (and the whole of the online team) apologise for the disruption you may have suffered over the last few days. As you can see, we reacted promptly to the situation but simply failed to communicate this in good time with the players.

To avoid this happening in the future, in the next week, we shall be publishing a "Server Status" dashboard which you can use to see the state of the online galaxy. From here we shall let you know immediately of any problems we are currently facing, any known issues and any scheduled down-time for maintenance. This should give us a single place for us to be able to communicate directly with you guys about any problems the servers are currently facing - or hopefully - simply that all is well in the galaxy.

Finally - if you are still affected by this issue in anyway, please do submit a support ticket: https://support.elitedangerous.com. Our apologies for the disruption and we shall endeavour to communicate with commanders more promptly and directly going forward.

As always, fly safe commanders!

Andrew Barlow

[Evilroco]

Many thanks for the info' Andrew I shall await the resolution of my tickets on this small problem.

[dextrovix]

Fair enough I think, I'm sure the community appreciates the explanations.

[Roybe]

Hello,

To avoid this happening in the future, in the next week, we shall be publishing a "Server Status" dashboard which you can use to see the state of the online galaxy. From here we shall let you know immediately of any problems we are currently facing, any known issues and any scheduled down-time for maintenance. This should give us a single place for us to be able to communicate directly with you guys about any problems the servers are currently facing - or hopefully - simply that all is well in the galaxy.

Andy Barlow

Thanks for this update and the incoming data! We are looking forward to the implementation of this!

[Francik Zelozko]

Hilarious. So how about all the people that are reporting the problems they STILL have in this tread:

https://forums.frontier.co.uk/showthread.php?t=90640

?

[DopeGhoti]

However, during this refund process, a very few (lucky) commanders received more credits than perhaps they were entitled to (5 billion credits in some cases)!
[...]
To be clear - these players will not be banned. [...] None of these players will not be eligible for the "Race to Elite" competition if we feel they have gained an unfair advantage over the competition.

Maybe something got a bit faffed up as you were editing, but.. If you think they did get an unfair advantage, the people who got five billion credits will not be eligible for the prize? It's written the other way, thanks to a presumably errant double-negative.

[Andrew Barlow]

Maybe something got a bit faffed up as you were editing, but.. If you think they did get an unfair advantage, the people who got five billion credits will not be eligible for the prize? It's written the other way, thanks to a presumably errant double-negative.

Good spot, I've corrected the OP.

[bitstorm]

Andrew, you star, that's brilliant.

Everything covered, people just want a bit of reassurance that it's all being sorted out. There may still be outstanding issues but I don't think anyone could ask anything more.

Thanks very much =D

[bitstorm]

Just as a suggestion, for those whose problem is fixed.. It's probably worth closing down your tickets, that way people still with issues will have their problems solved a little bit sooner.

[ThatFuzzyTiger]

"Again, I (and the whole of the online team) apologise for the disruption you may have suffered over the last few days. As you can see, we reacted promptly to the situation but simply failed to communicate this in good time with the players."

Finally - if you are still affected by this issue in anyway, please do submit a support ticket: https://support.elitedangerous.com. Our apologies for the disruption and we shall endeavour to communicate with commanders more promptly and directly in the future.

Reason for edits : Cleaned up typo, replaced consultancy speak "Going forward" with more understandable english. Performed by Tigerwikibot

[JPB]

Please don't forget us souls who had our save games corrupted by the server issue and have been locked out since

[MasterFlash]

Good to here - I want to play again after the '5 billion burning a hole in my wallet' rollback.

[Cosmos]

Hello,

I had previously posted a message similar to the following in the forums already (https://forums.frontier.co.uk/showth...=1#post1425751), but felt the players (you guys!) deserved a little more explanation and a plan of things to come!

Firstly, I'd like to apologies to all commanders affected by the recent server issue we suffered over New Year. On January 1st 2015 around 11:00pm we had a network issue with one of our databases. Our online engineering team spotted the issue and reacted to it immediately. During this period, a relatively small number of commanders were affected in various ways. Some having cargo stuck in their ship, others with missions that couldn't be completed/aborted and vanishing ships. Our customer services team have been working through tickets to resolve these issues on an individual basis since the event. We have also updated and improved the architecture and solidified our server side technology to avoid this happening again.

After shoring up the database, in the early hours of the 2nd of January, we automatically refunded all the affected players for their lost transactions and also granted them a 20% bonus for their troubles. However, during this refund process, a very few (lucky) commanders received more credits than perhaps they were entitled to (5 billion credits in some cases)!

To be clear - these players will not be banned. Over the next 24 hours we will be investigating every affected player and performing individual commander rollbacks where we feel it's fair to - which may include credits, assets and rank where appropriate. We may reach out to individual commanders where this situation may not be straight forward. None of these players will be eligible for the "Race to Elite" competition if we feel they have gained an unfair advantage over the competition.

On the morning of the 3rd of January, after reports that some players were still having issues, we ran a second pass over the database to remove any excess cargo commanders may have still been carrying, which should have unblocked the commodities markets for those remaining affected players. In the next 24 hours we shall also be issuing a server side fix to automatically issue a credit or asset refund on login after cases similar to these. Of course, if you are still having problems, please submit a support ticket at https://support.elitedangerous.com - our customer services team can help.

We are also aware of an exploit (I wont detail it here) which occurred due to this database error, where players have been abusing the broken state of their ship to "create credits". These players know they are exploiting the system and will also be investigated. Actions may be taken against players who are known to be exploiting the system and do stand out from the commanders who were simply given the credits during our credit refund process. These players will also not be eligible for the "Race to Elite" competition if we feel they have gained an unfair advantage over the competition.

Again, I (and the whole of the online team) apologise for the disruption you may have suffered over the last few days. As you can see, we reacted promptly to the situation but simply failed to communicate this in good time with the players.

To avoid this happening in the future, in the next week, we shall be publishing a "Server Status" dashboard which you can use to see the state of the online galaxy. From here we shall let you know immediately of any problems we are currently facing, any known issues and any scheduled down-time for maintenance. This should give us a single place for us to be able to communicate directly with you guys about any problems the servers are currently facing - or hopefully - simply that all is well in the galaxy.

Finally - if you are still affected by this issue in anyway, please do submit a support ticket: https://support.elitedangerous.com. Our apologies for the disruption and we shall endeavour to communicate with commanders more promptly and directly going forward.

As always, fly safe commanders!

Andrew Barlow

I'm extremely pleased to hear this and thanks for letting us all know.. I can get back to enjoying the game now.

If I were dealing with matters..

[Regulator]

well not everyone was fixed I still can't sell my cargo .....I put a couple tickets in with a big fat goose egg for a response!!

[Mobius]

I take it the issue with the group servers isn't fixed yet either, as I cant access the group management window due to the size of the group.