Self-fulfilling Prophecy - Frontier need Increased QA/Customer Support?

[Obsidian Ant]

Current Beta is Great - Elite have enough Customer Support at Release?

EDIT: Just a quick update to say this post isn't about Elite's current state, and has nothing to do with the Beta builds (which I think are great), it's actually about a different subject...so read on...

-----------
-----

First up I realise I have made what have been perceived as 'negative' threads in the past, but my intent is never to rail against Frontier or Elite. I love Elite - even in its current state it is in my top games of all time - and for me, goes way beyond being merely a game...it's something I feel very close to. I am also very pleased and impressed with how Frontier have managed the development of this game. So I implore you, please don't see this as a negative thread...I've created it so we can discuss a potential issue that may cause Frontier some problems.

A while ago, in another thread a seemingly knowledgeable individual made the following posts. At the time many people either disagreed with him, or misunderstood what he was trying to say. The posts address some of the potential pitfalls Frontier face regarding the success of Elite. In this case the poster proposed that Frontier may be under-estimating the resource requirements of QA, and Customer Support. That in fact, the poster believed based on his experience within the MMO industry, and the signs he had seen with both Frontier and Elite - that Frontier are currently supporting Elite in the fashion of a single-player game developer. This makes sense, as that is Frontier's history. The poster went on to say that in nearly all cases this has happened before the game in question inevitably went on to seriously struggle. Here are two of his posts:

Go take a look at the top-level of their forums for a moment. 4140 threads in the "Elite: Dangerous General" forum, 147,427 posts. Just in the one Elite forum. Versus 219 threads/1,750 posts for "RollerCoaster Tycoon".

ED isn't just a foray into Multi-Player Elite, they made the decision to go MMO-style, and making an MMO is very different than making the sort of single-/multi-player games Frontier have been (doing very well) making, and there are some fairly significant tells that Frontier don't have a good handle on it.

First, and fairly simply, they're describing this as "beta" when the product is clearly in an alpha state.

Secondly, it seems that this dev team thought that forums would be a good way to interact with the testers at this phase....

Sides... Hurt... Deep breath.

Third, it's pretty clear to me that they aren't aware of the potential data they could/should be gathering from this test, never mind gathering it and all over it.

Lastly, there's no customer-facing evidence of QA participation in the current work. That's the mindset of a box game dev; get it written, then get it tested.

A successful MMO would already have QA pretty well integrated, and the availability of a giant pool of free testers would have the QA team working us for all they're worth.

MMOs require live support, and for that you need continuous QA, which means you need to integrate QA early so that they can build up a massive repository of all the things they need to test for any given patch; so they can develop a QA-perspective understanding of the interplay of different systems so that they can make informed predictions as to what to test when a developer says "I only changed this one line of code".

At the same time, you need to develop a good workflow between QA and your dev team. Many game devs will react to QA feedback very negatively at first, until they learn to make use of them. Most software engineers work better when they get immediate feedback on something they've checked in. But if they finish a feature and it takes 3 months for someone to come back and say "it turns inside out. And then explodes", its disruptive on several levels.

But Frontier don't even provide a readme in the launcher when a patch comes out. If you're thinking, what does the readme matter, it means that they don't have any of the pre-requisites for regular readmes in their workflow yet.

Maybe there are forum posts with test direction because, yeah, there's not like a decade of proof that that just doesn't work.

There's no in-game bug reporting tool... Question: When did "WildStar" introduce the bug report feature? Answer: Alpha.

And I get to say "there's no sane bug feedback/reporting tool" because, well, this was me.

And they're in beta.

The last decade is littered with failed games and unemployed Austinians who didn't realize that you have to build the infrastructure for an MMO early.

One little known game, released in 2001, was to be a half-scale map of Europe, all those thousands of towns. The terrain editor seemed pretty trivial, so it wasn't a priority until the last few months.

With a little pressure from the producers, the engine guys took time out to put together a fairly simple tool for building towns approximately 2 months before gold.

It took approximately 3-4 days.

For a small town. Add another 3-4 for a large town, and 1-2 days to link the towns up together.

Lets do math.

3.5 x 1000 = 3500 days = 9.5 work years. Or 6 months of 22 terrain editors working flat out. Just to put scatter small, not very fun, hamlets around Europe.

Just to build the terrain, with no review or testing, assuming no crashes or network hiccups, would have required 68-70 people doing nothing but building terrain. Or, more realistically, 1 month of manic hiring and training and 150 or so people battering out towns for a second month.

You can't wait until the game is "ready" and then expect to build the tools to run and maintain it. Top of my head, I can think of 9-10 games that unwittingly tried that. I'd link one, but the only one still "in business" is WWIIOL.

It's still possible these guys can pull it off, but based on the evidence right now, my money would be on a continuation of the unfortunately negative trend of gameplay experience, stability and overshooting that ran from Elite -> Frontier -> First Encounters into Dangerous.

http://forums.frontier.co.uk/showpost.php?p=511910&postcount=61

Again, this is a normal part of the pattern. One of the usual big fails is to apply the box dev strategy of late qa. This is something my friends at gearbox learned the hard way (and my friend Gryf worked up to exec by getting the qa process integrated end-to-end on Borderlands, bringing the massively approach down into the box/multi shop). QA can be expensive and it seems counter intuitive to start spending money on it before you've finished the game. But in a persistent, massively game you are going to be doing qa week after week forever. It needs to be an integral part of the workflow. See my previous replies for a fuller explanation.

I had friends who worked on a particular studio's first massively, long story short, they just figured that their BI team would know what and obtain for them the live data they needed. That was more forward thinking than many shops, but it still want enough. The game ran for five years but they spent so much time fighting fire that they burnt all the money that was supposed to support new hires by finding expansion dev, had to lay people off and then try to dev the expansion while fighting fires and maintaining a sufficient player base to pay the bills.

When CoV finally came out it was too little too late and failed to show they'd learned many lessons because, frankly, employees had churned hard. A studio is the sum of its people and not some magical knowledge transfer system. CF Diablo 3 vs Diablo 2. Most of the guys from D2 had left for other studios (Carbine, Red5, etc) vs StarCraft which retained a lot of the same guys and Warcraft where they had a lot of carryover from W3 to WoW but only a handful of today's WoW team are originals (there are a few guys who have collected their 20 years crowns, but they are mostly working over on SC).

https://forums.frontier.co.uk/showpost.php?p=516640&postcount=144

Now as I said, at the time many people disagreed with the guy, and many others didn't quite understand what he was saying. Some suggested that Elite is not a MMO...others suggested that Frontier have plenty of money to handle the situation.

My thoughts were in line with kfsone's; it doesn't matter whether or not Elite is a MMO or not. Ultimately it will have hundreds or thousands of players online simultaneously. That means it will need to be supported in the same manner as an MMO.

Secondly, how much money Frontier have earmarked for Elite is irrelevant in this matter. Money - no matter how much - cannot instantaneously provide staff fully capable of supporting a large game...along with the resources to handle such support. All of this takes time and needs to be built up along side game development.

It is not something that can be added last minute.

So - here we are nearly six weeks later. Elite has come a long way in that short time - and the number of players has increased dramatically! The question then, is how far has support for the game been developed in this time? Have Frontier been building their QA department to handle this huge influx of players?

Here is a post from the Frontier Customer Support Team, made yesterday:

Hello, Commanders!

Firstly, on behalf of the team I'd like to thank everyone for your contributions towards the development of Elite: Dangerous in the form of your support tickets and bug reports. You're all actively helping to ensure that Elite: Dangerous becomes the smooth, exciting and enjoyable experience that we want to make and that you all want to play. Thank you all for your vigilance and dedication towards the game!

Now, since the release of the Standard Beta the number of players has sky-rocketed and, as such, the number of support tickets logged has also increased. For this reason we ask that you all offer us your understanding and patience in that it may take some time to respond to your reports due to the sheer number of tickets that we have accumulated since the launch of the Beta. We will be prioritising issues based on their severity, so those who cannot access the game will be responded to as a priority before others.

Once again, thank you for your time and commitment to the development of the game and for your patience whilst we try to get back to you all as quickly as we can.

See you out there!

The Customer Support Team

http://forums.frontier.co.uk/showthread.php?t=31613

So, is kfsone right, are Frontier underestimating the needs for a fully ongoing support team?

As Elite is essentially a perpetually online Universe the support requirements will be ongoing. This isn't an issue of having a large number of bug reports simply because it is Beta - once the game is released...there is going to be a huge number of on-going reports.

Do Frontier have this covered? How long does it take to get a large enough team trained up and in-place to support a game like Elite? I'd agree with kfsone, that fire fighting isn't the right approach for a game like Elite.

Remember this isn't intended to be a negative thread. I love Elite, and have huge respect for Frontier. Thing is, these questions occur in my mind - and I put them out there.

 

[EAGLE 5]

I think one needs to really sit back and analyze things as they really are.

FD is a company of roughly 200-500 people (this is according to their own LinkedIn site) that isn't a whole lot of people in the grand scheme of things. They themselves acknowledge that and are constantly advertising that they are hiring.

If you consider the scope of the product we have in our hands these days you can say that FD has put forth an herculean effort and the pitfalls of sluggish response from customer support coupled with the fact that honestly they are still working out the bugs themselves on a BETA product shows just that.

I have full faith in FD and it may take some time before the transition from Kickstarter to programming to support will be complete. We again are dealing with a beta and we ourselves should learn to recognize that and become instrumental in providing the development team with as much detail as needed to make it all happen smoothly.

Ultimately we "play" or choose to participate in the beta (including paying for it) knowing full well that this was a crowdfunded effort with a lot of novel ideas and an ambitious vision all from a company without the levels of staffing required, if we keep that in mind we shouldn't be surprised when things don't go the way we want them to...

At least we know that there is a vision and there won't be a marketing rush behind a decision to release an unfinished final product so.. just give it time.

 

[beelbeebub]

Hummmmm....

A well thought out set of posts and kfsone does seem to know his stuff.

It's possible that the QA team at FD are ramping up but that doesn't always mean that you an open with demand, especially for a release like beta.

There was a fairly large influx of new users, a bunch of new features and a load of major bugs.

The fact that they iterated through to 1.03 so quickly is a good sign I would think.

Right now there don't seem to be many game breakers, I get the odd crash, a few freezes, and the odd disconnection. More then a final game should have but it's a steady progression.

The bigger problem is with the incomplete feature set combined with an untrained tester pool. Real testers know it's their job to dock 100 times in a row to check the docking pads work. The untrained masses, some of whom appear to be here under the impression they've bought early access to a box ready game (and are busy trying to get the refund they feel entitled to), are just milling about whining that it's not like EVE (a bit harsh, sorry).

This generates a ton of tickets that have to be waded through, kinda clogging the process



This causes a lot of 'my Eagle can't jump more than 4ly' requests.

Hopefully this will smooth out as

 

[Apostle]

Yes and the fact that the builds that we have are in such great shape says a good deal about their QA process. Take a look at SC and see where they are at with over ten times the funding. The Frontier folks have been knocking this whole project out of the park in terms of what they are delivering. I have seen retail games in far worse shape then Elite is right now. Anyone play X3 (or any of the X series for that matter) on release? They need to be focused on getting the game done and the features in and that appears to be just what they are doing. They should not be dealing with people who have bugs wipe them out or other customer service issues.

 

[DraigUK]

A good read OP, and while I do see the points your making, and kfsone, I don't think it's quite at the point of panic stations yet.

Definitions of Alpha/Beta aside (because to me this is definitely a Beta now not Alpha, just a Beta in the traditional sense as opposed to what they have been marketed as in these recent few years).

The dev team put a lot of fixes in going from Beta 1.00 to Beta 1.03 in very very quick time, which was impressive. So on that side of things and the general pace of development I think all is going well.

I do think they have underestimated the amount of bonehead tickets they are clearly receiving since release of Beta to a wider audience, and that is reflected by the quality of posts on the forums in general since release of Beta also.

Getting tickets asking for "refunds" on loss of ships/credits due to bugs etc being a prime example, and far too many people on here not realising what a Beta test actually is, and their expectations and treatment of this test being something like a full release and the expectations that go with it are not helping at all.

It really does annoy me seeing so many of these posts at this stage and the sense of entitlement that goes along with them :-

But I PAID £50 for this!
I lost EVERYTHING I have WORKED for hours to get!!?!! REFUND!!
Game sucks!! I can't even get to Sol yet!
No tutorial!!!!!
etc etc

An argument could certainly be made that FD should have expected and managed this better, getting the message out good long and early that your not going to get refunds on anything in a Beta, your here to test and they have better things to do than pander to your special little snowflake syndrome, but honestly, I doubt it would have made that much of a difference.

Those types would still ticket regardless, swamping the system no matter how many staff they could reasonably have had at this stage of testing.

I'd agree they could direct the testing more, but again I'd question how many would actually be bothered to do it anyway, properly.

I'd also agree that now they have been able to see how strained the current system is, they should now be taking steps to get more people on board with a view to the next stage of testing, and especially in place before Gamma and eventual release, I just don't agree with all the points made or that it is quite at panic stations yet.

I'd disagree on the points of including readme files anyway. I don't think that many ever read them to begin with, and certainly not these days.

The first port of call is usually forums, so some work on making the forums easier to navigate with clearer, I dunno, Help sections I guess, that people can't post in but just refer to would help a bit, for example, and making the forums easier to find/access with some direction right from the get go directing people here would help as well, things like that would have more of an impact.

Also we as a community and the mod team really need to come down a bit harder on those waste of time posts in my opinion and not let them flourish and clogging up the front page of the forum and hiding the less toxic but more helpful posts to quickly.

Because this situation will only get worse. So we can do our bit as well to help the situation somewhat, and throwing money and staff at the situation isn't always the best answer, but as I said they need to start getting more support staff in about now ready for the gear up to Gamma/release.

 

[Hydrino]

Just ask FD already and don't multi-thread the same idea

This is why I wish the Mods would lock the new thread creation function and require permission to ensure the topic is new. Now that is a valid issue to start a new thread about.

You should send your question in much shorter form to the ask the newsletter email address if you want an answer and not just more forum posts.

What a wall of text taken from other threads. Why not just post where kfsone did and say you agree? Number of posts in a forum is not an issue. One of the previous threads you link had 28 pages of posts which require no response from FD as a mix of repetition, social interaction and random speculation.

I am sure kfsone could contact FD and offer his/her expertise. However, I am not interested in appeals to authority posted to a forum. He/She needs to ask FD and not ramble on for 22 lines about 9 years of work hand-crafting villages. Irrelevant in a game that already has procedurally generated a very beautiful 400 billion star systems galaxy, with PG naming, creation, population, etc. obviously working and being tweaked iteratively.

QA is obviously involved before each iterative release and I have had just one crash and that was after leaving the game running all night. Are there bugs and issues? Yes. Is it a finished game? No.

QA interface? Ticketing system is built into the launcher. I've used it, works a treat. In game ticketing might be nice if it captured exact game-state but it might be more development than it is worth and just code bloat hurting FPS performance. Of course FD have this in-house in their development environment and using tools like Visual Studio.

 

[tomba4]

As Elite is essentially a perpetually online Universe the support requirements will be ongoing. This isn't an issue of having a large number of bug reports simply because it is Beta - once the game is released...there is going to be a huge number of on-going reports.

Do Frontier have this covered? How long does it take to get a large enough team trained up and in-place to support a game like Elite?

Depends on what state are they going to release the game in. If they really rush it and do the full release in December, I think it will be as bad as you fear. If they do what they promised (i.e., release the game when it's ready), they should be fine - it's not going to be SC with gazillion of players.

The increased number of tickets after standard beta started makes sense - there are parts of the game that are badly broken and affect almost everyone, no wonder people are creating tickets.

That being said, I agree that there's still no need to panic. We can start worrying when the next big beta update doesn't close major issues known since forever (net code!) and doesn't bring improvements to existing features to get us closer to all the DDF promises.

 

[CMDR Charybdis]

Apologies for another wall of text, but some thoughts on putting some additional structure onto the front end of the QA process and crowd sourcing the testing.


I can see that there is a learning curve regarding MMO games that kfsone has highlighted, and like others have noted the change in the character of the forum as new players have joined in. Reading between the lines on the first week of the Beta release there was a team dedicated to supporting the production release and turning around the crash fixes to stabilise the release. Three releases in 3 consecutive days shows that this was there to kill off any initial flood of crash tickets and deal with the issues that occurred in the wild rather than the test lab.

The ticket process by its very nature is crowd sourced and brings with it a lot of enthusiasm that needs some channelling into existing processes. As with any QA process then the stages that need to be put in place are there to ensure that the higher cost teams are given higher quality items to work on.

To that end there is some potential untapped here for crowd sourcing to provide that additional QA control by guiding issues raised on the forum. This won't stop tickets being raised under all circumstances, but will help identify higher quality tickets. I'd suggest introducing part of the defect/issue management onto the forum in the following way:

The quality gate
A ticket submitted to Frontier with a link to a bug post on the forum has a higher priority than a ticket submitted without such a post.

The forum post has a "bug moderator" indicating that the issue should be reported to Frontier because it appears to be a new issue or a variant on an existing issue. There is also some measure of evidence to support the issue - it has been reproduced or there is additional evidence available.

To separate feature requests from bug reports.

A sub-forum per software release
The sub-forum acts a focus for bug reporting. One post per bug found by the community is the goal, duplicates closed.

One "maintained thread of reported bugs" submitted under that version (a one stop place to review against submitted bugs). This is community maintained but by a smaller group (see bug moderator), linking to the submitted post.

The release note copied into the forum as a reference, as an accumulation if there are earlier related versions.

Community members can view this forum and see if their issue has already been reported or if they've found something new to report.

Bug moderators
A team of "bug moderators" with limited moderation type powers to help manage the influx of posts and help support higher quality submissions to Frontier.

The bug moderator can check for duplicate issues already reported, advise/tweak the write up of the defect before it is submitted to Frontier, and help confirm/reproduce the bug (perhaps with a YouTube video if needed).

The bug moderator has limited power to lock a thread in that forum and to edit locked threads (to update the "maintained thread of reported bugs"). This does need some policing.

Possibly a private forum to discuss bugs and raise issues up to Frontier management (it'll be much quieter there).

Existing moderators
Support the activity by working with the bug moderators: moving posts into the correct sub-forum (e.g. defects in, feature requests out). Helping the bug moderators in a supporting and mentoring role.

Frontier Forum manager
To create the new forum with each release, and close off older forums as that version of software is no longer in the wild. Some care here as locking a forum too early means conversations in-flight need to be moved to the newly created forum for the latest release. No bad thing, it can encourage the issue to be retested on the latest release.

Also some updates into the ticket support process to point back into the forums so tickets raised know that there is a community process in place.

 

[CaptainKremmen]

It will be quite the challenge for FD, that's for sure.

Yes, there is a very good case for more in-game/QA infrastructure to be built up.


I should note that the following comment really resonates with me:

Yes and the fact that the builds that we have are in such great shape says a good deal about their QA process. Take a look at SC and see where they are at with over ten times the funding. The Frontier folks have been knocking this whole project out of the park in terms of what they are delivering. I have seen retail games in far worse shape then Elite is right now. Anyone play X3 (or any of the X series for that matter) on release? They need to be focused on getting the game done and the features in and that appears to be just what they are doing. They should not be dealing with people who have bugs wipe them out or other customer service issues.

The overall "togetherness" of their build impresses me no end. This means they aren't all running around the office with their hair on fire.

And a dev team that's not in desperate fire-fighting / siege-mentality mode is one that is at least capable of planning for the future.

 

[MrBungle]

I think people here would more than likely be completely unaware of what FD are doing with/their plans around the QA/customer service areas given few of us actually work for the organisation (and that's not to disrespect the opinions expressed by the OP/others re: this issue; on the contrary, these are good things to discuss ), so it might be a case of 'jumping the gun' a little with talk of a 'Titanic' situation re: QA/CS.

I'd be keeping my powder dry & see how it pans out, remember, we're a long way from release & I'd be very surprised if FD aren't already in the process of implementing whatever QA & CS plans they have for the finished game I.E I'd be inclined to believe they're ahead of the perception some people have re: these areas rather than being 'behind the eight ball' (but of course, like others, I have no direct knowledge of this).

Edit to add - Also, they might have some teething problems with these things. I hardly think that's a cardinal sin or the end of the world. This is an enormous project, they're allowed to make mistakes.

 

[StoneFeet]

This is a interesting thread given recent events i saw Frontier say they are hiring new support staff but it is late in the day for that now. Takes time to train support staff should have been done long ago.