FD QA hang your heads in shame
- Thread starter Excalibrates
- Start date
Coming from Mechwarrior Online I must say the QA process here is superior in every way. But, even then, sometimes QA does questionable things. I spent a lot of time making a video of the cargo drone exploding bug plus getting to the bottom of the problem, the cause and what makes it go away, posted it up in the support forum and QA has ignored it for days while happily replying to everything else that was reported.
https://forums.frontier.co.uk/showthread.php?t=156635
It is... irritating to say the least. It is an obscure bug that affects everyone who uses them (miners, pirates, traders) that royally breaks the cargo drones--you'd think there'd be interest in this issue. Nope. All that hard work flushed down the toilet.
Edit: Finally a response.
Yay. So back to my first comment--QA isn't that bad at all here, especially coming from other games where it is farrrrrrrrrrrrrrrrrrrrrr worse.
https://forums.frontier.co.uk/showthread.php?t=156635
It is... irritating to say the least. It is an obscure bug that affects everyone who uses them (miners, pirates, traders) that royally breaks the cargo drones--you'd think there'd be interest in this issue. Nope. All that hard work flushed down the toilet.
Edit: Finally a response.
Yay. So back to my first comment--QA isn't that bad at all here, especially coming from other games where it is farrrrrrrrrrrrrrrrrrrrrr worse.
Last edited:
No need to be nasty at the QA and development teams.
Having said that, I do agree something is up. It's like there is no unit tests or regression testing. This is hard to do for UI stuff (and for the most part, the game is actually just like a web client - it's just a fancy UI layer on top.) but for the code underneath in the controllers and models that calculate this stuff it seems like something is up.
I don't know if Frontier use a Continuous Integration server and test suite, but if they do it's not doing it's job.
Having said that, I do agree something is up. It's like there is no unit tests or regression testing. This is hard to do for UI stuff (and for the most part, the game is actually just like a web client - it's just a fancy UI layer on top.) but for the code underneath in the controllers and models that calculate this stuff it seems like something is up.
I don't know if Frontier use a Continuous Integration server and test suite, but if they do it's not doing it's job.
Last edited:
Yup, I don't understand why people are angry either. Guys, things will get better when the game hits release.
Well said OP, powerplay so far has been a massive detractor from the game due to its premature release as a way to try and placate PC players before the E3 announcement. Not just massive amounts of bugs and plainly broken features upon release, but subsequent patches bringing back bugs that were fixed in previous patches, not fixing what they were supposed to in the first place, complete lack of meaningful documentation that isn't open to interpretation, wiping of hours of players work etc etc, the list is huge.
2 weeks so far for PP and its arguably more broken and buggy now than it was 2 weeks ago, no way can any dev look at PP and the documentation we have available and say "yeah, im pleased with that, they have everything they need"
2 weeks so far for PP and its arguably more broken and buggy now than it was 2 weeks ago, no way can any dev look at PP and the documentation we have available and say "yeah, im pleased with that, they have everything they need"
I've been doing software development for over 25 years. I think the problem here is that they have a hard schedule in place. So, at that point in time, they need to have such and such out of the doors, no matter if it's fully tested or not. This or the head of QA should be fired. As someone else said, if I had been providing such buggy software, I would have been long without a job...
- - - Updated - - -
How many patches have there been since PP release? 5? 6? This is a sign right there that there are a lot of bugs that need to be fixed. I'm not sure I understand your comment. You must have been sleeping under a rock for the last couple of weeks.
I don't think switching to git, mercurial, TFS, alienbrain or perforce is necessary. Svn works perfectly well, it just seems to be their processes that are inefficient.
That and a lack of unit tests
That and a lack of unit tests
Last edited:
I have been around the block many times. My comment stems not from the number of issues ('bug' is so overused it has lost all meaning) but by the ridiculous response from the posters. Stamping your feet, and posing like martyrs gets old, and I feel compelled to say something.
Talking about issues is good and proper. Riding your high horse through a forum and making judgements on other people, and their jobs is childish, and you should know that.
I know it's hard to deliver perfect solutions under tight deadlines, but I don't think the issue is just hard schedules or tough management. They probably do some manual testing, but a regression like the repair cost bug (if it's back again, haven't played yet) seems to me they do not even have unit tests for their game mechanics in place or they don't write unit tests after fixing bugs.
That is very scary to me, since everything is a huge black box to the developers working in the code at that point. Especially in a complex game like Elite with different interacting components, you want the main components/mechanics tested and giving green lights every build.
How it works for me: if a bug is found by our QA team or reported by a customer, I fix it and at least write unit-tests (or automated test) to make sure regressions do not happen. If the unit-tests fail, QA never even gets to see the build. It's as simple as that. Developers have to fix the code or fix the test (if they changed the code). In any case, a fail is a big red flag: it means untested or broken code.
Our QA has the power to block releases and that should be the norm in a lot more software companies. It's a pain to miss a deadline, but it's worse to have all your customers call you and having to fix everything in a rush - introducing even more bugs - and looking amateurish in front of your whole customer base. Hotfixes are a rarity in my case, and customers are happy with a functional product (although bug-free is never possible). We even have 24/7 support... and we only build forums
Last edited:
I know it's hard to deliver perfect solutions under tight deadlines, but I don't think the issue is just hard schedules or tough management. They probably do some manual testing, but a regression like the repair cost bug (if it's back again, haven't played yet) seems to me they do not even have unit tests for their game mechanics in place or they don't write unit tests after fixing bugs.
That is very scary to me, since everything is a huge black box to the developers working in the code at that point. Especially in a complex game like Elite with different interacting components, you want the main components/mechanics tested and giving green lights every build.
How it works for me: if a bug is found by our QA team or reported by a customer, I fix it and at least write unit-tests (or automated test) to make sure regressions do not happen. If the unit-tests fail, QA never even gets to see the build. It's as simple as that. Developers have to fix the code or fix the test (if they changed the code). In any case, a fail is a big red flag: it means untested or broken code.
Our QA has the power to block releases and that should be the norm in a lot more software companies. It's a pain to miss a deadline, but it's worse to have all your customers call you and having to fix everything in a rush - introducing even more bugs - and looking amateurish in front of your whole customer base. Hotfixes are a rarity in my case, and customers are happy with a functional product (although bug-free is never possible). We even have 24/7 support... and we only build forums
Another practice is to not hoard code changes and assets for a huge "party" release down the road. Every single time a company does this....something goes wrong.
Maybe Bob was working on a script that seemed to be just fine..and then got moved onto something else. 10 jumps and 4 weeks later hes totally forgotten what his first project was...or where it is in the giant ball of additions. And then nobody understands why when you click on a tree a pink and black checkered mailbox goes flying across the screen.
I find it questionable and saddening that FD has the capacity to "hotfix" at a moments notice...and can throw out patches daily if need be ( as they have been doing for the last 2 weeks ), but when it comes to making content for the game ( outside of fundamental changes that need to be brought in together, like powerplay ), they opt to just wait to release it with their "big plan" later. I've seen so many companies, game dev teams, design planners, etc. ...and modders that get into "feature creep" with their project , and it bites them in the ass.
Got something done? Push it down the line and out the door. THEN if something crops up...your chest of toys is 1/10th the size to search for..and you don't risk blowing up your trade district because you wanted to increase the size of your gun model by 10%.
IMO the release of 1.3 and subsequent releases has demonstrated that the end to end quality of the product being delivered does not meet my expectations. I was locked out of the game for 3 days. There have been patch releases which have managed to break things that look unrelated (it a real symptom of teams being pushed too hard and quality assurance processes being cut). FD should be making more effort to ensure that the quality of the production releases is of a higher quality, and stop what feels like panicked, knee-jerk reactions to introducing defects. Hopefully for 1.4 we will see a much longer alpha/beta cycle so that the production release is much more stable - so it doesn't need 5(!) patch releases within a couple of weeks of release.
Where the blame lies I don't know, but I know in similarly malfunctioning software teams I've worked in, those symptoms have been a consequence of management and executive decision making, and the test leads have always been arguing that it wasn't ready to go.
TL;DR - FD - not enough quality in the release process. Do better next time.
Where the blame lies I don't know, but I know in similarly malfunctioning software teams I've worked in, those symptoms have been a consequence of management and executive decision making, and the test leads have always been arguing that it wasn't ready to go.
TL;DR - FD - not enough quality in the release process. Do better next time.
Yup and they could have spent the time in between fixing the existing game
What I dislike most about all this is the implication that Frontier aren't really in the business of bug checking themselves, that they're relying more on the community to do this.
What this means is that there's likely a whole class of bug that exist in the game that will never be fixed. Bugs that aren't visible to the user or immediately apparent. =(
Off the top of my head for example, a certain type of mission that never appears but is important somehow to gameplay, players would never know.
It's this that I don't like, relying on users checking for bugs in a game where the mechanics are intentionally opaque means players end up with a broken game without even knowing. Players don't get the intended game experience and neither the players or the developers actually know about it. End result is frustration for all concerned.
What this means is that there's likely a whole class of bug that exist in the game that will never be fixed. Bugs that aren't visible to the user or immediately apparent. =(
Off the top of my head for example, a certain type of mission that never appears but is important somehow to gameplay, players would never know.
It's this that I don't like, relying on users checking for bugs in a game where the mechanics are intentionally opaque means players end up with a broken game without even knowing. Players don't get the intended game experience and neither the players or the developers actually know about it. End result is frustration for all concerned.
Last edited:
This is what QA are doing.
Jim – Right now we've got our programmers and QA playing with us, which is really helpful, but I can’t wait to have more players out there with us. Balancing is ongoing, and the feedback we get internally is completely different from the feedback we’ll get from all the players on Xbox Live. I think Xbox One is absolutely the best place to test the game, not necessarily because console players play differently, but because they'll be new to the game. It'll be a fresh perspective.
Jim – Right now we've got our programmers and QA playing with us, which is really helpful, but I can’t wait to have more players out there with us. Balancing is ongoing, and the feedback we get internally is completely different from the feedback we’ll get from all the players on Xbox Live. I think Xbox One is absolutely the best place to test the game, not necessarily because console players play differently, but because they'll be new to the game. It'll be a fresh perspective.
IMO, so long as this is part of an open, voluntary, alpha/beta process its fine, otherwise spot on.
Completely agree with the OP. Everything has the feel of being rushed out and comments about FD being a small operation have nothing to do with this. As a fan of DCS I have been waiting for the release of DCS World 2 and several other expansions. Last week they mailed customers apologising and explaining that the releases would be further delayed. Disappointed, yes, but I completely agree with their approach and will patiently wait. FD you could learn a lot from this crew who put quality and customers first.
Please FD concentrate on fixing what is already in game before adding any more broken crap on top of what we already have to contend with. I love ED but it is this that will eventually drive me from this game.
Please FD concentrate on fixing what is already in game before adding any more broken crap on top of what we already have to contend with. I love ED but it is this that will eventually drive me from this game.