No one at Frontier catches these bugs before an update is released?

[Old Duck]

I can't believe I'm defending bugs in patches, but to be fair, there are so many ways to play Elite these days that even with a team of testers I don't think they could possibly catch every bug. I was just thinking yesterday, "When was the last time I was mining?" That's radically different gameplay than my normal "defend installations in a shieldless stealth Cobra" routine. This thought popped in my head while unloading my fleet carrier of some old, dusty cargo, using my T-9 and advanced docking computer for the first time in well over a year. If the advanced docking computer was bugged, I wouldn't even had known about it until yesterday. Oh, and it's probably been even longer since I've landed on a planet (designated docking pads notwithstanding).

Now this is no excuse for ignoring reported bugs for months or even years, that's an entirely different discussion, but I cannot imagine how long it would take for me alone to try every permutation of gameplay with all the available ships and module configurations available, and I'm not even talking about Odyssey which adds who knows how much extra complexity to the game. But regarding this latest update, I'm actually kinda impressed how quickly Frontier pushed out a patch (and then a patch for the patch) to fix some of the more critical issues.

 

[Biff Stryker]

I think I may be part of the blame here, you see I am pretty much an expert at testing updates and finding bugs. I offered to help once totally for free of course and I think they kind of expect it every time now. My humble apologies to the forum community.

 

[Stealthie]

I can't believe I'm defending bugs in patches, but to be fair, there are so many ways to play Elite these days that even with a team of testers I don't think they could possibly catch every bug. I was just thinking yesterday, "When was the last time I was mining?" That's radically different gameplay than my normal "defend installations in a shieldless stealth Cobra" routine. This thought popped in my head while unloading my fleet carrier of some old, dusty cargo, using my T-9 and advanced docking computer for the first time in well over a year. If the advanced docking computer was bugged, I wouldn't even had known about it until yesterday. Oh, and it's probably been even longer since I've landed on a planet (designated docking pads notwithstanding).

I've often said the same thing myself.

The awkward truth, though, is that most of the bugs tend to be things that are immediately obvious within a few minutes of playing the game... in any location, in any ship, in any circumstances.

If, for example, there was a bug whereby scanning a Guardian Beacon in A Federal Gunship with an SLF bay fitted caused the game to crash, I could understand that not getting spotted during testing.
Even if work had been done on the Guardian Beacons, it'd ne a big ask to expect FDev to test it with every ship, equipped with every combination of modules, to ensure that they all function correctly.
It's perfectly reasonable, in that situation, for that bug to only become apparent when somebody in an FGS fitted with an SLF bay does rock-up at a Guardian Beacon and the game crashes.

OTOH, when you've got bugs like the camera controls ceasing to function or (previously) the cockpit HUD becoming corrupt if you land while using the external camera, those things are likely to be discovered within minutes of playing.
I mean, they introduced the Mamba with it's dodgy landing gear.
Surely after putting a new ship in-game, it's reasonable to carry out a fairly thorough test of ALL aspects of the new ship?

So, erm, yeah.
While it's fair to say there's a lot of different things to check to ensure no bugs are present, FDev have often missed glaringly obvious bugs too.

 

[tolm]

Ack! No, not this again!

Quality Assurance and Quality Control are two (almost) completely separate things... even though a lot of companies do lump them in together.

Quality Assurance is responsible for ensuring that all the systems and procedures used in a business comply with legal and corporate standards.
Fundamentally, it has NOTHING to do with ensuring the quality of the end-product.
It's all about ensuring that everything used in every phase of the process complies with any applicable legislation and complies with regulations imposed by the business.

In terms of software development (an industry which I have no experience of) it will be responsible for stuff like ensuring that PCs and workstations are all build properly, that they have the correct versions of the correct software on them and that the people using them are adhering to corporate rules regarding how files are modified, saved and backed-up etc.
It is, basically, all about ensuring that all the employees have the correct tools, made to the correct standard, and that they're using them correctly.

Quality CONTROL, OTOH, is what's responsible for ensuring that the end-product isn't a steaming pile of poop.

Back in the late 1980s, QA became a big deal, with advocates claiming that sufficient QA would mean that you could, theoretically, employ chimps to do the work because the QA would ensure the quality of the end-product.
This was/is largely true in production-line work where, previously, skilled workers might be required to make the difference between producing, say, a car that was garbage and one that was acceptable.
These days, a production line doesn't need skilled workers because there'll be systems in place that mean, as long as the workers do what they're supposed to, the quality of the end-product is, erm, assured.

Thing is, though, that QA doesn't really help in creative industries.
In software development, for example, you can ensure that everybody's using the correct hardware and software and that they're following the correct procedures for file-handling but you can't really use QA to ensure that the code they create works in the intended way.
You can't use QA to ensure that a dev's brain is only capable of developing bug-free code.

I hate having to defend QA because, in my experience, it's one of the most irritating, meddling, parts of a business but it's NOT responsible for the bugs in ED.
You need to talk to the QC department about that.

Not disagreeing with your analysis of the English language terms but just Google “QA Testing” and you’ll see that the world of Software Development does not agree …

Now, I personally believe it’s just down to semantics when you’re in the Software business as opposed to producing physical products …and the question is what you consider to be “The Finished Product”?

If “The Finished Product” is what is delivered to your Testing Team, then they are arguably applying “Quality Control” to detect faults in the product after it’s been completed.

But if you consider “The Finished Product” to be the software once it has been released into the Production environment (which is certainly true for the companies that I’ve developed software for), then what you deliver to your Testing Team is just part of that development process and hence it’s a “Quality Assurance” exercise to prevent faults making their way into “The Finished Product” when it is released.

 

[Stealthie]

... just Google “QA Testing” and you’ll see that the world of Software Development does not agree …

If I was to google "QA Testing" and find anything other than procedures for auditing QA systems it would only help show that people often conflate QA with QC.

... then what you deliver to your Testing Team is just part of that development process and hence it’s a “Quality Assurance” exercise to prevent faults making their way into “The Finished Product” when it is released.

That's kind of like saying that because a factory bolts the wheels to it's cars in the paint shop, you want to call wheel fitment part of the paint process.

It's actually a bit disappointing to read a comment like this because I was trying to explain how QA is specifically related to the processes a business uses rather than the products they might create.
Still, with hindsight, I recall that a lot of people in businesses had the same confusion back when QA first arrived so I guess it shouldn't be surprising when people who aren't involved struggle with it.

 

[tolm]

If I was to google "QA Testing" and find anything other than procedures for auditing QA systems it would only help show that people often conflate QA with QC.



That's kind of like saying that because a factory bolts the wheels to it's cars in the paint shop, you want to call wheel fitment part of the paint process.

It's actually a bit disappointing to read a comment like this because I was trying to explain how QA is specifically related to the processes a business uses rather than the products they might create.
Still, with hindsight, I recall that a lot of people in businesses had the same confusion back when QA first arrived so I guess it shouldn't be surprising when people who aren't involved struggle with it.

Wasn’t looking for an argument … my job is in software development and I’ve worked with many QA teams who assure the quality of the finished product. I was merely trying to offer an explanation of why the two terms sometimes get interchanged in my industry because of the difference between a software product vs a physical product.

But sure, be disappointed if you want.

 

[galahad2069]

Not disagreeing with your analysis of the English language terms[...]

The acronym you choose is pretty irrelevant, the sad fact is they simply don't have any kind of testers.
They don't test their updates, not even once, that's blatantly obvious.

Even if they had only a single person person checking the update ingame before release, these kind of bugs could never happen:

Source: https://www.youtube.com/watch?v=Zt1O0BzMaS8

It literally does not take more than 2 seconds to discover.

 

[Muzz UK]

Most all of the bugs I have seen in EDH/O could have been found, had someone in-house played this game after applying each update.

I'm an old software developer. In those days, we at least tested the update in-house to make sure it addressed the problem, without breaking anything.

Here we are, a DLC and 12.01 updates and it seems like each update just fixes what the previous update broke. Most all of the bugs being reported to this forum find their way immediately into updates, while bugs reported to Issue Tracker seem to be included if they appear here on the forum first.

We are Frontier's update testers folks. We have become so accustomed to accepting beta code here, I doubt FDev bothers to test anything in-house anymore. They just give us an update hot off the press, we test it, report bugs to this forum or issue tracker and the next update is released with those bug's "fixed?" for testing again.

WE, here on this forum, are the "FDev QA QC Team", Folks.....

EDIT: Correction, Per @Stealthie's post. Sorry, I mis-spoke.

I don't want to be that guy but maybe modern game development isn't as simple as you think it is?

 

[ZeeWolf]

This is my third decade working in software, I've certainly put together testing/QA teams and been a line manager for them, I've never considered any differentiation in the terms indeed people that have filled the roles would have been QA engineers or testers with no discernable difference.

The testers have a frankly thankless task of following scripts/plans over and over and managing automation tools, filling out issue tickets with lengthy instructions on how to recreate bugs that developers have to prioritise around adding new features. Frontier absolutely have a QA team and process, as will any game with a studio based dev team. There are QA roles advertised on their website and marketing "meet the team" videos etc.

ED has a vast array of gameplay paths, open world games in general are maintenance/dev nightmares most game companies avoid, preferring to stick to scripted narrative and limited arena stuff precisely because they can be produced on a reasonable budget with static servers and have less unpredictable problems that can cause embarrassment.

Factor in a procedural galaxy that produces big numbers of related consistent variation that your trying to do stuff like route finding against when its not data that actually exists in any conventional way and online cloud infrastructure that needs to scale on demand and the unpredictability of the internet and difficulty in reproducing anything "in house" that simulates the scale and inconsistency of clients working across multi regions with data synchronising real time "twitch" gameplay etc.

That said I'm not saying that having a big online space trader is an excuse for poor quality or extensive bugs that hinder enjoyment. Its astonishing that Hello games manage to produce a similarly scoped title with a small studio.

 

[EdzUp]

Having actually done big projects where there are dozens of code files and megabytes of data etc, i can understand where some of the bugs get missed. Some of the more groundbreaking bugs may not show up on fdev hardware or can be caused by an obscure driver group or revision all of this might not be used by fdev.

 

[Caydran]

I don't want to be that guy but maybe modern game development isn't as simple as you think it is?

Well, today, that is certainly true. Other software applications aside, gaming code comes off the desks of the developer's at amazing speed these days. No time for in-house QA/QC, just build/versioning designations for source control. Then DLC's come in, adding more features before the base code ever moves from "beta" to "release candidate".

Updates seem to be just cumulative fixes for DLC's, with base code receiving a "hot-fix" every now and again, before the inevitable base code merge into the current DLC.

Of course you know that other space game still in "alpha", as an example. My question to Frontier would be, will ED on the whole ever be declared more than just a "beta" with "alpha" DLC's, or has the architecture of industry wide game code development and code QA/QC changed so much, that "Finished Product" and "Release Candidate" are now irrelevant for gaming software?

Will ED always be an unfinished product always being updated, until it reaches end of life?

 

[Mogen Teras]

I can't believe I'm defending bugs in patches, but to be fair, there are so many ways to play Elite these days that even with a team of testers I don't think they could possibly catch every bug. I was just thinking yesterday, "When was the last time I was mining?" That's radically different gameplay than my normal "defend installations in a shieldless stealth Cobra" routine. This thought popped in my head while unloading my fleet carrier of some old, dusty cargo, using my T-9 and advanced docking computer for the first time in well over a year. If the advanced docking computer was bugged, I wouldn't even had known about it until yesterday. Oh, and it's probably been even longer since I've landed on a planet (designated docking pads notwithstanding).

Now this is no excuse for ignoring reported bugs for months or even years, that's an entirely different discussion, but I cannot imagine how long it would take for me alone to try every permutation of gameplay with all the available ships and module configurations available, and I'm not even talking about Odyssey which adds who knows how much extra complexity to the game. But regarding this latest update, I'm actually kinda impressed how quickly Frontier pushed out a patch (and then a patch for the patch) to fix some of the more critical issues.

I completely get what you're saying, but we've seen time and time again how updates seem to introduce bugs that are widespread enough that they're reported en-masse in forums and subreddits, sometimes even being bugs that render core parts of the game unplayable. I mean this last patch made the entire text box unusable for basically everyone.

I get that code is hard but there is no way they couldn't have caught some of these issues considering how every player ends up experiencing them.

 

[cmdr milesred]

never forget the difference between a store bought off the shelf, long ago finished game and this, a live, daily changing, day after day for quite a few years now game.
a game that has so many components to it, that many are likely still loaded with bugs because no-one is using that area of code yet..aka the continual stories yet to be explored further.
as it sits, IMO, it has been quite a ride and still is.
bugs are forgivable and expected. bad planning however is another story. a few releases were extra bad due to the failure to plan properly.
I think ED is still ahead and far more interesting and fun than any of the other space games around trying to do their version of a space game.

as for the noise on the topic from this and all other sites, also normal and to be expected with a popular game with a lot of passionate cmdrs.

 

[Jack Winter]

Will ED always be an unfinished product always being updated, until it reaches end of life?

DBOBE is on the record as saying that he prefers to release a working game to trying for perfection and not releasing at all, and of course it has make a profit. I think this is probably the reason that ED is a game for close to 10 years while that other game is a 10 years demo..

Personally I think in a game like this it's impossible to catch all the bugs before release, though I'd suspect that they could do a better effort at it..

 

[aRJay]

I am reminded of a planet surface issue (something to do with dark squares?) from Horizons a few years ago that was widely seen and reported by players and acknowledged to have been seen by CMs playing at home but could not be replicated by FDev to isolate and fix until it was found that the coding tools on the machines in FDev towers blocked the bug so it wasn't present when they tried to find it.

 

[Darkfyre99]

Most all of the bugs I have seen in EDH/O could have been found, had someone in-house played this game after applying each update.

I'm an old software developer. In those days, we at least tested the update in-house to make sure it addressed the problem, without breaking anything.

Here we are, a DLC and 12.01 updates and it seems like each update just fixes what the previous update broke. Most all of the bugs being reported to this forum find their way immediately into updates, while bugs reported to Issue Tracker seem to be included if they appear here on the forum first.

We are Frontier's update testers folks. We have become so accustomed to accepting beta code here, I doubt FDev bothers to test anything in-house anymore. They just give us an update hot off the press, we test it, report bugs to this forum or issue tracker and the next update is released with those bug's "fixed?" for testing again.

WE, here on this forum, are the "FDev QA QC Team", Folks.....

EDIT: Correction, Per @Stealthie's post. Sorry, I mis-spoke.

There's an easy solution: find out exactly what hardware, software, and drivers quality control team uses when testing internal development builds, and you'll probably never encounter a game breaking bug again. After all, it worked for me for about eight years. As it turns out, I recently discovered that my old computer was almost identical to David Braben's old one, which he still uses to test the performance of games on "older machines." Needless to say, any QC team worth their salt would make sure they had a copy of their CEO's computer to test software performance on.

When it comes to PC gaming in particular, the huge variety of configurations out there means that once a build leaves the lab, bugs that are the result of a particular combination of hardware, software, and drivers are going to be encountered for the first time by the player base. It's inevitable. Frontier doesn't have the resources to test every potential combination out there, but it is in their power to allow the motivated part of the player base to test release candidates in advance, so that when an update goes live, the worst ones encountered in the wild will be identified and fixed.

Frontier has repeatedly failed to allow for adequate player testing over the years, and even worse continues to treat "alpha access" as a marketable coming attractions preview, rather than a true testing phase. This has been a detriment to the game.

 

[Caydran]

Frontier has repeatedly failed to allow for adequate player testing over the years, and even worse continues to treat "alpha access" as a marketable coming attractions preview, rather than a true testing phase. This has been a detriment to the game.

So True. It like Frontier never wants to declare this product finished, in order to keep the marketing dream alive.

Here is a game you can play your own way and be the first to test new features.

 

[Stealthie]

I completely get what you're saying, but we've seen time and time again how updates seem to introduce bugs that are widespread enough that they're reported en-masse in forums and subreddits, sometimes even being bugs that render core parts of the game unplayable. I mean this last patch made the entire text box unusable for basically everyone.

I guess some of these bugs provide an insight into the scary nature of ED's code.

I mean, if you're working on an update to, perhaps, the map screens, you might test it all out and decide that they're all working properly without realising that the code related to the maps also has it's hooks into the camera suite and, as a result, something that you've done to the maps - which works perfectly - has screwed up the camera suite.

Course, that doesn't excuse not finding bugs that, literally, half an hour of general playing would reveal but it does demonstrate the sort of "tightrope" that the dev's are walking when they update anything.

 

[Jack Winter]

I am reminded of a planet surface issue (something to do with dark squares?) from Horizons a few years ago that was widely seen and reported by players and acknowledged to have been seen by CMs playing at home but could not be replicated by FDev to isolate and fix until it was found that the coding tools on the machines in FDev towers blocked the bug so it wasn't present when they tried to find it.

A heisenbug, one of the nastiest.. https://en.wikipedia.org/wiki/Heisenbug

 

[Greasetrap42]

Most all of the bugs I have seen in EDH/O could have been found, had someone in-house played this game after applying each update.

I'm an old software developer. In those days, we at least tested the update in-house to make sure it addressed the problem, without breaking anything.

You are assuming that FDev isn't aware of bugs before release. Or the possibility of bugs and issues in known partially tested elements.

In my industry (industrial controls) it is common to implement projects with known bugs. Projects are so large that you would never release anything if you waited for all the bugs and issues to be fixed. Industrial software often comes with a list of "known issues", some of which might be quite significant. Control systems will be missing important features. Its known and hopefully gets dealt with later. Often because of time constraints it doesn't. I'm not saying this is good ,its just how it is. In this case QA is more involved than QC. Proceedures, regulations, and commitments to determine what is acceptable and what isn't.

I'm not defending the practice of releasing stuff with known bugs. Just stating that it is common.