The failure of this update just reinforces the fact that FDev need to have a period of beta testing before rolling out updates. As it is, the players are beta testing on live servers after an extended period of testing "in-house" yesterday which delayed the update. There doesn't appear to be enough staff to handle the issues being raised. Just because something "works" in-house doesn't mean that it'll "work" when CMDRs get their hands on it.
Bring back beta testing
- Thread starter colinhype
- Start date
Yea. I am kind of amazed here. Only a few days ago i still defended FDs QA department, that they generally do a comparatively good job. Sure not perfect, but for the size and scope of the game, quite good.
Now we have this upgrade. I mean, there's always a few things which can be seen as low enough priority to not delay the patch. There always can be one or another unintended side-effect of a last minute fix, which QA didn't get the time to look at any more. But the number of problems of this patch is too high and the scale of some of the problems too big.
I mean, when i read that it doesn't start any more on consoles... say what you want about consoles, but they have one big positive for a developer: you know which hardware there is. No big variance. And just starting the thing and see if it runs, before you do the rollout, is not impossible.
So really: bad QA job this time.
Now we have this upgrade. I mean, there's always a few things which can be seen as low enough priority to not delay the patch. There always can be one or another unintended side-effect of a last minute fix, which QA didn't get the time to look at any more. But the number of problems of this patch is too high and the scale of some of the problems too big.
I mean, when i read that it doesn't start any more on consoles... say what you want about consoles, but they have one big positive for a developer: you know which hardware there is. No big variance. And just starting the thing and see if it runs, before you do the rollout, is not impossible.
So really: bad QA job this time.
Yup - let QA test the new stuff, then let us do the regression tests. One day of beta would be enough for 90% of issues. Instead we now have days and days of people finding the same thing over and over and over again...
Deleted member 38366
D
Yep, absolutely....
And most importantly, give QA vast authority and the ability to override Management. Let them do their job, thoroughly.
Otherwise, they serve no purpose other than "damage control" which helps nobody. Just look around the Forum... this is bad. And it doesn't have to be.
And btw. give an incentive for Beta Testers (Players) to make qualified Beta Tests & Reports and reward them. Pay them for their work and professional reports, for their time.
ARX for every confirmed Report (Bonus for "1st Discovery") and useful participation is now an Option. And don't be shy.
That combination IMHO would work like a charm and help accelerate things significantly.
And most importantly, give QA vast authority and the ability to override Management. Let them do their job, thoroughly.
Otherwise, they serve no purpose other than "damage control" which helps nobody. Just look around the Forum... this is bad. And it doesn't have to be.
And btw. give an incentive for Beta Testers (Players) to make qualified Beta Tests & Reports and reward them. Pay them for their work and professional reports, for their time.
ARX for every confirmed Report (Bonus for "1st Discovery") and useful participation is now an Option. And don't be shy.
That combination IMHO would work like a charm and help accelerate things significantly.
I absolutely agree with this "bring back Beta testing" suggestion. 🕹
It's a shame we even have to have this thread. Requesting something that should actually be a pre-requisite.
Nope.
QA finds bugs and reports them.
Management makes decision either to release the product in this state or not.
That's how respensibilities should be applied.
While not disagreeing, I think it’s worth bearing in mind that 3.3 had a long Beta, and the worst bug to occur post 3.3 launch didn’t occur in the Beta whatsoever.
So don’t expect that a Beta would be a panacea for all ills.
Also, worth bearing in mind that with more of a focus on quality, something else is going to have to have the option to slip. Descoping probably isn’t going to fly once feature lists have been announced so there’d need to be more flexibility in the release date, though potentially more slack could be built into the scheduling to cover that (- don’t really know enough about how FD works internally to comment on how much of a realistic option that is though).
So don’t expect that a Beta would be a panacea for all ills.
Also, worth bearing in mind that with more of a focus on quality, something else is going to have to have the option to slip. Descoping probably isn’t going to fly once feature lists have been announced so there’d need to be more flexibility in the release date, though potentially more slack could be built into the scheduling to cover that (- don’t really know enough about how FD works internally to comment on how much of a realistic option that is though).
As a tester who a few years ago single-handedly delayed a softare rollout by almost 3 months, i kind of disagree. Management very much tried to talk the problem away at that time.
Of course, the big difference is customer acceptance. In any "normal" environment, the test report goes to the customer and only after he accepts the reported issues, the delivery and rollout can happen. I guess sending a test report to all players and waiting for all of us to sign it off, that we are happy with the result, might get a bit complicated. So unfortunately for games there's no customer acceptance part.
That were the usual last-minute changes, which didn't see proper testing any more. Unacceptable for any critical software, all to common for games and low criticality software. The 3.3 beta still caught a number of issues. It still would've been better if the fixes would have been given a few last days of the beta, instead of directly rolling them out. So yes, a bit better management, giving it a few more days of testing, might have helped. But it was well within normal parameters. Also keep in mind that players were demanding that 3.3 was put life, as most of the glaring bugs were fixed before the last week of the beta. So there was time pressure from customer side.
This patch is a bit different in my eyes. The release date was given to us customers rather late, anyway. Would they internally have moved it back a week, just a week ago, nobody would even have noticed. No pressure from customer side. They could've done a proper job. So for this patch, i don't see why they didn't do it.
Last edited:
I actually PAID REAL MONEY for beta access once and all I got are just two lousy decals! Thereby I feel very much entitled now to state FD PROMISED me to have lifetime beta access, so where is it? Scam!!!
And no more public betas please, it was an exclusive feature once and should stay so forever!
That said, when it comes to ED, beta testing is of limited use obviously. We had an extensive one for 3.3, and it didn't help much. What FD are quite good at, though, is fixing things after launch. They're remarkably quick with this, and some things can only be found out in the live environment anyway (hello Adjucation server!), in particular integration stuff like the ARX or Store blackout. UI-related stuff is another issue, and inexplainable regression bugs (like the firegroup settings or module priorities resets) are too. They simply have too few resources and/or too eager timeframes, and no testing can really remedy this. I don't think FD themselves are happy with this.
O7,
And no more public betas please, it was an exclusive feature once and should stay so forever!
That said, when it comes to ED, beta testing is of limited use obviously. We had an extensive one for 3.3, and it didn't help much. What FD are quite good at, though, is fixing things after launch. They're remarkably quick with this, and some things can only be found out in the live environment anyway (hello Adjucation server!), in particular integration stuff like the ARX or Store blackout. UI-related stuff is another issue, and inexplainable regression bugs (like the firegroup settings or module priorities resets) are too. They simply have too few resources and/or too eager timeframes, and no testing can really remedy this. I don't think FD themselves are happy with this.
O7,
I paid extra to be a Beta tester and I feel unloved these days
I paid extra to be a Beta tester and I feel unloved these days
Perhaps we should Appeal to The Supreme Court?
OOPS Sorry Mods wrong button, meant to edit.
For me it's probably this ^^. How would a public beta have been linked to the live store? It just creates another migration back to the live server later, very untrackable if you start getting cumulative errors. Think we have to live with this one.
Dunno... looking back at what things were like on day 1 of the 3.3 Beta... I'd have to say that the 3.3 Beta helped a great deal.
The most critical stuff that occurred in the 3.3 beta all got fixed. Arguably, it could have done with being longer given the scope of that release, but as @Sylow says, people were saying 'release it now' well before the end of the Beta.
All the issues to do with massive loads on the servers following update release - Beta's not going to pick that kind of stuff up anyway.
And as per post higher up, the worst issue post 3.3 release wasn't happening in the Beta anyway.
It's not like the store links are working even now - but a few eyes on the game would have spotted:
- Engineering broken
- Remote engineering broken
- Mining broken
- Ugly Blue shop link
- Remote modules not showing engineering
Sure, QA should concentrate on new stuff - store & new start stuff, but we have a good idea of how the existing game is supposed to hand together - they should make use of their resources
Yes. Of course most likely not on full scope any more. But there should be a baseline test suite, which at least takes a short look on all "legacy" functionality. And even "start the game on one unit of every platform we support" and "do one piece of engineering" would've rang a number of alarm bells.
But wasn't it the April update the (re) introduced the 10CR restock bug? And that was visible just by looking at the menu. I struggle to defend them on either of those releases, but at least the April ones were mostly cosmetic.
The difference here is severity. In the present case, just starting things on any supported platform would have resulted in the test result "does not start, but crash and burn on some consoles". Which clearly is cathegory 1. Gamebreaking. Logical action: stop the rollout, bugfix and test again.
In contrast what did the 10 CR restock bug do? It looked a bit ugly. Made people click a few additional times, then shrug their shoulders and continue playing. That's cathegory 3: minor problem visible to the customer, fix if possible but don't delay the delivery.
I think I said that (At least that was my intention
)The only caveat is that we know if a bug is prominent, even if it is trivial, it will generate a lot of ill will. So I would say that the restock one is worth fixing before generating 2 weeks of repeat complaints. But certainly console crashes / sections of the game being broken are much higher priority.