Has ED got to the point where experienced players do proper beta testing ?

[A D]

OK, title is a little controversial, but no criticism intended of any player who helped with the Chapter 4 public beta testing.

Also note that I'm not bagging FD either. The game overall gets better and better (despite complaints about this, that or the other) and there are some features of Chapter 4 I really like (thankyou FD). However, in a way that is the problem. Can see from Chapter 4 that some of the wheat has got mixed up with the straw. So really public beta testing is not the way to go, except maybe as a final "stress" test.

ED is now a mature game and has 100's of subtle features in outfitting, missions, combat, trade, mining, exploration, powerplay, BGS and factions, etc. that experienced players just know - stuff that surprises me and I've been playing for over two years. All good, but unless you have played ED for 100's of hours you don't know what you don't know - ignorance is bliss. So IMO FD really needs to consider beta testing with a group of very experienced players under NDA's who do actually test features. Can be volunteers, maybe a paint job and decal to proudly display, but otherwise for love. Just so long as they are experienced and agree to actually test things in a critical, thorough and thoughtful way.

TL;DR - 'Bout time FD started using experienced players under NDA's as rigorous beta testers in the development process.

 

[babelfisch]

Can you be more specific about the actual issues you see with the current approach? An example maybe?

 

[CMDR CPT. Boris]

they probably have a team in their offices doing that
but as you pointed out, there are 100's of subtle features in outfitting, missions, combat, trade, mining, exploration, powerplay, BGS and factions.
so there is no better way to let as many people as possible to go out there to hunt the bugs, especially if some have paid for it or doing it for free

 

[Stigbob]

If they are under NDA's you wouldn't know if they already were.

 

[galahad2069]

Can you be more specific about the actual issues you see with the current approach? An example maybe?

Immortal bugs maybe?

BTW, have they fixed the "navlock/wing signal drops you 1000s of kms from your wingmate above the far side of a nearby planet" issue already?

 

[Rubbernuke]

powerplay.

I submitted a bug that broke Powerplay during beta, and for a short while was in the release. I don't really think FD are using the playerbase as true beta testers and instead garnering opinion on features.

 

[A D]

Can you be more specific about the actual issues you see with the current approach? An example maybe?

Sure, public beta testing is a good sledge hammer. It may produce some good info around e.g. networking, instancing, etc. However, on finer detail it probably produces too much "straw". If you look at the bug list for Chapter 4 there are literally hundreds of reports, not prioritized or focused... pretty random actually. Beta testers comprised of experienced and dedicated players would be asked (and be able) to test specific things and would probably provide good consistent reporting as well as some insights that a 10 hr or even 100 hr player could not. I'm pretty sure for example that experienced explorers who were specifically asked to take the FSS mode and codex through their paces would not have missed some of the stuff I'm seeing on release.

Hope this answers your question, cos I'm really not having a go at anyone. Just my opinion, that's all. : )

 

[Aakkonen]

If they are under NDA's you wouldn't know if they already were.

Just like special/black operatives. You will never know they even existed ;}

 

[Ian Skippy]

OK, title is a little controversial, but no criticism intended of any player who helped with the Chapter 4 public beta testing.

Also note that I'm not bagging FD either. The game overall gets better and better (despite complaints about this, that or the other) and there are some features of Chapter 4 I really like (thankyou FD). However, in a way that is the problem. Can see from Chapter 4 that some of the wheat has got mixed up with the straw. So really public beta testing is not the way to go, except maybe as a final "stress" test.

ED is now a mature game and has 100's of subtle features in outfitting, missions, combat, trade, mining, exploration, powerplay, BGS and factions, etc. that experienced players just know - stuff that surprises me and I've been playing for over two years. All good, but unless you have played ED for 100's of hours you don't know what you don't know - ignorance is bliss. So IMO FD really needs to consider beta testing with a group of very experienced players under NDA's who do actually test features. Can be volunteers, maybe a paint job and decal to proudly display, but otherwise for love. Just so long as they are experienced and agree to actually test things in a critical, thorough and thoughtful way.

TL;DR - 'Bout time FD started using experienced players under NDA's as rigorous beta testers in the development process.

Open Beta is basically just 'stress testing', balancing info and Early Access toying around. It isn't 'real beta testing' in any sense of the word, nor does FD expects it.

 

[znôrt]

they probably have a team in their offices doing that

they must be working on other games then.

but as you pointed out, there are 100's of subtle features in outfitting, missions, combat, trade, mining, exploration, powerplay, BGS and factions.
so there is no better way to let as many people as possible to go out there to hunt the bugs, especially if some have paid for it or doing it for free

dunno about the release but the beta definitely had never been tested, ever, not even in a cursory way. the types of bugs were not at all 'subtleties'. so players effectively did the grunt work there. and they seem to be fine with that, and if frontier are ok with that image, so am i ...

 

[13A]

I think you mean kind of a pre-beta that lets them rectify some things before the beta is released to the torch and pitchfork brigade

As an example - the new exploration mechanics.
One of the first things I noticed was that I was having to come to a stop and open the FSS when I arrived in a system, just to discover that there was nothing there other than the star. After many posts in the beta forums they updated the UI so that the number of bodies show in the HUD.

they probably have a team in their offices doing that

I think the problem is that they will be testing specific actions; e.g. "we've created this new planet scanner. We want you to fly to planets and see what happens when you fire it."
Whereas players will be thinking "Cool, a new planet scanner. I wonder what happens when I point it at a ship?"

 

[Sylow]

The problem was not in the testing done. The bug report section was full. The problem rather was that for some bug reports, I very close to the end of the beta got a reply to almost all of them, which always was along the line: "Sorry, we don't have the capacity to check and confirm all of the bug reports. Can you please check again and make a new report, if the problem still exists."
.
Which is something I try to find for at the next weekend: to check for at least the most crucial bugs I found, if they still exist and if yes, create new bug reports.
.

 

[RamirezKurita]

I don't think a small cadre of dedicated players under NDA are going to really help beta testing, the problem is that many beta players are simply there to preview content and get a free second account for a while rather than actually test things. What we need is more players who actively try to break the game and push it to its limits during the betas to find balance issues and exploits rather than tooling around and pointing out the obvious. Maybe some kind of reward scheme would help, the first player to report each bug or discover a balance issue receives a free ship skin or bobblehead?


Likewise, they should probably organise a proper PvP tournament during the betas with in-game rewards for player's main accounts for those that perform well. A controlled event like that would help get proper balance data for duels, as they can identify if particular engineering mods, modules, weapons and ships are overrepresented in the higher rankings. Likewise, a similar thing could exist for credit earning, giving players rewards for their main accounts for those that have the highest earnings within a limited timeframe (such as a 2-hour timeframe) to encourage players to go out and actively find credit exploits in the betas.

Longer betas would also help, as they are pretty short and don't really allow for extensive testing nor for fine tuning of balance. Betas should last for months if they want to actually get a polished release.

 

[Rubbernuke]

There really should be a permanent Beta server where changes are made and allowed to be tested in a long term way. The BGS, Powerplay etc are huge systems that can't be tested in any other way, and are often pushed to 'live' without being properly poked.

 

[Jmanis]

Honestly, 99% of the problem with the Beta is, what I'd call the "geniune feedback" identifying *actual* issues (like bugs in missions) gets lost in the forum rage from people there to just get a sneak-peek of what's around the corner, about whatever major change is going ahead that they already hate... in this case the removal of the honk.

That's not to say some of the feedback/criticism of the new exploration mechanics isn't legitimate, only that a lot of it is just hot wind, which ties back to my original statement; the problem isn't a lack of people reporting actual issues, but the over-abundance of noise about whatever the FOTM forum rage is.

 

[Avago Earo]

When a member of the QA team reports a bug in Beta, and it persists in the Live game:

https://forums.frontier.co.uk/showt...en-functions?p=7248232&viewfull=1#post7248232

It makes me wonder whether there is any point in Beta 'testing' at all.

 

[Sylow]

Honestly, 99% of the problem with the Beta is, what I'd call the "geniune feedback" identifying *actual* issues (like bugs in missions) gets lost in the forum rage from people there to just get a sneak-peek of what's around the corner, about whatever major change is going ahead that they already hate... in this case the removal of the honk.

.
That's why there were actually two areas:
1. The beta feedback. Which for a short time was actual feedback, before turning into just a variation of DD.
2. The bug report area. While it also contained a bit of chaff, the signal to noise ratio there was much better. Almost all reports there were pointing out actual bugs and problems.
.
Unfortunately even the second one was overflowing and QA people themselves gave the feedback that they were unable to keep up with the number of bug reports. Thus a lot of bug reports were written and forgotten about, without ever being checked by QA, let alone seen and fixed be the coders.
.
So some things were done right. There was the "feedback" section to give people raving people a place to rant, while not getting too much of that into the bug section. A lot of people used the bug section to deliver bug reports. The bottleneck was processing the high number of bug reports. Fixes would be more QA people or a longer break between the beta and going life. Unfortunately good QA people don't come for free or can be picked up quickly and making the community wait for longer would also have been a disaster. Too many already demanded things to be rolled out.
.
I still hoped that some fixes for things which annoyed would've been done during the beta or in the week between beta and rollout. But it is what it is, my reports were not even processed before the last few days of the beta. Maybe we're lucky and still get some bugfixes before xmas.
.

 

[znôrt]

Honestly, 99% of the problem with the Beta is, what I'd call the "geniune feedback" identifying *actual* issues (like bugs in missions) gets lost in the forum rage from people there to just get a sneak-peek of what's around the corner, about whatever major change is going ahead that they already hate... in this case the removal of the honk.

That's not to say some of the feedback/criticism of the new exploration mechanics isn't legitimate, only that a lot of it is just hot wind, which ties back to my original statement; the problem isn't a lack of people reporting actual issues, but the over-abundance of noise about whatever the FOTM forum rage is.

mmmm ... no. i get where you come from but the signal to noise ratio is actually very high here:
https://forums.frontier.co.uk/forumdisplay.php/124-Bug-Reporting

so that excuse is 99% what comes out from the exhaust of an asp.

the confusion is understandable, though. a beta test usually has precisely the goal of gauging user experience, which is inherently subjective and will always be up to debate, but that's very difficult if you throw completely untested and broken code into it. i should probably remind you now that the topic is 'proper beta testing'

 

[Miko]

One thing is for certain. FD don't test a thing! They use you it's for betas and still they miss the major bugs.

 

[IndigoWyrd]

I’ll go with this:

Since the floodgates were opened and beta testing was not limited to the LEPers and those willing to dig into their wallets more productive feedback and bug reporting seems to be taking place. I’ve no doubt it’s a numbers game - let more people in, get more good data out.

So.. no Elite has not grown in any way meaningful to the question, nor has the community. But by involving more people overall the end result is improved.