Re: paying Beta testers not doing a proper job

TomP · Nov 6, 2016

Re: paying Beta testers not doing a proper job

I've seen the following credo expressed at least half a dozen times:

If only FDev employed proper Beta testers, and if only the paying Beta testers bothered to test it properly, rather than just doing pew-pew!

Well, you're wrong.

Many of us spend time, having paid for the privilege, submitting bug reports. It is not only inaccurate but also stupid to claim otherwise. I sincerely hope that FDev do actually gainfully employ Beta testers, as they should, but equally they make some dosh out of folks like me who just play the game early. And submit bug reports.

The irony is that not only do we get dissed by the community, for apparently just doing pew-pew, we seem to get dissed by the devs as well, as they rarely answer the persistent bug threads. Have a look. There are bugs in the machina which have been pointed out ages ago. Still there. Yet still, apparently the paying beta players are to blame. C'mon guys.

Sort your minds out. I'd happily get paid by FDev to test their architecture, but they don't pay me, I pay them. Funny, that.

Starstrydur · Nov 6, 2016

Oh Relax. I don't point to people for the failures. As I have heard the beta is not tested the same as when the new content is added to the live game, Perhaps this is more the problem. [heart]

SP45M · Nov 6, 2016

Irony
of, resembling, relating to, or tasting much like iron
eg1:
"The anvil was delicious, but irony."

X-Terminator · Nov 6, 2016

So because FD not listen to me I should be paid back

Wylie28 · Nov 6, 2016

Ill bet millions they have paid beta testers. The game would be significantly buggier than it is if they didn't have a QA team. Anyone that says differently has zero experience programming and can't even begin to imagine how much can go wrong when networking is involved.

Achilles7 · Nov 6, 2016

Have another swig from that bottle of Smirnoff..then pass out!

..like I need to advise you on the procedure!

vrseven · Nov 6, 2016

Paid beta tester? No, it goes against Elite lore.

TomP · Nov 6, 2016

Wow.

I pointed out that we are little more than paying early players, but pointed out that some of us actually bothered to submit reports.

Result?

Nastiness.

N.B. For future reference, don't bother with anything constructive.

- - - - - Additional Content Posted / Auto Merge - - - - -

PS love you guys. Sort of.

Alex Brentnall · Nov 6, 2016

Gotta agree, there's one guy in particular that is so against the paid beta I had to ignore him as he just got abusive when I provided counter-arguements. It became fairly obvious all he wanted was free beta after a few rant posts.

Either way yeah, so many bugs get reported in beta, not all can be fixed. Some bugs only appear in live not in beta yet for some reason there are people that think it's all the beta testers fault. Whilst it's true lots of people do use the beta just to try ships they can't otherwise fly or to mess around in risk free environment it is not true that all beta players do that. It's obvious enough looking at the pages of bugreports that get submitted during beta periods.

+1 anyway.

W4LD0 · Nov 6, 2016

TomP said:
Wow.

I pointed out that we are little more than paying early players, but pointed out that some of us actually bothered to submit reports.

Result?

Nastiness.

N.B. For future reference, don't bother with anything constructive.

- - - - - Additional Content Posted / Auto Merge - - - - -

PS love you guys. Sort of.

Oh the internet, humanity's greatest and worst invention. Giving connection to all individuals who possess access to it, and what is the end result?

A minimum of 50% with access who are highly opinionated and critical of all others. With something that is supposed to connect us, we seem more divided now than ever. Funny that. Oh well, thanks for your troubles and your efforts CMDR, have some Rep for a worthy cause.

TomP · Nov 6, 2016

SP45M said:
Irony
of, resembling, relating to, or tasting much like iron
eg1:
"The anvil was delicious, but irony."

I don't think you understand the word in question. Is your name Alanis, by any chance?

listerinfiji · Nov 6, 2016

I have zero criticism of the paying beta testers... You guys paid to have the beta and you are putting in work for free to assist FDev.

I do however completely agree with the fact that FDev should be paying dedicated beta testers.

I'm not gonna grind on with opinion but the above is simply what I think

TomP · Nov 6, 2016

listerinfiji said:
I have zero criticism of the paying beta testers... You guys paid to have the beta and you are putting in work for free to assist FDev.

I do however completely agree with the fact that FDev should be paying dedicated beta testers.

I'm not gonna grind on with opinion but the above is simply what I think

Agree 100%

Alex Brentnall · Nov 6, 2016

listerinfiji said:
I have zero criticism of the paying beta testers... You guys paid to have the beta and you are putting in work for free to assist FDev.

I do however completely agree with the fact that FDev should be paying dedicated beta testers.

I'm not gonna grind on with opinion but the above is simply what I think

I can understand that. My only counterpoint is if people are willing to pay for beta why not let them? It'll pay for the server costs at very least.
Ok thats the arguement that the other space game is using for £100+ ships but I personally think the £6 price point is about perfect for an entire season of early access/beta. If it were say £20 I'd be totally on the other side of the fence arguing it's a crazy strategy.

Paintjobs at that price are something I can't really get behind despite having paid for a fair few. Yeah I know... Anyway that's a different thread.

who? · Nov 6, 2016

They don't need to pay any Beta testers. As it stands, there are more than enough people willing to pay to test the game, and are finding bugs far faster than FD can fix them. If there's a bottle-neck, it's having the developer resources to fix the known issues, not additional testers to find more.

listerinfiji · Nov 6, 2016

Alex Brentnall said:
I can understand that. My only counterpoint is if people are willing to pay for beta why not let them? It'll pay for the server costs at very least.
Ok thats the arguement that the other space game is using for £100+ ships but I personally think the £6 price point is about perfect for an entire season of early access/beta. If it were say £20 I'd be totally on the other side of the fence arguing it's a crazy strategy.

Paintjobs at that price are something I can't really get behind despite having paid for a fair few. Yeah I know... Anyway that's a different thread.

Not suggesting the paid beta should not exist. I can understand people like to have access to early content and people who regularly play the game will have an understanding of it that a paid beta tester wouldn't. And of course as you are indicating it is a revenue stream, and that is important.

However paid beta testers should still exist. People who have a job to sit down with the beta 8 hours a day, 5 days a week (not distracted by another job or school), and who have checklists created for them of everything they need to test, forms to fill out detailing what they've experienced during their testing, etc.

I think the people who beta test now are doing a service to this game, and I think anybody who does criticise them might be well served being forced to play the original beta for a week so they can appreciate improvements were made based on their feedback. But these people also have real lives that consume their time, plus some of their game time will be spent in the live version of the game or even in other games, so this is where paid beta testers are an invaluable asset.

Damn I went back on my last post now and did grind on about my opinion

- - - - - Additional Content Posted / Auto Merge - - - - -

who? said:
They don't need to pay any Beta testers. As it stands, there are more than enough people willing to pay to test the game, and are finding bugs far faster than FD can fix them. If there's a bottle-neck, it's having the developer resources to fix the known issues, not additional testers to find more.

That's a fair point and now we are talking capacity of the developer vs a need to meet a deadline.

However let's take one specific bug that made it into the live version - the module storage issue that prevented players from accessing their CMDR data. That's a biggie. Having paid beta testers would allow for better identification and more importantly prioritisation of bugs prior to final release.

Alex Brentnall · Nov 6, 2016

listerinfiji said:
Not suggesting the paid beta should not exist. I can understand people like to have access to early content and people who regularly play the game will have an understanding of it that a paid beta tester wouldn't. And of course as you are indicating it is a revenue stream, and that is important.

However paid beta testers should still exist. People who have a job to sit down with the beta 8 hours a day, 5 days a week (not distracted by another job or school), and who have checklists created for them of everything they need to test, forms to fill out detailing what they've experienced during their testing, etc.

I think the people who beta test now are doing a service to this game, and I think anybody who does criticise them might be well served being forced to play the original beta for a week so they can appreciate improvements were made based on their feedback. But these people also have real lives that consume their time, plus some of their game time will be spent in the live version of the game or even in other games, so this is where paid beta testers are an invaluable asset.

Damn I went back on my last post now and did grind on about my opinion

Hehe, yeah theres been a lack of "go test this" but that changed a lot in 2.2 with active suggestions from FDev on what they want users to test - Heat meta, fighters etc etc. And they got tons of feedback. Check the sticked archived here https://forums.frontier.co.uk/forumdisplay.php/189-2-2-Beta-Feedback-and-discussion

I think if they continue in that vein and give direction to the hundreds if not thousands of beta players they'll get the data they need. The suggestions above provoked many changes, most obvious the neutron star/white dwarf boost kept at useable levels where it's actually a gameplay element now.

Either way I do agree with you but there's a lot of finesse, who gets paid and why. How many, how much etc etc. I can foresee it making a lot of other players angry if a subset get not only cash but also more influence. There's already people being annoyed that the biggest player groups representing thousands of players get more access to FDev via hidden subforms where they can discuss things about group mechanics and CG's and stuff like that.
Makes total sense if you detach your own personal involvement and say: That groups leader can represent 2000 players views to FDev in condensed format from 1 voice instead of 2000 shouting over each other, but some argue...

Ok you've set me off too now

I'm going to go drink some tea and chill out a bit although I will say I've enjoyed this conversation. Interesting points on both sides here without the usual mess that these things often turn into.

Dbrn47 · Nov 6, 2016

TomP said:
I don't think you understand the word in question. Is your name Alanis, by any chance?

What's that? Irony?

Well.. clearly, it's like rain on your wedding day. It's a free ride when you've already paid. It's the good advice that you just didn't take.

Who would've thought? It figures…

Oh I know, it could actually be like:

4d4f25515f030c88cb37a68c6b152ef5934f646e7bc90790ceb76848d4be4076.jpg

listerinfiji · Nov 7, 2016

Alex Brentnall said:
Hehe, yeah theres been a lack of "go test this" but that changed a lot in 2.2 with active suggestions from FDev on what they want users to test - Heat meta, fighters etc etc. And they got tons of feedback. Check the sticked archived here https://forums.frontier.co.uk/forumdisplay.php/189-2-2-Beta-Feedback-and-discussion

I think if they continue in that vein and give direction to the hundreds if not thousands of beta players they'll get the data they need. The suggestions above provoked many changes, most obvious the neutron star/white dwarf boost kept at useable levels where it's actually a gameplay element now.

Either way I do agree with you but there's a lot of finesse, who gets paid and why. How many, how much etc etc. I can foresee it making a lot of other players angry if a subset get not only cash but also more influence. There's already people being annoyed that the biggest player groups representing thousands of players get more access to FDev via hidden subforms where they can discuss things about group mechanics and CG's and stuff like that.
Makes total sense if you detach your own personal involvement and say: That groups leader can represent 2000 players views to FDev in condensed format from 1 voice instead of 2000 shouting over each other, but some argue...

Ok you've set me off too now I'm going to go drink some tea and chill out a bit although I will say I've enjoyed this conversation. Interesting points on both sides here without the usual mess that these things often turn into.

Yep always good when you can have a constructive conversation with somebody

You've mentioned possible resentment generated by paid beta testers being drawn from the community. However my idea was not to take them from the community, but hire them like any other employee.

Beyond what I've already stated, further benefits that spring to mind:

* They can be placed within a controlled environment.
* Different PC specs can be selected and used.
* Advanced logging can be ran on the PCs.
* Videos can be constantly recorded of activity.
* The above points mean that priority on fixing bugs can be more easily selected as everything required to tackle any given bug found is immediately available.

The idea with paid beta testers though is they really should be focused on bugs, especially game breaking ones. You mentioned the example with the paying beta testers helping shape game decisions, such as white dwarf/neutron star jumps, and I think this is an area where the paying beta testers will give more value than the paid beta testers, as these are the people who play the game regularly and this allows the community help to shape the game we all play.

xyphic · Nov 7, 2016

Paid QA engineers will generally be running tests against a fairly rigid test plan. This'll probably be a long and boring procedure. Good testers will understand the systems that they're testing, and will sometimes go outside the bounds of the test plan. But even with the best will in the world, any internal test team is going to be limited in size and won't be able to generate every edge case or recreate every set of conditions.

So having an external team, who aren't being paid to run a test plan, just playing the game and reporting bugs is invaluable as long as the tools are there for the developers to act upon the feedback. If anything, I think that Frontier's reporting tools are somewhere that the process could be improved. There are so many bug reports raised that even though the support team are able to read all of them, replying to them takes up an inordinate amount of time (time that would be better spent elsewhere). So sometimes if a bug is reported you might not get a response, even though internally it might make it onto the bug tracker and be progressed appropriately. Getting visibility of this bug tracker might be helpful, both because it would be searchable, and because we'd be able to see progress without requiring the support team to spend time replying to every thread. Where duplicates are raised, they'll simply get linked to the master tracker for the bug and it'll be clear then that's what's happened.

Of course, there's room for abuse but assuming Frontier are using something like JIRA I would expect them to be able to provide access to those with an active Frontier account.