Tracking exploration activity

[Orvidius]

It's definitely be interesting to see. I think we get a boost in activity after every major patch, but the next few won't be new features. That is likely to have some impact. But I've also seen others say that they would rather the game work correctly than get new features bolted on, so who knows. Some explorers just do their own thing and don't much care about new content either. Lots of variables here.

Thanks for pulling these numbers together! It's quite fascinating.

 

[marx]

It's definitely be interesting to see. I think we get a boost in activity after every major patch, but the next few won't be new features. That is likely to have some impact. But I've also seen others say that they would rather the game work correctly than get new features bolted on, so who knows. Some explorers just do their own thing and don't much care about new content either. Lots of variables here.

Yeah, but it probably also depends on player perception on what is a major update. For example, the 2019 April and September updates weren't even visible in the data. (Of course, the April one was during DW2.)
Looking back on historical data, I think that the updates which had new content added had seen bigger boosts in exploration as well. See the Return for a good example.

 

[FixMeNow]

I would not read too much into EDSM to speak for console players, you would be right to assume about 50% use it as half my friends list use it (Xbox)

I for one dont use EDSM and i make sure i scan every body in every system when exploring. Thats atleast 63000 systems not on EDSM.

 

[Ian Doncaster]

I would not read too much into EDSM to speak for console players, you would be right to assume about 50% use it as half my friends list use it (Xbox)

That feels way too high.

Out in Colonia, which is already filtering for explorers and experienced players, both more likely to link up to 3rd-party tools than others, only about 1/6th to 1/8th of player activity is picked up in 3rd-party tool reports. I would be inclined to think less than 10% of players (though probably about 10% of player activity, which is not the same thing) is recorded.

 

[marx]

I would not read too much into EDSM to speak for console players, you would be right to assume about 50% use it as half my friends list use it (Xbox)

I for one dont use EDSM and i make sure i scan every body in every system when exploring. Thats atleast 63000 systems not on EDSM.

You can read some analysis on how representative the EDSM sample might be under the spoiler cut in the first post.
That said, personal anecdotes are hardly evidence. For example, I think that 100% of the explorers on my friends list use EDSM, and of course that's not globally. A global 50% that you said would be wonderful, but it doesn't look like it would be the case.
Also, to put things into perspective: even these days, around 35,000 systems get added to EDSM daily.

Out in Colonia, which is already filtering for explorers and experienced players, both more likely to link up to 3rd-party tools than others, only about 1/6th to 1/8th of player activity is picked up in 3rd-party tool reports. I would be inclined to think less than 10% of players (though probably about 10% of player activity, which is not the same thing) is recorded.

By player activity, I assume you mean all kinds of activity, not just exploration? After all, explorers tend not to hang around inhabited space a lot. More importantly though, about that 1/6th to 1/8th (12.5-16.67%) ratio you said, I'm curious: how did you get that?

 

[Ian Doncaster]

By player activity, I assume you mean all kinds of activity, not just exploration? After all, explorers tend not to hang around inhabited space a lot. More importantly though, about that 1/6th to 1/8th (12.5-16.67%) ratio you said, I'm curious: how did you get that?

I'm going for the most directly comparable - the FSDJump event through EDDN and the in-game traffic report measure exactly the same event. I regularly collect the regional traffic reports in-game, so getting a ratio between them and the EDDN events is then easy. By player activity I'm trying to distinguish from number of players since 5 players jumping once and 1 player jumping 5 times show up the same - so on a hypothesis that the more active players are more likely to feed EDDN, the player activity ratio will be better than the player ratio.

And sure, explorers might not hang around that much - but lots of explorers do other things as well (one of the Distant Worlds squadrons got a top 10 place on the PC combat leaderboards), and lots of them pass through Colonia to repair, sell data, see the molluscs, take a break, etc.

(It also seems a reasonably consistent number with the percentage of discovered galaxy which EDSM knows about probably being in the 1/5th to 1/6th region, when considering that a system visited multiple times - more likely nearer the bubble - only needs to be visited by one EDSM user to be known)

 

[marx]

Ah, I see. There's also that then that while the traffic reports in-game are updated daily, many explorers don't automatically and regularly upload their info, but do larger dumps at once - which does lead to a time difference, events showing up days later than they actually happened. (You can see this "drift" on numbers of new bodies discovered on given dates too, but it's not large.) We can look this up later, but we can't look up earlier traffic reports in-game. Of course, "how many" is a good question, and I'm not sure how it could even be estimated. Nor the difference. Although at least there is the EDDN software chart, but only EDMC there is the one that AFAIK has to auto-update jumps. On the other hand, people uploading via the EDSM site probably don't do it regularly.

Still, yeah, that, and that explorers arriving to Colonia and hanging around there are not uploading new systems, would explain why your estimate is a bit lower than what I came up with. Then there's also that not all explorers in the galaxy are going to / from Colonia - although looking at the videos and the Commanders map, I wouldn't be surprised if the majority of them did go to.

 

[Ian Doncaster]

Ah, I see. There's also that then that while the traffic reports in-game are updated daily, many explorers don't automatically and regularly upload their info, but do larger dumps at once - which does lead to a time difference, events showing up days later than they actually happened.

Yes. For FSDJump only reasonably recent events get considered, so there will be a bit more submission of body data which isn't time-sensitive.

Still, yeah, that, and that explorers arriving to Colonia and hanging around there are not uploading new systems, would explain why your estimate is a bit lower than what I came up with. Then there's also that not all explorers in the galaxy are going to / from Colonia - although looking at the videos and the Commanders map, I wouldn't be surprised if the majority of them did go to.

Certainly - though I would expect "explorers passing through Colonia" and "other long-range explorers" to have generally similar rates of 3rd-party tool use.

 

[marx]

Before more on the squadron leaderboards, it's back to body types a bit: with @Orvidius's help, I added historical counts for WWs, WGs and RIWs - to a snapshot. It'll stay as a snapshot because I don't want to include periodical counts of these types in the future: the results look interesting, but there don't appear to be any big differences that would justify tracking their changes too. And, as you can imagine, there are plenty of WWs and a whole lot of RIWs added.
You can find these under the new "Extended snapshot" and "Charts - extended snapshot" subsheets.

So, some notes on the new stuff:

Water worlds turned out to look mostly the same as ELWs: a spike at first, drop later, then back to just under pre-Chapter Four levels these days... but the initial spike was much shorter than it was with other worlds. (Except for ELWs, which were unique in that there was no increase for them - rather, a decrease.) I suppose plenty of people went looking for these before already.

Water giants had a boom, having seriously "underperformed" before the FSS, and actually getting added at around the same rate as ELWs are these days. It makes me wonder how their actual distribution might be. WGs are right next to WWs on the barcode, so that could also help with more people scanning them now - maybe mixing them up, maybe scooping them up just to be sure.

Rocky ice worlds are probably the "winners" this list: there are far more of them discovered after the FSS. With all the other body types examined, their ratio of the total bodies scanned (ELW / Bodies, WW / Bodies etc) decreased, but in the RIWs' case, it increased instead. Said increase is just a single percent, but when it comes to millions of new planets per month, that still turns out to be a whole lot.
The curious thing though that after the initial burst right after the update hit, the RIW / Systems rate increased much less after DW2 set off, and the drop after the expedition reached the core was also much less. It appears to follow the Planets / Systems ratio the best, but it actually peaked a month before that one, and also pretty much everything else, did. Perhaps January was when the most people did full system scans, and they started getting bored of those not after February, but one month earlier? It would be good to know, but unfortunately, we don't have the percentages on bodies scanned per total bodies in the system.

Oh, and to see how console players might have contributed before EDSM supported their platforms, I've added a row where you can multiply their systems by a value of your choosing.

 

[marx]

Thanks to the excellent @Ian Doncaster, here come the records from the in-game squadron exploration leaderboards of all previous seasons on the PC, so that's some more data to analyze. Bear in mind that all this only applies to squadrons, and we have no idea how many explorers might be in squadrons. (Then there would be the question of how many members of squadrons are active explorers.)

The final caveat is that squadron points come mostly from the credit value of exploration payouts, which I couldn't directly compare with the same data of EDSM. Instead, I examined a scenario where the total of squadron points would come from ELWs and WWTCs, with ELWs being 16.37% of that total. (That ratio comes from the numbers of both uploaded to EDSM.) This of course overestimates the number of such bodies found by squadrons by some degree, but I wouldn't expect it to be too much.
While it would be possible to calculate and sum up by dates the payouts of every single body uploaded to EDSM, not only would this take a long while to run, but we also don't know which ones received what bonuses. Especially back when the game didn't log if a body was already discovered or not. (It does now, just not who the discoverer was.)

Also, one more thing: while the EDSM data counts for the time of scanning, the squadron data counts for the time of sale. This was probably the most relevant in DW2's time, with a lot of explorers selling the data at once.

So, with these points in mind, I'd advise anyone against taking far-reaching conclusions from this data and analysis. Just use them as rough guidelines perhaps.

A bit more about the methodology. Squadrons ranks 1 to 10 are exact, after which Ian sampled every ten ranks first, then starting from 100, every hundred, with the final being #1000. (This is because of the interface's limitations.) After the top ten, I did linear regression on the other two ranges, and calculated the areas under the resulting two equations. Non-linear would have been a better fit, but linear was also quite decent, and spreadsheet programs work much better with those. (Unless I'm unfamiliar with something there.)
No data after the #1000 position, because the number of squadrons on the leaderboard varies season by season - and as you can see from the data, even by that time, we are talking about squadrons selling at most 10 million Cr per week. Granted, thousands of those do add up to some degree, but I decided to disregard those.

Finally, it might not be readily apparent from the data, but let's not forget that unlike the other seasons, Season 2 lasted four weeks instead of the usual eight.
I also made exact ELW counts from EDSM (although these could be off, as I'm not sure I got the start and end dates of Seasons right) to compare these with, and from those, an estimate for how many credits ELW+WWTCs could have made. (Assuming a constant WWTC/ELW ratio.) This still wouldn't be the total sale value, of course, but still likely to be the majority of it. Then to better compare all these with the short duration of Season 2, I also did simple weekly and daily breakdowns.

So, what can we tell?
Squadron exploration activity over seasons was mostly the same for Seasons 1 and 4, and a lower amount for 2, 3, 5. However, as I mentioned, S2 was half of a regular season; on a weekly and daily basis both, squadron exploration activity peaked in Season 2. DW2 went on mostly during Seasons 2 and 3. As with the EDSM data, we see a big boost during DW2's "peak", so to speak, compared to which both Season 3 and 5 are roughly halved. So, even though we are talking about squadrons here, the effect was similar... except for Season 4 bouncing back.
For Season 6, I don't have all the data yet, it'll take some time to go through it - but the top 10 mirrors Season 4 there. If the rest of the squadrons there will too, then it's interesting that the activity would cycle as such there, because the EDSM activity is fairly stable - and let's not forget that there have been no changes to exploration ever since Chapter Four, the start of this data. (Although there have been two smaller updates focused on new players, and while I have no idea if they helped with squadron activity or not, on EDSM neither of the two made any difference.)

Unfortunately, in the end there are still too many unknowns, differences and estimates here to draw good comparisons between the two sets of data. Still, it does raise some interesting questions, and it's always good to see data from different sources, especially when the source is in-game.

 

[schlowi123]

That's a good writeup of the limitations of the data and the analysis and how the data was collected. Nonetheless is it of course totally valid resulting at conclusions within this framework.

Regarding data analysis: have you ever tried SciDAVis? It is free and open source software (and also free of charge). Albeit it is mainly aimed at visualization of your resutls does it also have an in-build fitting wizard that has no problems at all with non-linear functions. Well, non-linear fitting can always give you problems if you start with the wrong parameters, but within these limitations it has no problems .
It lacks some spreadsheet functionality but copy and paste from spreadsheets (or import of larger datasets) works well, too

 

[marx]

Thanks for the tip! I haven't yet, but will try it later.

Also, I updated the squadron leaderboard with the Season 6 PC data. No surprises there: it's mostly the same as Season 4. An interesting thought though is that while the #10-#100 ranks' contribution is increasing, #100-#1000 is decreasing. This could be a sign of increased competition among squadrons which do take exploration somewhat seriously, but I wouldn't read too much into it just yet.

 

[Deleted member 166264]

Actually, I wonder about something: how difficult is it in practice for a console user to regularly upload systems to EDSM? It would be good to know whether it's more difficult than the streamlined process(es) on PC or not. I'm wondering about how many "hidden" systems there might have been before console support was added.

Can't speak for others, but I didn't know about the site until someone posted in the PS4 section that it was now compatible with the console earlier this year. Once I set it up to work with my PS4 it worked well, but I was running missions in the bubble for the squad I was in at the time. Then in the beginning of March I started putting it to good use and went out in the black for a couple trips. Then in mid-March, during my second trip, the site just stopped working correctly for me. Prior to the update it was tracking my progress in real-time. Now if I tell it to update it tracks where I was two weeks ago, it never knows what ship I'm in, it says I'm carrying more cargo than the highest amount I can carry in a ship that doesn't have cargo racks, and it says I'm doing missions that I completed last month. All-in-all I currently find it too much of an annoyance to use and stopped using it.

 

[marx]

Hm, good to know. Thanks!

Also, another month's update is here. It looks like it's pretty much the same as last month's, although there is one thing that worries me a bit: looking at the daily data, the ten days since the carrier delay & content freeze announcement seems to have taken a considerable turn down. It's too soon to tell though it whether this is just a one-off thing, or if the decrease will remain.

 

[Vasious]

This reminded me to actually upload my journals to EDSM

 

[schlowi123]

Can't speak for others, but I didn't know about the site until someone posted in the PS4 section that it was now compatible with the console earlier this year. Once I set it up to work with my PS4 it worked well, but I was running missions in the bubble for the squad I was in at the time. Then in the beginning of March I started putting it to good use and went out in the black for a couple trips. Then in mid-March, during my second trip, the site just stopped working correctly for me. Prior to the update it was tracking my progress in real-time. Now if I tell it to update it tracks where I was two weeks ago, it never knows what ship I'm in, it says I'm carrying more cargo than the highest amount I can carry in a ship that doesn't have cargo racks, and it says I'm doing missions that I completed last month. All-in-all I currently find it too much of an annoyance to use and stopped using it.

For consoles EDSM is NOT supposed to just update in real time any longer. That didn't work properly and seemed to not have caught all of the systems where you've actually been.
Nowadays it downloads (I guess from FDev) the data for the last 30 days. Unless it has the information already.
That means if you turn EDSM dashboard on, it will first update the last 30 days before it reaches the point in time you would consider as "now". That however, takes some time. At least 30 minutes, but more if you were exploring a lot during that timespan.
So just activate EDSM dashboard in a browser tab and leave it active for a while. Once the updating process is finished the gathering of new data will than be (more or less) in real time.
One last comment: If you have a long backlog the updating process smetimes "chokes" on the amount of data. Simpky reload (F5) the dashboard.

Regarding correct ships / cargo / missions: It does update but it always seems to take a while. I haven't figured that one out since I'm neither do many missions nor am I changing ships often. But just leave the dashboard open and you should be fine after a while.

All in all is the "new" console EDSM functionality much better than the first version.

 

[Sir.Tj]

Title edited

 

[marx]

Thanks!

Time for this month's update then. Systems are continuing their steady but very slow decline. The bad news near the end of October did produce a significant hit, but looking at the daily graphs, things have almost recovered from that a month later. We'll see where things go now in December. It's quite remarkable though how stable systems have been since DW2 ended, especially compared to earlier times - too bad that that's half of what it used to be during the expedition, and most likely below previous levels. (Even more so if you count them from before the FSS reveal, which sent it to a record low that fortunately has not been met since.)

Next up should be the squadron leaderboards, IIRC the current season will end in a week or two, and then we'll see how things went on that front.

 

[marx]

Back to squadron leaderboards for a bit: it turns out that what I heard about how sale payouts are converted to leaderboard points was wrong, it's not 1:1, but 2:1. Meaning for 2 Cr of data sold, the squadron gets 1 point on the leaderboard. At least, the data points I recorded were close enough to that that I assume last digit differences were down to rounding only. I've corrected the squadrons sheet to account for this.

This is the same as how exploration rank progression works, by the way: for every 2 Cr there, you gain 1 XP. Back when this change was introduced though, it was meant to adjust the pace of rank progression (after payouts were doubled, then there was the mess with tourism missions as well) without messing earlier progression up. It's quite interesting how it made its way to squadron leaderboard calculation too then. It's most likely just made up of exploration rank progression then.

 

[marx]

We have an update on how many systems Commanders have explored, a bit buried in the fifth anniversary celebration post. The figure is 0.042% of the galaxy explored.

Now, assuming that that figure is for today, that means that since 2018 March 2, when FD tweeted that Commanders have discovered 112,863,791 systems, which was 0.028% of the galaxy, players have discovered +0.014% of the galaxy in the 654 days since. That would be about 169.3 million systems (169,295,687, but since we only got a percentage this time, there would be rounding errors), which is +56.4 million systems, or 86,300 new systems per day. How does EDSM stack up against that?

EDSM had 20,853,341 systems added on 2018-03-02, and today on 2019-12-16 it has 46,624,091, or 0.011652% of the galaxy. (Meaning that despite the long "head start", the whole database still covers 27.74% of the total.) So that's +25.8 million new systems added, or 39,400 new systems per day.

To sum it up: during the last 654 days (1.79 years), with regards to new systems, EDSM's activity covered an impressive 45.66% of the total.