Tracking exploration activity
- Thread starter marx
- Start date
That's a great summary and discussion of the data. Your explanation regarding what I "observed" here seems intuitively right. I think there used to be the date of discovery in the EDSM bodies-file. Than that could have been tested. But this information is gone now 
Anyway, thank you for doing all the work .
Anyway, thank you for doing all the work
.
Last edited:
It has been a while since I asked Anthor about this, and so, things might have changed since then. We'll see.
However, even if it turned out that upload date is all that matters now, the difference coming from a small amount of people mass-dumping systems still doesn't make much sense. The chart follows a nice regular "highest on Sunday, lowest on Wednesday" pattern, and the big increase in February lasted three weeks. So you'd need a group of people gathering roughly 20,000 systems in the span of a week, and coordinatedly selling them on Sundays.
However, even if it turned out that upload date is all that matters now, the difference coming from a small amount of people mass-dumping systems still doesn't make much sense. The chart follows a nice regular "highest on Sunday, lowest on Wednesday" pattern, and the big increase in February lasted three weeks. So you'd need a group of people gathering roughly 20,000 systems in the span of a week, and coordinatedly selling them on Sundays.
Last edited:
As promised: results. Illustrating quite well that even if you say you're 95% sure, it can pay off to check. As it turns out, the date currently in the EDSM data dumps is that of the upload date, not of the discovery. Take, for example, the system Eodgorsts WC-V d2-66: the discovery date is 2020-03-05 12:50:25, the systemsWithCoordinates7days dump lists it as 2020-03-06 12:37:30, when I uploaded it.
I don't know how long it has been this way, as I don't have much earlier dumps. (Although I'll see if I can't find some that I forgot to delete.) In any case, this is actually the more interesting result: since the systems uploaded definitely do appear with their upload date now, not their discovery, it still ends up quite cyclic. But then... looking at the EDDN status page, yesterday of the total software used, EDMC made up 42.7%, EDDiscovery 34.3% and EDSM (two versions' total) 13.2%. Of course, that's not systems uploaded, but total hits, so it's just an indicator that the majority of explorers probably use software that automatically uploads their finds as the discoveries are made. (Although I confess I have no idea about the default settings in EDDiscovery, so if I'm wrong there, do please correct me!)
An interesting aside: that EDSM total was 8.8% regular and 4.4% on Console. In other words, a third of the EDSM hits came from its console version - quite a lot more than I expected.
But then, if a Commander discovers an average of 200 systems daily, and only uploaded once a month, that would produce a 6,000 systems "spike" on one day. (This would make them "produce" more systems in a month than most explorers do in a year.) There aren't really such, the only ones that happen are the weekend spikes that come like clockwork.
So, anyone want to save up a lot of systems then sell them all on a Wednesday?
I don't know how long it has been this way, as I don't have much earlier dumps. (Although I'll see if I can't find some that I forgot to delete.) In any case, this is actually the more interesting result: since the systems uploaded definitely do appear with their upload date now, not their discovery, it still ends up quite cyclic. But then... looking at the EDDN status page, yesterday of the total software used, EDMC made up 42.7%, EDDiscovery 34.3% and EDSM (two versions' total) 13.2%. Of course, that's not systems uploaded, but total hits, so it's just an indicator that the majority of explorers probably use software that automatically uploads their finds as the discoveries are made. (Although I confess I have no idea about the default settings in EDDiscovery, so if I'm wrong there, do please correct me!)
An interesting aside: that EDSM total was 8.8% regular and 4.4% on Console. In other words, a third of the EDSM hits came from its console version - quite a lot more than I expected.
But then, if a Commander discovers an average of 200 systems daily, and only uploaded once a month, that would produce a 6,000 systems "spike" on one day. (This would make them "produce" more systems in a month than most explorers do in a year.) There aren't really such, the only ones that happen are the weekend spikes that come like clockwork.
So, anyone want to save up a lot of systems then sell them all on a Wednesday?
Last edited:
Yeah one additional caveat with the dates is that they appear to change if data for that record changes, which includes a second visitor corroborating the coordinates. The vast majority of systems only get visited by one commander, unless they're near something popular, or along one of the popular routes. Once those have been visited by a second commander, then the date is usually more stable, as far as I know.
The effect is that some of the older history in the database probably gets "eroded" over time, appearing like slightly newer activity than it should. I don't think the effect is terribly noticeable.
The effect is that some of the older history in the database probably gets "eroded" over time, appearing like slightly newer activity than it should. I don't think the effect is terribly noticeable.
Hm, I thought that the dump date doesn't update if no new information is added - no more precise coordinates, no total bodies count, and so on. But again, I could be wrong on this.
There's actually data on this without me having to do anything now. Compare the current "Monthly activity" subsheet (on the sheet in the first post) to the "Extended snapshot" subsheet, which is data from five months ago. Generally, the difference between then and now is around 1% - and that includes going all the way back to 2017 April.
Hm, another thing I just remembered: we can use the EDSM ID that is in the dumps to judge if a system might be "revisited" or not. The ID isn't randomized, but auto-incrementing. A quick look at 2019 December (the oldest one I found lying around without having to download the entire thing, plus convert it from JSON) showed that around 10% of daily systems were likely revisited. To determine that, I would take the highest ID of a day's systems, then count how many systems had an ID lower than that on the next day.
Of course, many of them appear to come from the vicinity of the bubble.
But say, @Orvidius: could you run this on the whole database please?
Of course, many of them appear to come from the vicinity of the bubble.
But say, @Orvidius: could you run this on the whole database please?
Right, I think it's the only exception. I think Anthor as a field for "validated coordinates" (yes/no) that isn't in the dumps, but counts as an update on the second visitor, but not again afterward.
Huh, that might be interesting to see. I'll see if I can script something up to make a report on this.
I think it's also if the total bodies count was added, even though that's not in the dumps either. So for example, you could have a system visited before the FSS, validated by someone else before the FSS, then visited again with the FSS, now with a total body count added. But unfortunately, I can't tell this for sure, since the total bodies info isn't available in any dumps.
OK, I have a spreadsheet with this now. It's happening a lot more than I thought it was, but I guess I shouldn't actually be surprised. If an updated body-count also causes an update, then that would certainly account for more dates getting pushed into the future.
But what's interesting of course is that this makes it possible to make an adjusted/estimated date based on the ID ranges. It still probably won't be 100% accurate, since if the highest-ID systems are the ones that get updated, then the entire range scale will slide accordingly. However it'll give much more accurate dates than we have at the moment. So what I might do is add an "estimated date" column on my side, which I can then also put into the spreadsheets, and maybe use that for the exploration history time lapse videos too, or whatever.
But what's interesting of course is that this makes it possible to make an adjusted/estimated date based on the ID ranges. It still probably won't be 100% accurate, since if the highest-ID systems are the ones that get updated, then the entire range scale will slide accordingly. However it'll give much more accurate dates than we have at the moment. So what I might do is add an "estimated date" column on my side, which I can then also put into the spreadsheets, and maybe use that for the exploration history time lapse videos too, or whatever.
Yeah, it would be quite interesting, and to compare videos of the same dates using estimated dates and not.
Oh, and as a fun note: if you plot the adjusted "new" systems (total - revisited), then the big outlier on 2017. Sept. 20 disappears. The one there was a whopping 93372 systems likely revisited, out of 112113. I'll add some interesting charts to the sheet later.
If everything works correctly, I'll have the new version of the discovery history spreadsheets added with additional columns. It has the same columns as before, but added to it are a new adjusted systems column, and new body/star/planet averages based on that.
For the bodies, it's still using the EDSM dates since it looks like the bodies don't have nicely sequential ID numbers for some reason. My script makes a mess of it if I try to figure out a timeline from that. I'm wondering if EDSM did some strange back-dating on later ID numbers, or some bodies were removed and re-inserted with new IDs. Whatever the case, there are large ID numbers early in the timeline, and each day's maximum zig-zags up and down the scale.
For the bodies, it's still using the EDSM dates since it looks like the bodies don't have nicely sequential ID numbers for some reason. My script makes a mess of it if I try to figure out a timeline from that. I'm wondering if EDSM did some strange back-dating on later ID numbers, or some bodies were removed and re-inserted with new IDs. Whatever the case, there are large ID numbers early in the timeline, and each day's maximum zig-zags up and down the scale.
I made some edits and new charts then, based on erosion. First, there's a new column called Adjusted Systems: it's calculated as the systems minus the probable revisits, and there's also a multiplier that you can tweak for the dates before console support was added. If you wish to try different values, make a copy of the sheet and modify the value, it's on the far right. (I used +20% by default, depending on whether you look at DW2, EDDN hits or squadron leaderboards, it could be anywhere within 20-30%. If you just wish to ignore this altogether, set it to 1.)
The calculated ratios that used systems all use the adjusted value now. The difference is really quite little across all of them, both there and most anywhere else, except during DW2: if you take a look at the comparison chart which includes both adjusted and dumped system numbers, there difference in revisits increases significantly in the time between DW2 started and when it left the core. So unfortunately, things look worse there.
I'll probably do next month's update and the squadron update together, since they ended up quite close to each other anyway.
The calculated ratios that used systems all use the adjusted value now. The difference is really quite little across all of them, both there and most anywhere else, except during DW2: if you take a look at the comparison chart which includes both adjusted and dumped system numbers, there difference in revisits increases significantly in the time between DW2 started and when it left the core. So unfortunately, things look worse there.
I'll probably do next month's update and the squadron update together, since they ended up quite close to each other anyway.
Here come this month's update and this squadron season's update as well. But first, something more interesting I noticed while running the counts of ELWs. Usually, I just did it for every new month, but this time, I decided to take a look again, and was surprised at how many were gone from earlier times, to the point that I went to double-check if I mixed something up. But it doesn't look that way. In other words, it looks like that ELWs can be revisited and reuploaded as well, and at that time, they'll show up again at their new date.
I did the same for AWs, where the same results came, although the differences were much smaller. Looking at it over time, it seems that pre-FSS times were the ones which "lost" ELWs, and post-DW2 the ones which gained (starting from May though) - with DW2 actually losing some as well. But let's not forget that this is shifted by time. I think the difference between ELWs and AWs is probably from people using the Road to Riches tool and/or simply ignoring or not going after the much less valuable (in terms of credits) AWs.
In any case, this unfortunately means that the ELW and AW data on the chart is worth somewhat less now, so take it with another spoon of salt. I debated removing them entirely, but decided not to. It might still be informative.
Let's see, what else... In terms of systems, March was a return to previous form and stability, which I suppose means that the influx of new players mostly wasn't retained. Speaking of which, the Steam player count increase from the fleet carriers price announcement doesn't seem to have materialised at all in exploration, which of course makes sense, but I thought I'd include it here. "Go exploring" isn't what people tell others to do if they want that five billion, "go mining in Borann" is. Plus with fleet carriers not looking very interesting for exploration, it doesn't look like many had stopped exploring and went to the asteroids either.
Moving on to squadrons then, there has been an increase there since last season's all-time low. Competition has at least returned to the top, and it was quite close between #2-#4. As a fun fact, if any of them made the same amount of points last season, they would have been #1 then. Activity and competition have also improved under the top 10, so it's all across the board then.
I'm looking forward to this new season's results, since it should end just around where the fleet carrier update release is expected: we'll see if the rush for credits will have had any effect on squadron exploration, and can compare that to EDSM uploaders' activity.
I did the same for AWs, where the same results came, although the differences were much smaller. Looking at it over time, it seems that pre-FSS times were the ones which "lost" ELWs, and post-DW2 the ones which gained (starting from May though) - with DW2 actually losing some as well. But let's not forget that this is shifted by time. I think the difference between ELWs and AWs is probably from people using the Road to Riches tool and/or simply ignoring or not going after the much less valuable (in terms of credits) AWs.
In any case, this unfortunately means that the ELW and AW data on the chart is worth somewhat less now, so take it with another spoon of salt. I debated removing them entirely, but decided not to. It might still be informative.
Let's see, what else... In terms of systems, March was a return to previous form and stability, which I suppose means that the influx of new players mostly wasn't retained. Speaking of which, the Steam player count increase from the fleet carriers price announcement doesn't seem to have materialised at all in exploration, which of course makes sense, but I thought I'd include it here. "Go exploring" isn't what people tell others to do if they want that five billion, "go mining in Borann" is. Plus with fleet carriers not looking very interesting for exploration, it doesn't look like many had stopped exploring and went to the asteroids either.
Moving on to squadrons then, there has been an increase there since last season's all-time low. Competition has at least returned to the top, and it was quite close between #2-#4. As a fun fact, if any of them made the same amount of points last season, they would have been #1 then. Activity and competition have also improved under the top 10, so it's all across the board then.
I'm looking forward to this new season's results, since it should end just around where the fleet carrier update release is expected: we'll see if the rush for credits will have had any effect on squadron exploration, and can compare that to EDSM uploaders' activity.
I'm having an issue with Earthlike planets. I've been out there for over a year and not come across one Earthlike. Lots of water worlds, but zero earth likes. I wonder if there has been a tweek somewhere behind the seems. Not sure how possible that is though. Really want to find an Earthlike around a gas giant or an Earthlike with rings, but I'm not even getting a regular one.
I don't think anything has changed. I have found lots of ELW, sometimes two almost in successive systems and other times thousands of Ly without a sniff. I always try to filter out as many 'low probability' systems as I can - keep mainly A, F, G, (plus O, B, NS for interest) and K if I need it to plot say a 1k route. The first ringed ELW (in fact anytime) is a thrill
Just my bad luck. Hi hum, I'll just keep going.I don't think anything has changed. I have found lots of ELW, sometimes two almost in successive systems and other times thousands of Ly without a sniff. I always try to filter out as many 'low probability' systems as I can - keep mainly A, F, G, (plus O, B, NS for interest) and K if I need it to plot say a 1k route. The first ringed ELW (in fact anytime) is a thrill
@Max Factor: that's off-topic here, next time please post in a relevant thread instead. There were no tweaks to the Forge, so it's just you. If you need help on how to find Earth-like worlds, see my guide on the subject. Given enough systems, you'll find one, but you can improve your chances.