I've graphed a whole bunch of planet/star scan data from EDDN with a lot of useful insights

As some of you are already aware, I created this site recently:

http://eddne.io

This site fills a niche I couldn't find anyone else filling quite the way I wanted it filled. What I wanted was a serious aggregation of scan data to really understand things like "the likelihood of certain planets appearing around certain star types and mass codes," "which planets harbor the highest concentration of various elements," and "what kind of exploration scanning activity is going on on a day to day basis."

I'm still adding graphs and charts to this site and would love feedback. I have amassed hundreds of thousands of star scans as well as planet scans. I can join these two bodies of data together to come up with all sorts of neat observations. If anyone can think of something I should try to extract from that data, please let me know. If there's some other kind of aggregate data you're looking for that you haven't found some other site providing yet, I'd love to hear about it. I honestly like compiling this stuff more than I like playing video games sometimes, because I'm nerdy like that. [big grin]
 
Great stuff!

Could your method be tailored to create statistics from specific expeditions too? It would mean having a way to draw data from commanders who are on a specific expedition for a set duration. Like taking a snapshot of data from time period A to time period B by commanders XYZ.. so not so much a collation of all the data gathered from E.D.D.N.

Something we're keen to include on the Distant Worlds II Expedition for example, is scientific research. Research both via discovery of new in-game content and anomalies and cataloguing them, but also if possible be collating statistical info on systems charted and elements discovered that fleet members can submit and eventually have presented in charts or graphs specific to DW2.

I think this would be a cool project to have ran alongside major expeditions (for the stats nerds among us if for nothing else! :))
 
Cool stuff ! Another nifty thing you could maybe add is a "per-sector" statistics page, showing things like total body mass recorded, planet/star type counts, that would give an excellent observation point about galaxy mass repartition beyond "there is about that many stars in this area" :p

Great stuff!

Could your method be tailored to create statistics from specific expeditions too? It would mean having a way to draw data from commanders who are on a specific expedition for a set duration. Like taking a snapshot of data from time period A to time period B by commanders XYZ.. so not so much a collation of all the data gathered from E.D.D.N.

Something we're keen to include on the Distant Worlds II Expedition for example, is scientific research. Research both via discovery of new in-game content and anomalies and cataloguing them, but also if possible be collating statistical info on systems charted and elements discovered that fleet members can submit and eventually have presented in charts or graphs specific to DW2.

I think this would be a cool project to have ran alongside major expeditions (for the stats nerds among us if for nothing else! :))

Hey that's an excellent idea there ! It's most definitely possible, given you have a full list of commander names participating to DW2 (to check against other messages sent in the network, and store only the meaningful one for this particular use), and also a little server that can run everything 24/24, even just a small NAS would certainly do that nicely.
 
Last edited:
I'm curious what you're counting on http://eddne.io/#/explorers - I didn't think EDSM sent any scans to EDDN, just consumed them.

Every piece of data coming over eddn has a softwareName key who's value is filled in by whatever software was used to send it. I'm not exactly sure what software is putting "EDSM" in that field but whatever it is, 24% of the scans are coming from it.
 
Great stuff!

Could your method be tailored to create statistics from specific expeditions too? It would mean having a way to draw data from commanders who are on a specific expedition for a set duration. Like taking a snapshot of data from time period A to time period B by commanders XYZ.. so not so much a collation of all the data gathered from E.D.D.N.

Something we're keen to include on the Distant Worlds II Expedition for example, is scientific research. Research both via discovery of new in-game content and anomalies and cataloguing them, but also if possible be collating statistical info on systems charted and elements discovered that fleet members can submit and eventually have presented in charts or graphs specific to DW2.

I think this would be a cool project to have ran alongside major expeditions (for the stats nerds among us if for nothing else! :))

This can be done with the data for sure even in retrospect back to December 15th when I started recording it. I have the uploaderId and timestamps for when the data was uploaded which is all I need in the data itself to tie them to an expedition.

The thing I don't currently have is a mechanism to know which uploaderIds are a part of an expedition, what dates they're a part of it and which expeditions are out there. My guess is that I would have to create such a mechanism because it probably doesn't exist (correct me if I'm wrong). For sanity's sake, it would have to be a self serving mechanism with expedition organizers registering their expedition and then keeping that expedition up to date with who's a part of it, who's not, and when they joined/left. The interesting thing is I can provide data like "last known scan location / time" for members of an expedition and maybe even sync it to a galaxy map or something. You'd end up with a scatter plot of sorts where you can determine if someone has left the expedition or something like if they went back to the bubble.

This is an interesting idea and one I'm definitely interested in looking into but given my day job and all, it'll probably take me months to put it all together. However, if anyone is interested in contributing code and time to this, PM me. Maybe we could roll it out faster.
 
Last edited:
Great work!
If I were you I would also include a link to Jackie Silver's thread regarding the decoding of universal cartographics,
https://forums.frontier.co.uk/showthread.php?t=196297
alongside the one to Marx's guide.

Ah yes, didn't know about that thread. The original reason I did the mass code graphs was because of something I was talking to Marx about on his thread. But that forum post provides a much more in depth look at what's involved with those procedural system names.
 
I never really looked into it, but do EDSM and EDDN have APIs for pulling this kind of data? I'm assuming so, based on this thread. It could be interesting to try to do some things with that, with regards to the expeditions, for sure. Does the data include CMDR names or IDs to tell it apart (I'm assuming that the "scanners" in the graphs refers to commanders)? It looks like it probably has plenty of dates/timestamps.

Assuming that all that is available is the start and stop dates for the expedition, and the list of participants, there would be a great deal of inaccuracy due to people joining late, quitting early, starting early, or whatever else that would cause them to have data from the expedition that is outside of the date range, or non-expedition data that is included in the date range.

In my day job I often write perl scripts to grab data from APIs in JSON or XML, munge it, and shove it into a database, or vice-versa, or from one DB to another. With enough raw data, I'm sure we could have some fun. ;)
 
This is fantastic! Nice work, oldmanklc!

In the grand tradition of "Wonderful! Could you add...", I wonder how feasible it would be to add some way to create custom graphs. Like, if I wanted to graph planet types on a per-region basis (or per-region with filtering to only see them for certain stellar classes, for instance). There have been some interesting papers correlating the presence of gas giants with stellar metallicity, which has been making me really curious if we can extrapolate metallicity of various regions by looking at the number of gas giants in young stars.
 
This is fantastic! Nice work, oldmanklc!

In the grand tradition of "Wonderful! Could you add...", I wonder how feasible it would be to add some way to create custom graphs. Like, if I wanted to graph planet types on a per-region basis (or per-region with filtering to only see them for certain stellar classes, for instance). There have been some interesting papers correlating the presence of gas giants with stellar metallicity, which has been making me really curious if we can extrapolate metallicity of various regions by looking at the number of gas giants in young stars.

It is possible to do that but I back this site with a mongodb in part because I've never used mongodb before and wanted to try it out. It turns out, I've learned the hard way that mongodb is terrible at doing things a relational database could do easily, like aggregating large data sets without using a ton of processing time and other things Amazon would charge me a lot of money for. The current site doesn't have an API for that reason. Instead, I'm just compiling json files with a script on my home PC and uploading them nightly - a task which I will eventually automate.

When and if I migrate away from using mongodb as my database, I'll be better able to do something like what you're talking about without huge wait times and a big AWS bill.
 
EDDB provides https://eddb.io/api, but since the bodies.jsonl isn't being generated anymore it has become obsolete for my own purposes.

EDSM has recently started to provide https://www.edsm.net/en/nightly-dumps, which is really nice (though still missing some data, but that's being worked on by Anthor).

Awesome, thanks. I've started looking through the API docs on EDSM as well, and pulled a few of the dumps to look at. Pretty straightforward stuff, but it lacks commander/discovery information in the systems data, and the commander flight logs are considered private and require API keys to pull. This will limit what kinds of statistics can be generated for expeditions. Interesting though.

But for general statistics like the OP is using, it seems like there's quite a lot to draw on. Pretty cool.
 
Last edited:
Awesome, thanks. I've started looking through the API docs on EDSM as well, and pulled a few of the dumps to look at. Pretty straightforward stuff, but it lacks commander/discovery information in the systems data, and the commander flight logs are considered private and require API keys to pull. This will limit what kinds of statistics can be generated for expeditions. Interesting though.

But for general statistics like the OP is using, it seems like there's quite a lot to draw on. Pretty cool.

The raw eddn object has a message key which contains things like scan data. The object also has a header key which contains a timestamp and uploaderID, among a few other things. EDSM only saves the message without the header at all. The schema for eddn itself can be found here though:

https://github.com/EDSM-NET/EDDN/blob/master/schemas/journal-v1.0.json

In my data, I pipe the uploaderid and timestamp into the message and save the message to my db so I can use it for tying individuals to statistics. The code for my collector is here:

https://github.com/kcgolden/eddn-exploration-data-collector/blob/master/index.js
 
Last edited:
Top Bottom