This is not very exciting right now, but I thought I'd share progress on how adding services are going for folks who like boring updates with stats.
Current Progress
Development can be slow going as building a new indexed DB from raw data can take a long time (hours to import and create the data) as there are millions of datapoints and indexing it all in a performant way is just a bit slow after the first few million systems, and I've had to rebuild the index from scratch a few times (locally and in the cloud) to experiment with different approaches to indexing the data in a way that is highly performant, scalable to query and going to be reasonably cheap to run.
Data for Systems, Stations, Trade, Exploration are all stored in different data stores, to support the best balance of performance and being able to run cost effectively (I don't mind spending a few credits but I'm only willing to do it at all if I can do it reasonably cheaply). This turns out to work really well for how data works in Elite Dangerous and the sorts of queries players make - and is also necessary to make queries performance - but requires a little upfront planning and testing that the integrations are right (it's okay if they aren't, it's just slow and annoying to redo work.
The most fun thing was testing different ways to slice up the galaxy to make queries performant. I ended up splitting the galaxy into 'sectors' 100 Ly across (effectively cubes along XYZ axis) to make it much faster to run complex arbitrary queries. Splitting into even smaller chunks has some benefits but also some trade offs and hasn't felt worth it.
Service Stats
Using an import of the >77 million systems in the EDSM data dump to kick start things (with additional new data recorded directly from EDDN as it is logged) and collecting Commodity / Trade data from EDDN directly, I've been running the system for a while and tweaking the software, configuration and hardware. I'm very confident at this rate that the system could tick over happily for the next decade at the current capacity.
There are currently only about 1 million commodities in the trade database. I have had to build this myself as you can't get an export of this data from anywhere else (e.g. INARA doesn't share it). I did have a much larger commodity database but I had to reset that DB last week and didn't bother to preserve the old data as it wasn't worth the time and effort at this stage.
It grows again pretty quickly as players dock at stations, megaships, etc and the old data isn't as useful anyway so I'm not too worried about having restarted the collection process for it, it will take about a month or so to get it all populated (and data from > 48 hours is often not super reliable anyway, although knowing what sort of commodities are likely to be in demand is handy).
Performance
This is an example of how long it takes to do a typical set of queries but in a 200 Ly radius around Sol (a distance which covers about the entire bubble):
Get system location information: 0.159ms
Count number of known systems in galaxy: 527.151ms
Count number of trade commodities: 23.34ms
Find a specific commodity on nearby markets: 51.908ms
Find all nearby systems: 5.286s
Test data: {
originAndDistance: { distance: 200, systemName: 'Sol', x: 0, y: 0, z: 0 },
systemAddress: 10477373803,
systemName: 'Sol',
systemX: 0,
systemY: 0,
systemZ: 0,
systemXGrid: 0,
systemYGrid: 0,
systemZGrid: 0,
source: 'ESDM',
updatedAt: '2019-01-21T02:11:52.000Z'
},
totalKnownSystemsInGalax: '77,250,773',
totalCommodities: '1,098,836',
instancesOfSpecificCommodityOnNearbyMarkets: '3,663',
numberOfNearbySystemsFound: '56,629'
}
Simple queries like getting system location take less than a millisecond. Getting more useful information like commodity data takes about 0.05 of a second. The only query in the examples above that is not a sub-second query is 'Find all nearby systems' which is rather extreme, as it is returning data for 56,629 systems (note: it is not just counting them, it is actually returning useful data for them!). This sort of query scales interestingly with distance.
e.g.
- if you ask for the same data in a 50 Ly radius it will find 1,000-2,000 systems and will take 1-2 seconds
- if you ask for the same data in a 500 Ly radius it will fetch over 500,000 systems and that takes 10-15 seconds
This isn't actually a super useful query but I just point it out it's fun to make it possible to do queries like this at all. In reality most useful queries are either very targeted (e.g. commodities, services, stations, system states, etc) and those are much quicker and all sub-second.
Even in the case of a large search of 500 Ly ICARUS Terminal it would still be able to fetch information about all nearby systems before you had even finished jumping to a location. In a lot of cases data can be prefetched and/or cached, so that even with lots of folks using it, it should happily scale. They way it's designed I can scale as much as I want, but I'm trying to avoid spending much on running costs (so that I can leave it running and ignore it) so am optimizing for fastest possible raw times, cost and scaling.
What's Next
More news on the web based front end (which folks can use instead of / as well as ICARUS) soon, probably in next 2-3 weeks.
Like ESDM does for map data (and unlike what some other sites do...) I'll be making all data available via an API and data dumps if anyone finds that useful.
Probably the data / API will be opened up first. Not sure if ICARUS Terminal integration or a website will come first after that.
I'm happy to share code (ICARUS is entirely open source and folks can clone it, of course), but it's not really sure for a service it's all that useful in isolation with out the infrastructure to run it on and it ends up slowing down development as it's a bit of a time sink to deal with support queries so I'll likely get round to that later once it's public in some form.
