Yes, essentially skip the first and last line, and strip off the trailing comma of the other lines and you have one json object per readline, you can simply do it in your code as well, I just found the sed method more efficient. If you use pysimdjson (https://pypi.org/project/pysimdjson/) you potentially even get better performance than my rapidjson streaming parse
I'm afraid I don't have anything to do with any of the plugins which use my site. You'll have to contact the plugin author.
The original Spansh plugin hasn't been maintained for a while and stopped working some time ago after a specific EDMC update. There is a fork of the plugin that seems to be working.
Here is a short install help: https://forums.frontier.co.uk/threa...-neutron-highway-in-edmc.518339/post-10074492
I have been working on an upgrade to the site (which I have been putting off for a while). It does not add any new features but brings brings parts of the infrastructure I use up to date. I have done my best to ensure it does not break anything, but with such a large change there could be a number of problems.
As such I have updated the beta site so people can test it and notice any problems https://beta.spansh.co.uk/. I would very much appreciate people trying to use the beta site and making bug reports (either on this thread, on discord, or via github at https://github.com/spansh/elite-dangerous-issues/issues)
After upgrading this infrastructure there are a few more behind the scenes upgrades I'd like to make, but those changes might make it possible to resume work on the standalone app for the site (no promises).
As such I have updated the beta site so people can test it and notice any problems https://beta.spansh.co.uk/. I would very much appreciate people trying to use the beta site and making bug reports (either on this thread, on discord, or via github at https://github.com/spansh/elite-dangerous-issues/issues)
After upgrading this infrastructure there are a few more behind the scenes upgrades I'd like to make, but those changes might make it possible to resume work on the standalone app for the site (no promises).
Last edited:
I have same problem too, altrough after like 20 minutes waiting i got route plotted once
Same problem here. Wonder what's up?
Update: Mine finally went through after a long time as well.
Things that make you go "Hmmmmmm..."
Things that make you go "Hmmmmmm..."
Thanks for the info! Made me realize how much I've come to depend on your stuff...
Just out of interest, how big is the star database so far and is it in danger of becoming too big at any stage? It's a big old galaxy out there. Can the infrastructure cope with billions of stars?
I believe in the area of 400 million systems.
It is unlikely I can cope with the entire galaxy the way I currently do things. We anticipate that the galaxy size is 400 billion systems. If I miraculously managed to compress every system into only 10 bytes then the galaxy would take up 4TB of space which would in theory be doable (but 10 bytes is not a lot of space to store all of the information about a system).
Currently we know roughly 105million systems which contain 400million stars/planets (and even more with rings, belts and barycentres) and the entire data warehouse takes roughly 500GB. That means the average size of a system is roughly 5KB. At that size trying to store the entire galaxy would take up 2PB (petabytes) of space.
That is just for the data warehouse, the indexed part of the galaxy which powers the search is currently 600GB, so for the entire galaxy it would probably be a similar amount (2PB). That only covers storage though, I would almost certainly have to increase CPU power by adding more machines to the search to keep it responsive, potentially dozens.
I have other indexes as well which power some of the more modern plotters but those would probably be able to cope without much change. Those currently use 40 bytes per system although if I was pushed I could probably cut that down to 16 bytes for no change in functionality.
That is just for the space, it would almost certainly take more processing power to index all of that as well, and I would be unable to produce the galaxy dumps each night which currently take 5 hours to generate.
All that said, with the current rate of exploration I don't think we're in any danger of getting anywhere close to running me out of space. Also at some point we'd have enough data that we could probably reverse engineer enough of the stellar forge to correctly predict star positions (if people were so inclined and determined to work it out). That could reduce storage costs by several orders of magnitude as well.
Currently we know roughly 105million systems which contain 400million stars/planets (and even more with rings, belts and barycentres) and the entire data warehouse takes roughly 500GB. That means the average size of a system is roughly 5KB. At that size trying to store the entire galaxy would take up 2PB (petabytes) of space.
That is just for the data warehouse, the indexed part of the galaxy which powers the search is currently 600GB, so for the entire galaxy it would probably be a similar amount (2PB). That only covers storage though, I would almost certainly have to increase CPU power by adding more machines to the search to keep it responsive, potentially dozens.
I have other indexes as well which power some of the more modern plotters but those would probably be able to cope without much change. Those currently use 40 bytes per system although if I was pushed I could probably cut that down to 16 bytes for no change in functionality.
That is just for the space, it would almost certainly take more processing power to index all of that as well, and I would be unable to produce the galaxy dumps each night which currently take 5 hours to generate.
All that said, with the current rate of exploration I don't think we're in any danger of getting anywhere close to running me out of space. Also at some point we'd have enough data that we could probably reverse engineer enough of the stellar forge to correctly predict star positions (if people were so inclined and determined to work it out). That could reduce storage costs by several orders of magnitude as well.
Last edited:
That was my guess.
I would also speculate that the current rate of expansion of your database is probably now in single-digit percentages per year? (In other words, slower than the annual rate of increase of disk space and CPU power for a typical server setup.)
Yes, though part of me hopes that never happens (another part of me is surprised it hasn't happened already!
).Hmmm, I don't know about that, I think reverse engineering procedural generation is really hard. Good hashing algorithms are supposed to be hard to crack as they are also used for crypto - sure, you don't need to use a crypto-level hash, but there is very little reason not to use a reasonably good one as the overhead isn't that great.Yes, though part of me hopes that never happens (another part of me is surprised it hasn't happened already!
On the other hand, we have the code on our hard drives so someone could pick it apart, but that isn't reverse engineering. That's disassembling or decompiling.
hey, just a thought, TOTALLY understand if it cant be done due to the sheer size of the game but can we get an update on the road to riches plotter that avoids thargoid controlled areas?? i keep getting spanked by those ugly SOBs every time i try to do a riches route