Release Neutron Highway long range route planner

Feature Request: Would you please create a plotter tool that solves the "most efficient path" problem? For example, trying to get to 7 engineers in one trip, what's the most efficient order based on distances. Your Neutron Mapper, with its Add Via waypoints, has the beginnings of this functionality.
 
Feature Request: Would you please create a plotter tool that solves the "most efficient path" problem? For example, trying to get to 7 engineers in one trip, what's the most efficient order based on distances. Your Neutron Mapper, with its Add Via waypoints, has the beginnings of this functionality.
try edpathfinder
 
Feature Request: Would you please create a plotter tool that solves the "most efficient path" problem? For example, trying to get to 7 engineers in one trip, what's the most efficient order based on distances. Your Neutron Mapper, with its Add Via waypoints, has the beginnings of this functionality.
That's actually on the list of things I'm going to do, the road to riches pathfinder actually works like that anyway, so the actual code is basically done.
 
Fantastic job on the route planners Cmdr Spansh! (y):) ...currently using the NS Highway to speed my return from Beagle Point on the DW2 expedition. Along the way, I made an add-on so I don't have to toggle between apps to get the next waypoint - see video below. I expect there are things out there that do this already, but happy to make it available if anyone thinks it might be useful.

Source: https://youtu.be/A85y2zLZHd4
 
Mister Spansh, have a shower of compliments for your tool! I especially love the 'bodies' search, which is now my go-to when going for certain minerals (now that eddb.io doesn't do more than 25 light years)

I now have fun searching for bad data so I can clean up EDSM a bit. (For example, planets with zero gravity, zero distance from arrival star, that sort of thing) and in so doing I discovered something: Sometimes a bad planet is -removed- from EDSM's database, but is not removed in your copy of the database...
For example if you search for zero-gravity worlds, you'll get Ceeckia DY-I b29-0 one, two and three. That system has been cleaned up in EDSM.net, some hand-placed planets with wrong ID were removed.


I would propose a cleanup pass once a month or some such. ;) There's some crazy folks out there hunting down bugs in the database, correcting planet names, what have you. 'The right kind of crazy' or so I've been told.
 
Mister Spansh, have a shower of compliments for your tool! I especially love the 'bodies' search, which is now my go-to when going for certain minerals (now that eddb.io doesn't do more than 25 light years)

I now have fun searching for bad data so I can clean up EDSM a bit. (For example, planets with zero gravity, zero distance from arrival star, that sort of thing) and in so doing I discovered something: Sometimes a bad planet is -removed- from EDSM's database, but is not removed in your copy of the database...
For example if you search for zero-gravity worlds, you'll get Ceeckia DY-I b29-0 one, two and three. That system has been cleaned up in EDSM.net, some hand-placed planets with wrong ID were removed.


I would propose a cleanup pass once a month or some such. ;) There's some crazy folks out there hunting down bugs in the database, correcting planet names, what have you. 'The right kind of crazy' or so I've been told.
I was actually doing automatic daily full refreshes, but a few updates ago it was starting to cause issues with the live site. I believe I have mostly solved those issues now but I'm still loathe to re-enable automatic refreshes so I'm trying to kick off one or two a week manually (there's one running now). Once I'm confident it won't cause any live issues (I also have some ideas about minimising the impact further) then I'll re-enable the automatic daily upload (This is only for the bodies search, systems search, road to riches and soon to come stations search, the neutron plotter and trade planner update live).
 
Thank you! Oh, small mishap: That Ceeckia system is found by doing a search for planets with zero radius, that gives a small list that I'm trying to correct to zero. ^^
 
Yes! Ceeckia corrected, thanks a lot!

and... That's odd, some things that were corrected in EDSM are still showing up oddly in the spansh website... Let me throw you a list, in case it helps crack down on a bug. (all found from doing a 'planets with zero radius' search)


Streau Eop SS-B c16-1 AB 1 reads as zero-radius in spansh but has proper data in EDSM.
M7 Sector MV-J b10-0 1 4 is a badly named planet that's been removed in EDSM, still visible here.
Hill Pa Hsi A2 : Ditto, another badly named world removed from EDSM.
Ngobe B10 : third 'badly named world' removed from EDSM, still visible in spansh.

Praea Theia RA-D d13-1 2a will suffer the same fate in the future, just trying to get an explorer to scan that system properly so I can ask the EDSM folks to delete the bad planet. le sigh, that one could take a while.

Anyway, thanks a ton for your help in cracking down on bad data! ^_^
 
Last edited:
Yes! Ceeckia corrected, thanks a lot!

and... That's odd, some things that were corrected in EDSM are still showing up oddly in the spansh website... Let me throw you a list, in case it helps crack down on a bug. (all found from doing a 'planets with zero radius' search)


Streau Eop SS-B c16-1 AB 1 reads as zero-radius in spansh but has proper data in EDSM.
M7 Sector MV-J b10-0 1 4 is a badly named planet that's been removed in EDSM, still visible here.
Hill Pa Hsi A2 : Ditto, another badly named world removed from EDSM.
Ngobe B10 : third 'badly named world' removed from EDSM, still visible in spansh.

Praea Theia RA-D d13-1 2a will suffer the same fate in the future, just trying to get an explorer to scan that system properly so I can ask the EDSM folks to delete the bad planet. le sigh, that one could take a while.

Anyway, thanks a ton for your help in cracking down on bad data! ^_^
It's likely removed bodies will take longer to purge, as EDSM only generates the full bodies file once a week, as such it'll be around until he regenerates it (as I import the EDSM exports).

Basically, whatever is in EDSM is exactly what I import (from scratch every time, I don't keep data around for that index) there's no data left over to be wrong. If it's in the EDSM exports it's on my site. That said, there is a slight lag with the full bodies files (new and updated bodies will get data as there is a patch file, deleted will have to wait for a full regeneration which EDSM currently does once a week).
 
Wall of text alert, for those who say TL;DR. The site has been slow and has some issues, I've already improved quite a lot of things and am working to improve things further.

So I'm aware there has been some slowness over the last week or two (Especially yesterday when I was trying a few things), after a fair amount of work yesterday I believe I have mitigated most of the issues. Listed below are a few of the issues related to speed, along with a few other issues which have come to light.

1. Loading 1.2million Neutron Stars into the plotter was causing a few issues with the database every 10 minutes or so causing routes to take up to 5+ minutes to generate, I've added a second caching layer which is populated every 10 minutes from the database, routes are now taking a few seconds to generate once more.
2. The website was checking for completed routes every second, causing a small amount of load on the api. This wasn't causing major issues but was adding to the problems when the site was under load. I've altered this so it now throttles down gradually. This means that it'll check for completed routes like so. 1 second, 2 seconds, 4 seconds, 8 seconds, 16 seconds, 30 seconds (it will then remain at 30 seconds). This means that in the worst case scenario routes may seem like they are taking longer (as the site won't check for up to 30 seconds if the route has been completed), but will still generate. Under normal conditions people hopefully won't notice.
3. An upgrade to Elasticsearch caused some of my queries to underperform for the Road to Riches (in some cases taking up to 30 seconds to generate even when not under load). Some aggressive optimisation and tweaks have brought this back down to 1-2 seconds.
4. A while back I optimised my Elasticsearch import (what causes the Road to Riches and the Systems and Bodies searches to be up to date), this brought the time taken to import the roughly 100GB of data to go from 12+ hours to 4 in the best case scenario. The same upgrade to Elasticsearch along with the increased import time caused the entire backend machine to slow down quite severely. I have throttled this import process severely such that it will once again take quite a long time but should not impact the live site. This isn't a long term solution but should make sure that the data is up to date whilst still allowing the site to work.
5. When the backend was under several of the API calls which gather data to display the site (the range sliders, dropdown boxes etc) could take a long time to generate. Especially on the bodies page which has 15-20 of those that could cause some quite awkward display issues. The other fixes I have done should alleviate this issue for the meantime.
6. Not backend related, but the Neutron Star map on the site now brings almost any browser to a halt (1.2million stars is a lot). I have an improved version of that map which needs some final polish. It should be substantially more performant and will also allow partial maps to be displayed whilst the data to display is being loaded. I'm not sure when I'll get time to finish that off however.
7. The user interface has atrophied quite a lot with recent additions and looks quite ugly in a lot of places.
8. Neutron routes generated around the edges of the galaxy often plot people through the voids around the spiral arms.
9. New Neutron Stars were not being picked up by the plotter. This was due to a simple coding error on my part and has now been fixed. This has meant the site was only using 700,000 Neutron Stars rather than the 1.2million it was supposed to be using. I've reprocessed things and filled in the missing data so routes should now be substantially better.
10. When generating an A to B Road to Riches you often end up with a large number of stops at the start and the finish. This is due to the shape used to pick out viable systems (a cylinder), so you can end up going 90 degrees to the way you want for quite some time. I'd like to do a second processing step to even this out a little. As such I'm going to attempt to change the shape from a plain Cylinder to a shorter Cylinder with two half spheres on either end (like a sausage shape). This should mean that when people regenerate a route from a new place it won't immediately take them directly away from where they intended to go.
11. The results URLS for the plotter are very long. I'm in two minds about this so I wouldn't mind some feedback. I don't particularly mind the long URLs and they show you exactly what the route was without having to load the page. It would be relatively easy for me to save the parameters for each generated route in the database and have them loaded. It would require a small extra load step when loading but should not cause much extra time. If people really prefer this I'll look at adding it.

With regards to Elasticsearch I have several other options I'm considering for making sure the Elasticsearch instance (which, to be honest is much too small for the amount of data I'm currently throwing at it) remains performant. These include

1. Generating the index on a different machine then transferring it over. The downside of this would be having to bring down the index to refresh.
2. Keeping the index up to date on the fly rather than a full import every day, then refreshing the entire index once a week. The downside of this is that incorrect data which needs to be removed will remain in the index for longer, potentially generating invalid or incorrect routes.
3. Increasing the hardware so the index has more processing power. The downside of this is obviously cost.

On the other issues I am hoping to get some time to do the following.

1. Routes through the void. I'm hoping to generate a mask for the Neutron plotter which will avoid and route around the voids in the galaxy. Now that the data we have on the galaxy is substantially more complete (though still only 0.001% of the whole), I should be able to generate a mask and adjust the plotter to use a combined A*/Geographic method to route people around the voids. This will likely involve a preprocessing step to break the route down into parts, so that when you generate a route it will automatically break it down into sections to route around the voids, then generate individual routes inbetween those sections. Since these sections are fairly (read: completely) static a lot of this precalculation can be done beforehand.
2. Search user interface ugliness/complicatedness. Because of the sheer number of fields on the search pages there is a lot of space taken up by fields people probably don't use for every search (I imagine most people probably search on 2-5 fields). Rather than have all of those fields available all of the time I'm probably going to split the form into 2 parts. A list of fields and the actual fields. This technique is related to faceted search and you can see something similar to what I'm aiming for here ( https://blog.linkedin.com/2009/12/14/linkedin-faceted-search ). Because some of the numbers on those fields are quite long, I'm still intending to have the fields displayed on the main page, but instead of displaying the search form at the same time as the results, I'm intending to have the results page on it's own, with a small summary of the search performed at the top. This will mean people don't have to scroll down as far in order to see their results.
3. Route results pages can break quite severely on smaller windows/mobile devices. Due to the number of fields I show on the route results pages (especially on the road to riches and trade planner) the tables get quite drastically cut off. I've already added a "hide the search" button to allow people to see only the results rather than the search. I'm likely going to take this further and stop displaying the forms on the results pages at all, instead displaying a summary of the route). In both this and the previous point there will be no load time to bring up the search again because of how the site is built. There will be a link/tab which allows people to switch between the form and the results instantly (it's loaded and will still have the data, just hidden).

All of this is related to problems/issues with the site. I am still intending to improve things further and add extra features like.

1. Fuel Stops. Probably the biggest remaining ask from people. This will make the form more complicated as I will need to know the fuel tank size and a few other details in order to calculate when you need to refuel.
2. Non primary Neutron Stars. There are occasionally Binary (or greater) systems with Neutron Stars which are close to the main star which are currently not used, these systems are actually often ideal as they can allow a refuel and supercharge at the same time (Also systems which have a main neutron star and a refuelable star close by). These are relatively rare and probably won't come up much but I'd like to do some more investigation.
3. Stations search. Self explanatory.
4. An A to B route planner. Just a generic, give me an example route from A to B so I can see how far away I am roughly without logging into the game.
5. Visit these places route planner. Often people have a list of places they'd like to visit and want to do them in an optimal order to minimise time. The code for this is essentially identical to the Road to Riches, I just need to make it a little more generic.

The last thing I'd like to say is thank you to everyone who has been donating to the site to help out and also to those people who have emailed me to report bugs or other issues. I try to stay on top of the site performance and pick out bugs but people emailing me to alert me (or ping me on the EDCD discord or fuel rats IRC) to make sure I know about things is not a bad idea.
 
"Non primary Neutron Stars. There are occasionally Binary (or greater) systems with Neutron Stars which are close to the main star which are currently not used, these systems are actually often ideal as they can allow a refuel and supercharge at the same time (Also systems which have a main neutron star and a refuelable star close by). These are relatively rare and probably won't come up much but I'd like to do some more investigation. "

Yes, PLEASE!!!!!

As always, great work you have going here, its much appreciated!
 
I spent some time last night tweaking the Neutron Plotter and managed to get a 4-5 fold speedup, route generation is now taking a quarter of a second or less in most cases. There is still a little issue with loading the data but the two layers of cache I have put in minimise that.

With routes now generating that quickly, it does open up the possibility of heuristically deriving the efficiency to produce the best route which I will consider at some point so that people don't have to guess as much.

There is the possibility that in some very extreme cases it generates slightly less optimal routes but I have not been able to produce one. All the cases I checked produce the same route. It is likely to happen when there are large neutron gaps (1000+ LY) in the route, though those are likely to be across the void anyway which the router already has some problems avoiding. If someone does think they've found one please send me a link so I can investigate. Though the best way to alleviate that would be to find some neutron stars in that gap.
 
Superb work as always Spansh, many compliments...

As a big user of the 'bodies' search, I can't help but have a question: When I search for landmarks the site offers little maps under the 'landmarks' column results. I have to ask: where do you get that data from? I usually use Extool plugin within EDMC to 'give data' about that sort of thing. But it seems you're using another source as Extool's contains more.

So my question: Where do you get the landmarks data? How do I help fill things out?
133447
 
Superb work as always Spansh, many compliments...

As a big user of the 'bodies' search, I can't help but have a question: When I search for landmarks the site offers little maps under the 'landmarks' column results. I have to ask: where do you get that data from? I usually use Extool plugin within EDMC to 'give data' about that sort of thing. But it seems you're using another source as Extool's contains more.

So my question: Where do you get the landmarks data? How do I help fill things out?
View attachment 133447
I use the data from Canonn which I believe is extool so you're contributing anyways. There is some data in there (mostly older stuff) which I can't import due to it missing collaborating information (like system/body name etc) but 90+% of it should be there. What makes you think that there is data missing?
 
Superb work as always Spansh, many compliments...

As a big user of the 'bodies' search, I can't help but have a question: When I search for landmarks the site offers little maps under the 'landmarks' column results. I have to ask: where do you get that data from? I usually use Extool plugin within EDMC to 'give data' about that sort of thing. But it seems you're using another source as Extool's contains more.

So my question: Where do you get the landmarks data? How do I help fill things out?
View attachment 133447
He pulls that data from our Canonn APIv2, If you would like to you can install our EDMC Plugin: https://github.com/canonn-science/EDMC-Canonn

EXTool is in the process of merging with Canonn R&D but with no solid ETA on that yet. We will take all of the current EXTool and add it to our existing Canonn Data thus in the future EXTool will use the Canonn API as well.
 
As so many have told you over the years- thank you for dedicating your time and effort to creating such a phenomenal set of tools. It was one of the first tools I heard about when I picked up ED in November and I haven't looked back since. I can't thank you enough (and I need to figure out how to donate for its maintenance/your sacrifice).

Question (which I'm sure you've heard before, but I couldn't find references throughout this long thread):

When I run the neutron plotter, I find that even though my jump range is set to the current amount for my ship (45, not to be confused with the unladen range of 47), the number of jumps between neutron stars is often off by a significant amount- 25% or more in most cases.

For example, it said the first neutron star from my current system was 4 jumps away, but when I input the route on my galaxy map, it says it's 7. Once I got to the next one, the plotter said the next neutron was 21 jumps away- when I plot it on the galaxy map (using Fastest Routes + Use Jet Cone Boost, since using FSD boost "Supercharged" always refuses to work and says it's out of my jump range), my route is now 29 jumps long.

Over time on long trips, these extra jumps add up. Is there something I'm doing wrong when calculating my routes (I'm optimizing the efficiency % so the number of jumps it the lowest value possible). I do top off my fuel scoop every time I enter a system, but I would expect that the 45 ly jump range is taking that into account since I know my next jump point before I ever start scooping, and my route does not change as I get more fuel into my tank.

Thank you in advance!
 
Last edited:
Top Bottom