So you can set max trades to 1 and it will only give you direct trade routes from source to destination. However it sounds like you want something else... one (or more) trade routes along the route but not necessarily from source to destination. You might want to have an empty cargo hold at times to save time? When I was implementing the tool I wondered if it would ever/often be more profitable in terms of profit/hour to skip around with an occasionally empty cargo hold. It turns out that this rarely the case. Since ETN functions by maximizing profit/hour, it's not quite clear how to give you what you are looking for. Once interesting feature would be to maximize profit/hour for routes with total time capped at a specific value. That way you could get to your destination in a reasonable time but still make some nice cash on the way. I'll have to give this some thought but it might be doable without too much work.
So you can set max trades to 1 and it will only give you direct trade routes from source to destination. However it sounds like you want something else... one (or more) trade routes along the route but not necessarily from source to destination. You might want to have an empty cargo hold at times to save time? When I was implementing the tool I wondered if it would ever/often be more profitable in terms of profit/hour to skip around with an occasionally empty cargo hold. It turns out that this rarely the case. Since ETN functions by maximizing profit/hour, it's not quite clear how to give you what you are looking for. Once interesting feature would be to maximize profit/hour for routes with total time capped at a specific value. That way you could get to your destination in a reasonable time but still make some nice cash on the way. I'll have to give this some thought but it might be doable without too much work.
Awesome news! ETN is my favorite trading tool!
Last edited:
Fantastic! Thank you very much, ETN is easily my favourite trading tool!
Seems to be working again now (03:36 UTC). And the prices seem a bit more current too. Hope this is a good sign.
EDIT: No, broken again at 05:05 UTC.
Last edited:
Performance Improvements
Fixed now. It was a combination of a few things:
1. Increased demand since proper support for the new commodities was implemented.
2. Slower queries due to drastically larger database.
3. My server (AWS T2.small) was running at near-zero CPU credits so it was unable to boost to handle spikes in usage.
It took a few days, but I implemented some clever techniques to speed up the four primary ETN queries by an order of magnitude. Each time you hit the search button, ETN needs to fetch nearby systems, stations, buy prices and sell prices from the database, which is around 600 MB in MySQL at present, with 700,000 total systems! The trade search algorithm is far too expensive to run on all those systems, so I limit it to the 1024 systems closest to the source system and destination system (if there is one). This forms an ellipsoid with the source and destination systems as focus points. Filtering out these 1024 systems was the bottleneck for ETN, because I had to calculate the distance between every star and the source + destination. Distance calculation involves a good amount of math including square root calculation which is slow, and even scanning 700,000 rows in a table isn't fast. So I needed a way to get to the systems quicker. There are spatial indexes implemented specifically for this purpose, but a lot of them (like the spatial extensions for MySQL) are for 2D and spherical datasets only. All I needed was a simple 3D r-tree but I didn't want it in memory (T2.small only has 2 GB RAM) and MySQL was already running for WordPress. I decided to try a MySQL composite index (b-tree) on xyz. Surprisingly it does a really good job with range queries.
In order to find my ellipsoid with a range query I needed an estimate on the size of the ellipsoid - a bounding box. Using the backs of several envelopes, I came up with a formula for the appropriate size of an ellipsoid given the distance between the focus points and target volume. I was able to estimate <target volume> = <target star count> / <stellar density>. This gave me a cubic polynomial to solve. Haven't done that in a while! After a few mistakes and some ugly formulas I threw my function into Wolfram Alpha and got a nice closed form solution. Well, relatively nice:
Armed with my bounding box, I was able to do the range query and limit the distance calculations to a few thousand stars near the target area. Queries now take 0.5 seconds instead of 5. The site is much more responsive and the server has a big buffer of CPU credits once again.
Fixed now. It was a combination of a few things:
1. Increased demand since proper support for the new commodities was implemented.
2. Slower queries due to drastically larger database.
3. My server (AWS T2.small) was running at near-zero CPU credits so it was unable to boost to handle spikes in usage.
It took a few days, but I implemented some clever techniques to speed up the four primary ETN queries by an order of magnitude. Each time you hit the search button, ETN needs to fetch nearby systems, stations, buy prices and sell prices from the database, which is around 600 MB in MySQL at present, with 700,000 total systems! The trade search algorithm is far too expensive to run on all those systems, so I limit it to the 1024 systems closest to the source system and destination system (if there is one). This forms an ellipsoid with the source and destination systems as focus points. Filtering out these 1024 systems was the bottleneck for ETN, because I had to calculate the distance between every star and the source + destination. Distance calculation involves a good amount of math including square root calculation which is slow, and even scanning 700,000 rows in a table isn't fast. So I needed a way to get to the systems quicker. There are spatial indexes implemented specifically for this purpose, but a lot of them (like the spatial extensions for MySQL) are for 2D and spherical datasets only. All I needed was a simple 3D r-tree but I didn't want it in memory (T2.small only has 2 GB RAM) and MySQL was already running for WordPress. I decided to try a MySQL composite index (b-tree) on xyz. Surprisingly it does a really good job with range queries.
In order to find my ellipsoid with a range query I needed an estimate on the size of the ellipsoid - a bounding box. Using the backs of several envelopes, I came up with a formula for the appropriate size of an ellipsoid given the distance between the focus points and target volume. I was able to estimate <target volume> = <target star count> / <stellar density>. This gave me a cubic polynomial to solve. Haven't done that in a while! After a few mistakes and some ugly formulas I threw my function into Wolfram Alpha and got a nice closed form solution. Well, relatively nice:
Code:
x = 1/6 ((3 sqrt(3) sqrt(27 v^2-4 d^3 v)-2 d^3+27 v)^(1/3)/2^(1/3)+(2^(1/3) d^2)/(3 sqrt(3) sqrt(27 v^2-4 d^3 v)-2 d^3+27 v)^(1/3))-d/6Armed with my bounding box, I was able to do the range query and limit the distance calculations to a few thousand stars near the target area. Queries now take 0.5 seconds instead of 5. The site is much more responsive and the server has a big buffer of CPU credits once again.
Great job, Josh!
Thank you for your time and the great tool you made Josh!
There are a few things, I'd like to ask:
1.) If you ever quit this project, would you please give others the sources and some documentation to continue your great work?
2.) I guess a real-time update of EDDB/N Data isn't possible, as I guess you've implemented a 24h timer for updates - could that update period be decreased (e.g. 2 Hours?) to have a bit more actual data? Or is it impossible due to traffic and load?
Why: for me it often happens, that I end up in a corner of the bubble, where etn.io only gives me very few options, and most of them are based on outdated information (commodities no longer available, changed prices, etc.). So I pick up a Sidewinder and do missions to check the stations and ports in the neighbourhood for updates (EDMC always on). When done with all that missions (I usually try to visit around 50 destinations within 30ly around) I get back to my t9 the same evening, I get 7-2 days old date and next day I have much more options, but outdated again;(
3.) There is an option to set the ships weight. Do you take the loaded or the empty weight into account?
For the optimized T9 it tells me 1681tonnes, when visiting outfitting in a station, it shows me 2430 tonnes.
Thanks again! Great work Josh!
There are a few things, I'd like to ask:
1.) If you ever quit this project, would you please give others the sources and some documentation to continue your great work?
2.) I guess a real-time update of EDDB/N Data isn't possible, as I guess you've implemented a 24h timer for updates - could that update period be decreased (e.g. 2 Hours?) to have a bit more actual data? Or is it impossible due to traffic and load?
Why: for me it often happens, that I end up in a corner of the bubble, where etn.io only gives me very few options, and most of them are based on outdated information (commodities no longer available, changed prices, etc.). So I pick up a Sidewinder and do missions to check the stations and ports in the neighbourhood for updates (EDMC always on). When done with all that missions (I usually try to visit around 50 destinations within 30ly around) I get back to my t9 the same evening, I get 7-2 days old date and next day I have much more options, but outdated again;(
3.) There is an option to set the ships weight. Do you take the loaded or the empty weight into account?
For the optimized T9 it tells me 1681tonnes, when visiting outfitting in a station, it shows me 2430 tonnes.
Thanks again! Great work Josh!
wow awesome work man! Keep it up!
There is one thing I don't understand.... when I select "trade loop only" the last trade does not take me back to my original station... is this a bug or i'm just not getting what this option means?
You need to put your original station in the "destination" box.
I noticed that there is some bug with G 89-32, It always ask me to take 116 tons of Superconductors at that station and bring them to toolfa. The problem is that here is only just a few tons there. I think the problem lies with EDDN. Since I use E : D Market Connector and that the system data says it is up to date when I dock... Thrudd's has the same issue also.
Last edited:
Digging deeper this looks like a programming bug on both ETN and Thrudd's because on https://eddb.io/station/market/22 the quantity is right... does ETN check for availability before suggesting a trade?
Last edited:
There may be two issues at play here:
1. Because the stock is low, one ship can completely clean out the supply very easily. The stock numbers reported are correct at a certain point in time. The stock levels increase slowly over time recovering to the standard level. Prices are reported/updated by cmdrs using a tool to EDDN but the messages then need to be received and processed by the various databases. There may be an issue receiving EDDN messages from time to time and there also may be different timing by different databases processing the messages. This can result in different databases reporting different numbers but also more importantly the numbers will be different from in-game.
2. The companion API used by ED Market Connector and other tools sometimes does not report market data when there is a market at the outpost. This results in no updated market data being sent to EDDN and thus databases don't get updated market data.
In any case, use of a tool where you can manually change the stock level when necessary can be useful so that trade runs can then be generated avoiding the location with bad data. I don't know how often ETN updates its data, but I have had the same problem with it considering stations with obviously old data. With Thrudds, you can edit the price/stock data manually if you have enough access.
Great tool !
The only thing i miss is something to copy the name of the system to clipboard (like in eddb.io )
- - - Updated - - -
or the ability to select the name of the system with my mouse
The only thing i miss is something to copy the name of the system to clipboard (like in eddb.io )
- - - Updated - - -
Great tool !
The only thing i miss is something to copy the name of the system to clipboard (like in eddb.io )
or the ability to select the name of the system with my mouse