Release Trade Dangerous (Est. 2015) Power user's highly configurable trade optimizer
- Thread starter Tromador
- Start date
Now that there are five environment variables with 'TD_SERVER', 'TD_FALLBACK', and 'TD_SHIPS' recently added, I'm thinking there should probably be an update on the wiki that talks about them. Question is, where? Somewhere on an existing page, on its own page, both?
TD_SERVER = Location of the TD server.
(Currently 'http://elite.tromador.com/files/'.)
TD_FALLBACK = Location of the EDDB.io server.
(Currently 'https://eddb.io/archive/v6/'.)
TD_SHIPS = Location of EDCD ship info file.
(Currently 'https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json'.)
Setting the environment variable allows updating these web addresses by
the user without needing to wait for TD to update.
(Currently 'http://elite.tromador.com/files/'.)
TD_FALLBACK = Location of the EDDB.io server.
(Currently 'https://eddb.io/archive/v6/'.)
TD_SHIPS = Location of EDCD ship info file.
(Currently 'https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json'.)
Setting the environment variable allows updating these web addresses by
the user without needing to wait for TD to update.
Last edited:
That's not enough information to help you.
Firstly, is your copy of TD up-to-date?
Secondly, what is the command you ran TD with when you encountered the error?
Thirdly, what is the console output of the execution of that command?
Finally, the best place for this is the Issues on TD's Github.
Firstly, is your copy of TD up-to-date?
Secondly, what is the command you ran TD with when you encountered the error?
Thirdly, what is the console output of the execution of that command?
Finally, the best place for this is the Issues on TD's Github.
Last edited:
No worries. This isn't a bad place. After all, this is the thread for discussing TD and TDH. It's just the dedicated Issues on Github is better, as it's specifically for that purpose. 
Okay, from testing, looks like a data issue on the server. The Foreign Key that's failing is the station id 66469.
There's no station with id 66469 on EDDB, so I don't know why that's happening.
My old testing database that hasn't been updated in a very long time says that's the station id for Dixon Station, did that station get destroyed?
Okay, from testing, looks like a data issue on the server. The Foreign Key that's failing is the station id 66469.
There's no station with id 66469 on EDDB, so I don't know why that's happening.
My old testing database that hasn't been updated in a very long time says that's the station id for Dixon Station, did that station get destroyed?
Last edited:
Let me have a look...
I have a RABAKSHANY/Dixon Station dated 2019-02-21 02:32:50
It's in listings-live.csv for some reason. I'll clean it out from there and run clean against the sanitized data.
(on an unrelated note, apparently the Chinese really want to hack our TD server)
Edit - station_id 66469 no longer in server files. If it reappears, then it's dodgy data arriving from the feed.
I have a RABAKSHANY/Dixon Station dated 2019-02-21 02:32:50
It's in listings-live.csv for some reason. I'll clean it out from there and run clean against the sanitized data.
(on an unrelated note, apparently the Chinese really want to hack our TD server)
Code:
140.143.71.191 - - [24/Feb/2019:11:11:11 +0000] "GET /cmd.php HTTP/1.1" 404 205 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36"
140.143.71.191 - - [24/Feb/2019:11:11:13 +0000] "GET /cmv.php HTTP/1.1" 404 205 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36"
140.143.71.191 - - [24/Feb/2019:11:11:13 +0000] "GET /cmdd.php HTTP/1.1" 404 206 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36"
140.143.71.191 - - [24/Feb/2019:11:11:13 +0000] "GET /knal.php HTTP/1.1" 404 206 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.91 Safari/537.36"
etc...Edit - station_id 66469 no longer in server files. If it reappears, then it's dodgy data arriving from the feed.
Code:
[elite@quoth files]$ awk '{print $2}' listings-live.csv | grep 66469
[elite@quoth files]$ awk '{print $2}' listings.csv | grep 66469
Last edited:
I made a mistake and said Dixon Station when I meant to say Dixon Colony:
station_id:"66469", name:"Dixon Colony", system_id:"10358", ls_from_star:"1483", blackmarket:"N", max_pad_size:"M", market:"Y", shipyard:"N", modified:"2019-01-20 12:28:18", outfitting:"N", rearm:"N", refuel:"Y", repair:"Y", planetary:"N", type_id:"4"
system_id:"10358", name:"Kamba", modified:"2019-01-20 12:49:02"
And now for the really weird thing:
https://eddb.io/station/70679
Why did Dixon Colony suddenly go from having id 66469 to having id 70679? Did EDDB.io change the id for some unknown reason, or did TD screw up? What the heck, man?
[EDIT] For comparison, here's how it looks in the up-to-date DB:
"70679", "Dixon Colony", "10358", "1483", "N", "M", "Y", "N", "2019-02-20 10:18:14", "N", "N", "Y", "Y", "N", "11"
The station_id, modified, and type_id are different between the old and new DB, and what's really weird, the older DB data says it was updated more recently than the new DB data.
Argh.
[EDIT2] And the best part: According to EDDB.io, the primary star of the "Kamba" system is called "Mbooni", which is also what the system is called on EDSM, just like how every other system in ED is named after the primary star, so something really weird is going on with "Kamba", and I'm thinking it's on EDDB's end.
[EDIT3] Yup. EDDB.io done messed up something. It has TWO Kamba's.
The Kamba that is actually Kamba: https://eddb.io/system/8716382
The Kamba that is actually Mbooni: https://eddb.io/system/10358
Let's see if I can at least figure out how to fix Mbooni to be Mbooni in ROSS....
station_id:"66469", name:"Dixon Colony", system_id:"10358", ls_from_star:"1483", blackmarket:"N", max_pad_size:"M", market:"Y", shipyard:"N", modified:"2019-01-20 12:28:18", outfitting:"N", rearm:"N", refuel:"Y", repair:"Y", planetary:"N", type_id:"4"
system_id:"10358", name:"Kamba", modified:"2019-01-20 12:49:02"
And now for the really weird thing:
https://eddb.io/station/70679
Why did Dixon Colony suddenly go from having id 66469 to having id 70679? Did EDDB.io change the id for some unknown reason, or did TD screw up? What the heck, man?
[EDIT] For comparison, here's how it looks in the up-to-date DB:
"70679", "Dixon Colony", "10358", "1483", "N", "M", "Y", "N", "2019-02-20 10:18:14", "N", "N", "Y", "Y", "N", "11"
The station_id, modified, and type_id are different between the old and new DB, and what's really weird, the older DB data says it was updated more recently than the new DB data.
Argh.
[EDIT2] And the best part: According to EDDB.io, the primary star of the "Kamba" system is called "Mbooni", which is also what the system is called on EDSM, just like how every other system in ED is named after the primary star, so something really weird is going on with "Kamba", and I'm thinking it's on EDDB's end.
[EDIT3] Yup. EDDB.io done messed up something. It has TWO Kamba's.
The Kamba that is actually Kamba: https://eddb.io/system/8716382
The Kamba that is actually Mbooni: https://eddb.io/system/10358
Let's see if I can at least figure out how to fix Mbooni to be Mbooni in ROSS....
Last edited:
As I sanitised station_id 66469 from the server dataset, I can't help with the diagnosis (I probably should have backed it up, but I was half asleep and "just fixed it").
Recommend we put it down to sunspots until/unless we have a recurrence of something similar. It's just as likely that it came from the feed as TD messing up.
Recommend we put it down to sunspots until/unless we have a recurrence of something similar. It's just as likely that it came from the feed as TD messing up.
The more I look into this, the more certain I am that this originated in EDDB, and regardless of the cause, I still need to figure out how to prevent future similar things from breaking TD like this one did.
[EDIT] Okay, so, technically, the problem originated with FDev. The 2.3 update included fixing the fact that there were two different systems that were both called Kamba. The one with stations was left alone. The one without any stations was renamed Mbooni.
Snippet of the changelog:
The problem is, this means that Kamba & Mbooni have been broken on EDDB for basically the entire existence of EDDB- or at least the entire time that Kamba and Mbooni existed in ED are were both called Kamba, because EDDB doesn't deal well with multiple systems having the same name. (Which makes sense, since every system is supposed to have a unique name, there shouldn't be a need to handle that situation.)
This also means Kamba is still broken, because EDDB now thinks that Mbooni is where the stations are, I assume at least in part because Mbooni is still called Kamba on EDDB.
I'm hopeful that once EDDB fixes Mbooni to be called Mbooni, most if not all the other problems associated with this name duplication will sort themselves out.
[EDIT] Okay, so, technically, the problem originated with FDev. The 2.3 update included fixing the fact that there were two different systems that were both called Kamba. The one with stations was left alone. The one without any stations was renamed Mbooni.
Snippet of the changelog:
The problem is, this means that Kamba & Mbooni have been broken on EDDB for basically the entire existence of EDDB- or at least the entire time that Kamba and Mbooni existed in ED are were both called Kamba, because EDDB doesn't deal well with multiple systems having the same name. (Which makes sense, since every system is supposed to have a unique name, there shouldn't be a need to handle that situation.)
This also means Kamba is still broken, because EDDB now thinks that Mbooni is where the stations are, I assume at least in part because Mbooni is still called Kamba on EDDB.
I'm hopeful that once EDDB fixes Mbooni to be called Mbooni, most if not all the other problems associated with this name duplication will sort themselves out.
Last edited:
The idea is to handle the case where a station used to exist but no longer does. This particular scenario is only one of the possible reasons for that outcome.
The fix I have in mind is simply to compare the list of stations in the DB to the list of stations in the dump file, and run a DELETE on any that don't show in both.
OR, alternatively, DROP the station table at the beginning of the station import, recreate it, and build it from scratch, which might be faster. (Although since the StationItem table has a foreign key to it, that makes this way more difficult to actually do....)
Also, what free time?
The fix I have in mind is simply to compare the list of stations in the DB to the list of stations in the dump file, and run a DELETE on any that don't show in both.
OR, alternatively, DROP the station table at the beginning of the station import, recreate it, and build it from scratch, which might be faster. (Although since the StationItem table has a foreign key to it, that makes this way more difficult to actually do....)
Also, what free time?
Last edited:
You'll have to try both and see. From our experience, rifling through the data and doing a single DELETE or REPLACE seems to take an excessively long time, but building a table is multiple INSERT operations, so I suspect it won't be particularly quick either way.
Maybe that was the advantage Maddavo's system had - all this nonsense was dealt with on the server (probably with a much quicker SQL engine on the back end) so you only ever downloaded a pre-sanitised set of .prices.
I like what we are doing much better - for a variety of reasons, a big one is that it's open and anyone can take over if we get smushed into paste - but it's interesting to ponder history.
Maybe that was the advantage Maddavo's system had - all this nonsense was dealt with on the server (probably with a much quicker SQL engine on the back end) so you only ever downloaded a pre-sanitised set of .prices.
I like what we are doing much better - for a variety of reasons, a big one is that it's open and anyone can take over if we get smushed into paste - but it's interesting to ponder history.
Last edited:
The idea is to handle the case where a station used to exist but no longer does. This particular scenario is only one of the possible reasons for that outcome.
The fix I have in mind is simply to compare the list of stations in the DB to the list of stations in the dump file, and run a DELETE on any that don't show in both.
OR, alternatively, DROP the station table at the beginning of the station import, recreate it, and build it from scratch, which might be faster. (Although since the StationItem table has a foreign key to it, that makes this way more difficult to actually do....)
Also, what free time?
OR we can just skip any station in the listings that aren't in the station table during the listings import. Thanks Gazelle.
Very nice. Reminds me of that one quote at the end of 2001.
Update to TD: Realized the RareItem table never got filled in from the csv file, now it does.
Also made it so rare items show in the normal items list so that they can show up in market listings. This means that you may start seeing rare items show in regular trade runs once the server is updated, but it's not super likely, as a few criteria have to be met: You have to be within range of the station that sells the item (according to your jump distance, max hops, and jump values), and there must be another station, also in range, that has that item listing in its market with a buy price, AND it has to be on the "most profitable" run (i.e., no route with no rare items is more profitable than the one(s) with rare item(s)).
Oh, and no, I still haven't gotten around to the excessively complicated process of automagically updating the RareItem.csv file, so if you use the rares command and don't see a rare item you should, let us know so we can manually update that file with the new rare item.
Also made it so rare items show in the normal items list so that they can show up in market listings. This means that you may start seeing rare items show in regular trade runs once the server is updated, but it's not super likely, as a few criteria have to be met: You have to be within range of the station that sells the item (according to your jump distance, max hops, and jump values), and there must be another station, also in range, that has that item listing in its market with a buy price, AND it has to be on the "most profitable" run (i.e., no route with no rare items is more profitable than the one(s) with rare item(s)).
Oh, and no, I still haven't gotten around to the excessively complicated process of automagically updating the RareItem.csv file, so if you use the rares command and don't see a rare item you should, let us know so we can manually update that file with the new rare item.