In-Development TradeDangerous: power-user trade optimizer

Tromador

Tromador
eyeonus said:
No idea. Based on the page, commodities got updated first (5 hours ago) and the rest got updated later (2 hours ago). That corresponds to about midnight UTC for commodities and 3am UTC for the rest, so maybe play it safe and call it 4am UTC?
Click to expand...

It's not pretty but it's functional - http://elite.ripz.org

The timestamps shown are as supplied by EDDB in "Last-Modified", based on them I'm pulling @ 0700 UK time. Any unchanged files (again based on Last-Modified) won't be pulled. I'll be keeping 2 rotated backups of previous versions.
 
Last edited:

Tromador

Tromador
Ran fine yesterday - today not so much.

D:\games\Game Tools\Trade Dangerous>trade.py import -P eddblink -O clean
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "D:\Games\Game Tools\Trade Dangerous\data\TradeDangerous.prices" file - no price data.
NOTE: Processing Upgrades.
NOTE: Downloading file: 'eddb\index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded 0.6MB of gziped data 11.5MB/s
NOTE: Processing Ships.
NOTE: Processing Systems.
NOTE: Processing Stations, this may take a bit.
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
Traceback (most recent call last):
File "D:\games\Game Tools\Trade Dangerous\trade.py", line 104, in <module>
main(sys.argv)
File "D:\games\Game Tools\Trade Dangerous\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "D:\games\Game Tools\Trade Dangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "D:\games\Game Tools\Trade Dangerous\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "D:\games\Game Tools\Trade Dangerous\plugins\eddblink_plug.py", line 616, in run
self.importStations()
File "D:\games\Game Tools\Trade Dangerous\plugins\eddblink_plug.py", line 329, in importStations
upgrade))
sqlite3.IntegrityError: FOREIGN KEY constraint failed
 

eyeonus

E
Tromador said:
It's not pretty but it's functional - http://elite.ripz.org

The timestamps shown are as supplied by EDDB in "Last-Modified", based on them I'm pulling @ 0700 UK time. Any unchanged files (again based on Last-Modified) won't be pulled. I'll be keeping 2 rotated backups of previous versions.
Click to expand...

Okay, but the site says the times are given in UK time. The plugin assumes the "Last-Modified" tag is GMT. (Specifically "%a, %d %b %Y %X GMT".) Anything else will mess it up.
 
Last edited:

eyeonus

E
Tromador said:
Ran fine yesterday - today not so much.

D:\games\Game Tools\Trade Dangerous>trade.py import -P eddblink -O clean
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "D:\Games\Game Tools\Trade Dangerous\data\TradeDangerous.prices" file - no price data.
NOTE: Processing Upgrades.
NOTE: Downloading file: 'eddb\index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded 0.6MB of gziped data 11.5MB/s
NOTE: Processing Ships.
NOTE: Processing Systems.
NOTE: Processing Stations, this may take a bit.
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
Traceback (most recent call last):
File "D:\games\Game Tools\Trade Dangerous\trade.py", line 104, in <module>
main(sys.argv)
File "D:\games\Game Tools\Trade Dangerous\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "D:\games\Game Tools\Trade Dangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "D:\games\Game Tools\Trade Dangerous\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "D:\games\Game Tools\Trade Dangerous\plugins\eddblink_plug.py", line 616, in run
self.importStations()
File "D:\games\Game Tools\Trade Dangerous\plugins\eddblink_plug.py", line 329, in importStations
upgrade))
sqlite3.IntegrityError: FOREIGN KEY constraint failed
Click to expand...

That error means that whatever it was trying to insert into the UpgradeVendor table had an unknown value. Run it again with '-www', that'll spit out the line that's being added to the database so I can see what the data it's trying to insert is.

I'd do it myself, but I found a bug where updating the Vendor tables when the Station table has no updates doesn't happen even with the force option that I'm working on right now.
 
Last edited:

Tromador

Tromador
eyeonus said:
Okay, but the site says the times are given in UK time. The plugin assumes the "Last-Modified" tag is GMT. Anything else will mess it up.
Click to expand...

"Last-Modified" header is always given in GMT per RFC 7232, regardless of the server's localtime, so that is a non-issue (as I assure you my web server is standards compliant).
 
Last edited:

Tromador

Tromador
eyeonus said:
That error means that whatever it was trying to insert into the UpgradeVendor table had an unknown value. Run it again with '-www', that'll spit out the line that's being added to the database so I can see what the data it's trying to insert is.

I'd do it myself, but I found a bug where updating the Vendor tables when the Station table has no updates doesn't happen even with the force option that I'm working on right now.
Click to expand...

I rather thought it was a data problem... let me run that for you...

So I get a vast (obviously) amount of output, here are the lines which lead up to the error. Hopefully that helps.

# upgrade_id:1437,station_id:5258,cost:1437
# upgrade_id:1439,station_id:5258,cost:1439
# upgrade_id:1440,station_id:5258,cost:1440
# upgrade_id:1441,station_id:5258,cost:1441
# upgrade_id:1444,station_id:5258,cost:1444
# upgrade_id:1445,station_id:5258,cost:1445
# upgrade_id:1449,station_id:5258,cost:1449
# upgrade_id:1454,station_id:5258,cost:1454
# upgrade_id:1485,station_id:5258,cost:1485
Traceback (most recent call last): ... etc
 

eyeonus

E
Looks like the upgrade item with the id 1485 doesn't exist in the modules json. I'm almost done with the fix to the vendor update bug, I'll check it in a bit.

EDIT: Hmm, no, it is in the modules json, apparently there's something wrong with the upgrade import, because it's not showing in the Upgrade table. Let me look into it.

Figured out the problem:
# upgrade_id:1485,name: Prismatic Shield Generator,weight:2.5,cost:132200
# upgrade_id:1486,name: Prismatic Shield Generator,weight:5,cost:240340
# upgrade_id:1487,name: Prismatic Shield Generator,weight:10,cost:761870
# upgrade_id:1488,name: Prismatic Shield Generator,weight:20,cost:2415120
# upgrade_id:1489,name: Prismatic Shield Generator,weight:40,cost:7655930
# upgrade_id:1490,name: Prismatic Shield Generator,weight:80,cost:24269300
# upgrade_id:1491,name: Prismatic Shield Generator,weight:160,cost:76933670
# upgrade_id:1492,name: Prismatic Shield Generator,weight:320,cost:243879730

Each instance is replacing the one before, so the only one that stays is the last one, in this case 1492. I'm going to go watch Infinity War right now, I'll figure out the fix when I get back.
 
Last edited:

eyeonus

E
Okay, I think I've fixed it, had to change most of the UNIQUE keys in the tables from "name" to the relevant ID#. I'm testing it now.

EDIT:
Here's an excerpt from the exported Upgrade.csv with the new code, looks like it works to me.

1485,'Prismatic Shield Generator',2.5,132200
1486,'Prismatic Shield Generator',5,240340
1487,'Prismatic Shield Generator',10,761870
1488,'Prismatic Shield Generator',20,2415120
1489,'Prismatic Shield Generator',40,7655930
1490,'Prismatic Shield Generator',80,24269300
1491,'Prismatic Shield Generator',160,76933670
1492,'Prismatic Shield Generator',320,243879730

As you can see, all the versions of the generator are showing now.

You'll want to run with '-O clean', just so the database changes don't muck up your data.

Changelog:
Added Tromador's EDDB mirror as the go-to site for downloading the source files, plugin will automatically fallback to eddb.io as a just in case.
Added the 'fallback' option: "Fallback to using EDDB.io if Tromador's mirror isn't working."
Updated ShipVendor and UpgradeVendor importing to work correctly with the 'force' option, rather than the old behaviour of only updating the vendor if the station itself is updated.
Added 'modified' field to the UpgradeVendor table so as to be able to check if the source json data is newer than the database's.
Changed the UNIQUE keys in most of the tables from 'name' to the relevant ID field to prevent entries with the same name overwriting previous entries.

EDDBlink TD plugin
 
Last edited:

Avi0013

Avi0013
Eyeonus, under what license are you allowing us to use the plugin? If it's FLOSS, you'd probably have an easier time of it if you had it in a github or bitbucket repository :)
 

eyeonus

E
Arg. Merging updates is still ridiculously slow. SQL, why you gotta be like that? I think I need to figure out if it's possible to insert/replace a whole batch at once, rather than one at a time. That would hopefully make things faster....
 

Tromador

Tromador
eyeonus said:
Arg. Merging updates is still ridiculously slow. SQL, why you gotta be like that? I think I need to figure out if it's possible to insert/replace a whole batch at once, rather than one at a time. That would hopefully make things faster....
Click to expand...

Define slow? TD Never created new DB from scratch particularly fast when we had maddavo, so -O clean is entitled to take a few minutes. (A little under 12 mins on my 4.2Ghz I7-7700K)
Once we get the listener and potentially other options running, time consuming imports will become an infrequent task.

Given the timestamp of 06:58 on last night's modules.json, I'm going to move the daily download back a couple of hours and pull at 0900 UK.
 

eyeonus

E
I'm not talking about running with '-O clean'. I said "merging imports". '-O clean' destroys the database and builds a new one from scratch. I ran '-O listings,skipvend,force' yesterday when I went to sleep. ~8 hours later, it was about 12% through processing the updated systems. Each system update - the 'db.execute("INSERT or REPLACE ....)" line - was taking at least 2 or three seconds apiece to run.
 

eyeonus

E
Tromador said:
Given the timestamp of 06:58 on last night's modules.json, I'm going to move the daily download back a couple of hours and pull at 0900 UK.
Click to expand...

Maybe instead of doing that, adjust the cron task (that I assume you're using) to compare the timestamp on EDDB with the local copy, download if newer, and check the timestamp every hour or so?
 

Tromador

Tromador
eyeonus said:
I'm not talking about running with '-O clean'. I said "merging imports". '-O clean' destroys the database and builds a new one from scratch. I ran '-O listings,skipvend,force' yesterday when I went to sleep. ~8 hours later, it was about 12% through processing the updated systems. Each system update - the 'db.execute("INSERT or REPLACE ....)" line - was taking at least 2 or three seconds apiece to run.
Click to expand...

To be clear, that was: trade.py import -P eddblink -O listings,skipvend,force

That took about 10-15 minutes for me, starting with a populated and working dataset. Either I am not correctly repeating what you did, or you have a problem.
What have I missed?
 

CMDR Ibizan

C
Ive just timed my updates. The clean import of everything took 1 hour 45 minutes and the listings update took 46 minutes. That’s seems only slightly longer than the standard full updates from Maddavo’s site, although there were some outrageously long ones that took all night sometimes.
 
You must log in or register to reply here.
Back
Top Bottom