In-Development TradeDangerous: power-user trade optimizer

[gazelle]

It should not be that hard to look up the system name from a station name. As much as I thank you for supporting these entries do not help at all in the prices file. Please take these few extra seconds to make your contributions useful.

And how exactly should I do that? The plugin can only work with data it get's from EliteOCR. I'll not include a complete database with systems and stations just because someone is to lazy to input a systemname.

Edit: And just to be clear, the plugin does not upload the data, it only generates a "import.prices" file which should be imported with "trade.py import import.prices". This would do the error checking before it gets into the database and after that you can upload your new TradeDangerous.prices to maddavo.

 

[maddavo]

Prices file now has errors in, referencing a system called !UNKNOWN!

I have removed the station in the !UNKNOWN! system from the database and I have added a bit of a check to see if the system is a known system. I was kinda hoping to avoid checking for a valid system because that requires more maintenance to add valid systems. But hopefully when we have EDDN all working well with TD then we won't need the shared .prices file anymore.

 

[AngusRuffnut]

Incorrect station name

Just thought i should report an incorrect station name (affects the station.csv and prices files)
in MCC 467 there is a station called Ron Hubbard Ring
in the station and prices files it has incorrectly been named RoB Hubbard Ring

 

[maddavo]

Just thought i should report an incorrect station name (affects the station.csv and prices files)
in MCC 467 there is a station called Ron Hubbard Ring
in the station and prices files it has incorrectly been named RoB Hubbard Ring

Thanx - fixed.

 

[kfsone]

I have removed the station in the !UNKNOWN! system from the database and I have added a bit of a check to see if the system is a known system. I was kinda hoping to avoid checking for a valid system because that requires more maintenance to add valid systems. But hopefully when we have EDDN all working well with TD then we won't need the shared .prices file anymore.

Unless I'm missing something, you kind of will: It's a stream of updates.

 

[Daniel Jackson]

im still getting this issue and when ever i delete that station from the station file more and more stations appear

trade.py import -i --maddavo
NOTE: Rebuilding cache file: this may take a moment
*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id
CSV File: tradedangerous\data\Station.csv:133
SQL Query: INSERT INTO Station (system_id,name,ls_from_star) VALUES((SELECT System.system_id FROM System WHERE System.name = ?),?,?)
Params: ['Apalai', 'Gubarev Base', '0']

 

[kfsone]

im still getting this issue and when ever i delete that station from the station file more and more stations appear

trade.py import -i --maddavo
NOTE: Rebuilding cache file: this may take a moment
*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id
CSV File: tradedangerous\data\Station.csv:133
SQL Query: INSERT INTO Station (system_id,name,ls_from_star) VALUES((SELECT System.system_id FROM System WHERE System.name = ?),?,?)
Params: ['Apalai', 'Gubarev Base', '0']

This error is telling you "I tried to add Gubarev Base, but there's no system called Apalai in the database".

Basically you need to choose whether to use the supplied Station.csv and System.csv - you can still use maddavo's .prices file you'll just get some warnings about unknown stations and/or systems. If you specify "-i" to import, it will still report them but it loads the rest of the data into your database.

Or, you need to download the .csv files from maddavo when you do your import -- in theory, someone could extend the current plugin to do exactly that.

If you grab the latest TD you can use this workflow:

git checkout data\System.csv
git checkout data\Station.csv
git pull
trade.py import --opt=syscsv --opt=stncsv --plug=maddavo

HOWEVER: You'll need to work with Maddavo/the files in order to work around issues arising from the way he puts together the files. For instance, his current System.csv has some foreign characters in them which will cause TD to be unable to read the file.

Choose your own difficulty level.

 

[Daniel Jackson]

If you grab the latest TD you can use this workflow:

git checkout data\System.csv
git checkout data\Station.csv
git pull
trade.py import --opt=syscsv --opt=stncsv --plug=maddavo

HOWEVER: You'll need to work with Maddavo/the files in order to work around issues arising from the way he puts together the files. For instance, his current System.csv has some foreign characters in them which will cause TD to be unable to read the file.

Choose your own difficulty level.

i tried that and it got

trade.py import --opt=syscsv --opt=stncsv --plug=maddavo
Connecting to server: http://www.davek.com.au/td/System.csv
data/System.csv: 1,401,854/1,401,854 bytes |  82.40KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/Station.csv
data/Station.csv: 42,684/42,684 bytes |  50.77KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/prices.asp
import.prices: 344/344 bytes | 335.88KB/s | 100.00%
Traceback (most recent call last):
  File "trade.py", line 72, in <module>
    main(sys.argv)
  File "trade.py", line 47, in main
    results = cmdenv.run(tdb)
  File "tradedangerous\commands\commandenv.py", line 78, in run
    return self._cmd.run(results, self, tdb)
  File "tradedangerous\commands\import_cmd.py", line 101, in run
    if not plugin.run():
  File "tradedangerous\plugins\maddavo_plug.py", line 51, in run
    pathlib.Path(self.filename),
  File "tradedangerous\cache.py", line 913, in importDataFromFile
    pricesPath=path,
  File "tradedangerous\cache.py", line 572, in processPricesFile
    items, buys, sells = processPrices(tdenv, pricesFile, db, defaultZero)
  File "tradedangerous\cache.py", line 293, in processPrices
    systemByName = getSystemByNameIndex(cur)
  File "tradedangerous\cache.py", line 264, in getSystemByNameIndex
    """)
sqlite3.OperationalError: no such table: System

 

[NeilF]

I've tried using this a couple of times and found the results odd in two ways.
1) At one station it suggested I split my purchases? That surely doesn't make sense? One will give more profit than the other so get 100% of it?
2) Most times I've used it's resulted in no profit and indeed it told me once to buy a product that wasn't even available at a particular station.

Are these problems down to the prices file simply being out of date? Any suggestions on keeping it upto date? - I've re-downloaded it from the website. Is there a method to get better/more accurate prices etc.

 

[DRY411S]

I've tried using this a couple of times and found the results odd in two ways.
1) At one station it suggested I split my purchases? That surely doesn't make sense? One will give more profit than the other so get 100% of it?
2) Most times I've used it's resulted in no profit and indeed it told me once to buy a product that wasn't even available at a particular station.

Are these problems down to the prices file simply being out of date? Any suggestions on keeping it upto date? - I've re-downloaded it from the website. Is there a method to get better/more accurate prices etc.

1) is correct. It is to maximise the CR spend. You should probably find that it splits because either a) it thinks that there is only a certain amount of the higher priced commodity available to buy, or b) it purchases other commodities because you just don't have enough CR to buy any more of the previous one in the buy list

2) is because the prices file is not up to date. You should keep downloading maddavo's data regularly (or use trade.py import option)

 

[xlg47]

i tried that and it got

trade.py import --opt=syscsv --opt=stncsv --plug=maddavo
Connecting to server: http://www.davek.com.au/td/System.csv
data/System.csv: 1,401,854/1,401,854 bytes |  82.40KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/Station.csv
data/Station.csv: 42,684/42,684 bytes |  50.77KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/prices.asp
import.prices: 344/344 bytes | 335.88KB/s | 100.00%
Traceback (most recent call last):
  File "trade.py", line 72, in <module>
    main(sys.argv)
  File "trade.py", line 47, in main
    results = cmdenv.run(tdb)
  File "tradedangerous\commands\commandenv.py", line 78, in run
    return self._cmd.run(results, self, tdb)
  File "tradedangerous\commands\import_cmd.py", line 101, in run
    if not plugin.run():
  File "tradedangerous\plugins\maddavo_plug.py", line 51, in run
    pathlib.Path(self.filename),
  File "tradedangerous\cache.py", line 913, in importDataFromFile
    pricesPath=path,
  File "tradedangerous\cache.py", line 572, in processPricesFile
    items, buys, sells = processPrices(tdenv, pricesFile, db, defaultZero)
  File "tradedangerous\cache.py", line 293, in processPrices
    systemByName = getSystemByNameIndex(cur)
  File "tradedangerous\cache.py", line 264, in getSystemByNameIndex
    """)
sqlite3.OperationalError: no such table: System

I got exactly the same error when using that command, but when I download the System/Station/Prices file with Firefox and place them in the data folder of TD and run

trade.py buildcache -f

everything runs perfectly with no warnings or errors.

 

[xlg47]

Any suggestions on keeping it upto date? - I've re-downloaded it from the website. Is there a method to get better/more accurate prices etc.

As mentioned above you should download Maddavo's prices file regularly, but also you can manually update the prices for a station (trade.py update "station") and submit the updated.prices file to http://www.davek.com.au/td/ or at the end of a session upload your TradeDangerous.prices to the same site.

The prices file is totally dependant on people submitting updated information.

Use trade.py local --ages "system name" to see how old the data is around you (of course, replace "system name" with your current system)

 

[NeilF]

As mentioned above you should download Maddavo's prices file regularly, but also you can manually update the prices for a station (trade.py update "station") and submit the updated.prices file to http://www.davek.com.au/td/ or at the end of a session upload your TradeDangerous.prices to the same site.

The prices file is totally dependant on people submitting updated information.

Use trade.py local --ages "system name" to see how old the data is around you (of course, replace "system name" with your current system)

Thanks! I've simply downloaded the tradingdangerous.prices file manually and put it in DATA twice so far?

Looking at the OP it mentions:-

trade.py import -i http://kfs.org/td/prices
or
trade.py import -i --maddavo​

Is that the prefered method?

 

[Coolts]

Thanks! I've simply downloaded the tradingdangerous.prices file manually and put it in DATA twice so far?

Looking at the OP it mentions:-

trade.py import -i http://kfs.org/td/prices
or
trade.py import -i --maddavo​

Is that the prefered method?

I use the second one. Don't forget the 2 x .csv files as well. ATM the prices are way out of date but the markets have been al over the place this week. Now the game is out proper and I'm fairly confident FDev aren't gong to remodel 1/2 the systems in each patch, I will start updating as I go. Happy days!!!!!!

 

[NeilF]

I use the second one. Don't forget the 2 x .csv files as well. ATM the prices are way out of date but the markets have been al over the place this week. Now the game is out proper and I'm fairly confident FDev aren't gong to remodel 1/2 the systems in each patch, I will start updating as I go. Happy days!!!!!!

The CSVs are for stations/systems? What's the best method of keeping those upto date then?

 

[Coolts]

I just rt-click "save as.." all 3 files and dump them in my local "data" folder before each elite session. Not exactly elegant, but its early days I guess. When the trade data is up to date this tool is peerless.

 

[psuamier]

1) Concerning "Unknown" Systems:
As you seem to have a station name: You can look it up in the Galaxy Map.
Editing: TradeDangerous.prices is a text file. You can edit if with your favourite Text Editor (I use Notepad++, but I only need to do that very seldom)
Search in the prices file for your Station Name (or "Unknown") and replace it by the System name.

As said: Both System and Station Name are essential requirements for the price data to be useful. I really appreciate the input.

2) Concerning Data exactness:
- Use the relatively new "ages" options from the different commands.
(Right now 2 days old data could be already unreliable due to the recent AI changes that have a large impact on the market data.)
- There can be typing errors (or scan errors). Double check when buying stuff if the price is close to the one expected by the database
If not it should be updated (and uploaded). We're all human and make mistakes
- The tool can have bugs too, so if you see any inconsistencies report here (or open an issue).
- Be especially careful on LOW and MED number of trade items - you easily can make losses here.

3) Data Update:
Station/System.csv from Maddavo's page can be overwritten (be sure to first upload your changed prices though)
TradeDangerous.prices: I make a backup before Update.
The format of the original and Maddavo is different, so it's not easy to find changes between them both.
After having integrated the changes with trade.py you can compare it as the prices file is rewritten in the original format.

 

[NeilF]

Is there a way to ask TD to work out a trade loop? ie: Buy in X and up selling in Y, and then buy in Y and end up selling in X? With 1 or more hops?

- - - - - Additional Content Posted / Auto Merge - - - - -

1) Concerning "Unknown" Systems:
As you seem to have a station name: You can look it up in the Galaxy Map.
Editing: TradeDangerous.prices is a text file. You can edit if with your favourite Text Editor (I use Notepad++, but I only need to do that very seldom)
Search in the prices file for your Station Name (or "Unknown") and replace it by the System name.

As said: Both System and Station Name are essential requirements for the price data to be useful. I really appreciate the input.

2) Concerning Data exactness:
- Use the relatively new "ages" options from the different commands.
(Right now 2 days old data could be already unreliable due to the recent AI changes that have a large impact on the market data.)
- There can be typing errors (or scan errors). Double check when buying stuff if the price is close to the one expected by the database
If not it should be updated (and uploaded). We're all human and make mistakes
- The tool can have bugs too, so if you see any inconsistencies report here (or open an issue).
- Be especially careful on LOW and MED number of trade items - you easily can make losses here.

3) Data Update:
Station/System.csv from Maddavo's page can be overwritten (be sure to first upload your changed prices though)
TradeDangerous.prices: I make a backup before Update.
The format of the original and Maddavo is different, so it's not easy to find changes between them both.
After having integrated the changes with trade.py you can compare it as the prices file is rewritten in the original format.

What the best means of downloading the CSV files to ensure things are as upto date as possible? Just download from http://www.davek.com.au/td/ ?




EDIT:

Ahh there's my problem. The system I'm currently in has price data from a month ago! I suspect the same is true/similar for the surrounding ones to!

Spoiler

@ Kokoimudji/Siodmak Station
+ Chemicals
Hydrogen Fuel 200 203 ? 62086? 2014-11-22 18:41:19
+ Consumer Items
Dom. Appliances 480 502 ? 6337? 2014-11-22 18:41:19
+ Foods
Food Cartridges 37 47 ? 722123? 2014-11-22 18:41:19
+ Legal Drugs
Narcotics 70 79 ? 13027? 2014-11-22 18:41:19
+ Machinery
Atmospheric Processors 371 389 ? 43160? 2014-11-22 18:41:19
Crop Harvesters 2212 2245 ? 10192? 2014-11-22 18:41:19
Marine Equipment 4339 4401 ? 6211? 2014-11-22 18:41:19
Mineral Extractors 441 462 ? 210317? 2014-11-22 18:41:19
Power Generators 484 507 ? 5337? 2014-11-22 18:41:19
Water Purifiers 191 205 ? 816643? 2014-11-22 18:41:19
+ Medicines
Basic Medicines 244 261 ? 97936? 2014-11-22 18:41:19
+ Metals
Cobalt 523 539 ? 668144? 2014-11-22 18:41:19
+ Minerals
Bertrandite 2527 2565 ? 2756? 2014-11-22 18:41:20
Coltan 1391 1429 ? 4372? 2014-11-22 18:41:20
Gallite 1938 1967 ? 3353? 2014-11-22 18:41:20
Indite 2212 2245 ? 3042? 2014-11-22 18:41:20
Lepidolite 559 584 ? 901? 2014-11-22 18:41:20
Rutile 294 309 ? 15792? 2014-11-22 18:41:20
Uraninite 739 761 ? 160419? 2014-11-22 18:41:20
+ Technology
Computer Components 366 383 ? 13379? 2014-11-22 18:41:20
+ Waste
Biowaste 23 30 ? 1221? 2014-11-22 18:41:20
Scrap 35 40 ? 52841? 2014-11-22 18:41:20
+ Weapons
Battle Weapons 6105 6191 ? 1912? 2014-11-22 18:41:20

Now if only a program monitored the screenshots folder, and when files appeared in there it put them into the OCR projects being developed at the moment! And if it spotted the screen shots from the top to the bottom of a stations market it simply fired in the data to update a database realtime.

If all it took to keep the database upto date was 4-5 scroll downs and F10s, imagine that! Hell Voice Attack might even be able to automate the whole thing with a single voice command!

 

[gazelle]

1) Concerning "Unknown" Systems:
As you seem to have a station name: You can look it up in the Galaxy Map.
As said: Both System and Station Name are essential requirements for the price data to be useful. I really appreciate the input.

Only one Problem: Station names are not unique, that's why TD needs both. It would be better to just delete the unknown system because you can't be sure which one the station is in. The Galaxy Map will only show you one even if there are others.

BTW: This is also a reason why I didn't bother to try and guess a system name in TD_Export plugin for EliteOCR. TD might find a system for the station but it can't be sure if it's the right one. Better give the user an error message when he tries to import it, to remind him, that he has to input a system name. Unfortunately the system name is nowhere on the commodity market screen, so EliteOCR can't suggest one for the user.

 

[wolverine2710]

And how exactly should I do that? The plugin can only work with data it get's from EliteOCR. I'll not include a complete database with systems and stations just because someone is to lazy to input a systemname.

Edit: And just to be clear, the plugin does not upload the data, it only generates a "import.prices" file which should be imported with "trade.py import import.prices". This would do the error checking before it gets into the database and after that you can upload your new TradeDangerous.prices to maddavo.

As your tool is plugin for EliteOCR perhaps Seebek is willing to built in in support for TGC (for retrieving systems with an economy) and store the data in a small local sqlite database. That could be used for his existing Approximate string matching algorithm which he currently uses for the commodity names to improve the OCR-ing results for names. What is then needed is an api which can be accessed by the plugin(s) to do their magic. Just my 2 euro cent.