In-Development TradeDangerous: power-user trade optimizer

[Snake Man]

Funny thing, the first real search for trade routes came up empty, I searched for current system I'm in: Parjanya,Volterra Enterprise which is not included in the csv if I'm correct. How do we go about adding these, like is maddavo adding systems daily or?

EDIT: the system is included it seems, but the station is missing: trade.py: ERROR: No stations at origin system, PARJANYA

 

[ArcM]

Thank you for the clarifications guys!

Funny thing, the first real search for trade routes came up empty, I searched for current system I'm in: Parjanya,Volterra Enterprise which is not included in the csv if I'm correct. How do we go about adding these, like is maddavo adding systems daily or?

You can manually add systems that are not there in Systems.csv with notepad (or similar). If the system is already there, then you just need to add this line in Stations.csv:
'Parjanya','Volterra Enterprise',0

0 refers to the distance from the star, but if you want you can leave it as 0.

 

[ArcM]

Hm.. How long should one wait between uploading TradeDangerous.prices to Maddavo, and then re-downloading the 3 files?

Because I did so, and apparently I've lost all the data I had updated manually.

 

[psuamier]

Hm.. How long should one wait between uploading TradeDangerous.prices to Maddavo, and then re-downloading the 3 files?

Because I did so, and apparently I've lost all the data I had updated manually.

At least 5min, could take longer now, the file is already quite large. I always make a backup before overwriting the prices file.

 

[Snake Man]

When you update a station,

$ trade.py update -GF whiteorbital

when you're done, it puts the changes into a file called "updated.prices" so you can easily upload just your changes back to maddavo's site.

Do you mean that any price updates we do, has to be processed through maddavo's site?

If so... umm that's bad, I mean if I'm trading and have updated price info which effects my routes as this happens almost hourly, then I have to wait X amount of time until my edits are processed through... that cant be it, heh please tell me so

Now I used trade.py update aulin/enterprise --editor nano, did some updates and it said Copy of changes saved as 'updated.prices', I tried to do trade.py import updated.prices and it processed something but no effect in the database.

 

[xlg47]

Import works now, but the first run command fails (note that there were very many unknown star systems):

kfsone-tradedangerous-94576a2c9be9>trade.py import --plug=maddavo --opt=syscsv --opt=stncsv
NOTE: Rebuilding cache file: this may take a moment
NOTE: Missing "kfsone-tradedangerous-94576a2c9be9\data\TradeDangerous.prices" file - no price data
Connecting to server: http://www.davek.com.au/td/System.csv
data/System.csv: 1.393.426/1.393.426 bytes |  45.40KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/Station.csv
data/Station.csv: 42.684/42.684 bytes |  20.56KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/prices.asp
import.prices: 3.603.361/3.603.361 bytes |  42.90KB/s | 100.00%

There is an issue with import including the csv files. It doesn't rebuild the database/cache before it tries to import new prices.

you will need to manually rebuild the cache with the new csv files, then reimport the prices until the rebuild step is added to the import when you use either the --opt=syscsv or --opt=stncsv switches

trade.py buildcache -f
trade.py import --plug maddavo --skipdl

Edit: Above should work, but I got an error doing the buildcache.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 253: invalid continuation byte

I'll keep looking into it. I guess until this is bullet proof, manual download of System.csv and Station.csv is the better option

 

[gazelle]

Do you mean that any price updates we do, has to be processed through maddavo's site?

No, only if you want to share the data immediately. Every change you are doing via the "update" command will be in the database and used for the next calculation.

 

[Coolts]

I get an error with this new command.
.
trade.py import --plug=maddavo --opt=syscsv --opt=stncsv
.
...NOTE: Rebuilding cache file: this may take a moment
.
...TradeDangerous.prices:813 Unrecognized STAR/Station, '36 URSAE MAJORIS/ARISTOTLE PLATFORM'.
.
I'm guessing its falling over either because a new station has been added or a typo'd name?
.
**EDIT**

manually downloading all three files works fine so it looks like the update code needs to wait for all 3 to finish downloading before going any further? Just a guess.

 

[Snake Man]

Ah I see it does update them right away, sorry about that I was looking at wrong field

When station's prices are updated, they are ALL updated (time stamped), you cannot just change one commodity, you have to check them all?

updated.prices file is saved and overwritten every time you run update, so you have to update, then upload to maddavo, update and upload to maddavo... loop this as many times needed?

 

[gazelle]

When station's prices are updated, they are ALL updated (time stamped), you cannot just change one commodity, you have to check them all?

It's the preferred way to update all items, yes.

I don't know about the internal gui but if you're using an editor you could add "-T" to the command line options. This will add the timestamp to each item line. You than can only correct the items you want and remove the timestamp or set it to "now".

updated.prices file is saved and overwritten every time you run update, so you have to update, then upload to maddavo, update and upload to maddavo... loop this as many times needed?

If you want to share it immediately, yes. You could also keep playing and only upload your "data/TradeDangerous.prices" at the end of your gaming session.

 

[Snake Man]

Next problem I ran into was operating Lakon Type-9 which requires large landing pad. Is there any way to --avoid all stations with only small/medium pads available?

 

[psuamier]

Next problem I ran into was operating Lakon Type-9 which requires large landing pad. Is there any way to --avoid all stations with only small/medium pads available?

To my knowledge the database does not have this information (only one trading tool actually has it AFAIK, but this one is much better for round trips).

 

[maddavo]

Some people had recently reported using EliteOCR for updating prices. I just installed it and spent the evening updating prices from about 30 stations. It works very well.

For those who are not familiar with this tool, see this thread . You can use screenshots of the market screen and it will convert them to a TD-compatible .prices file which can then be imported to TD or uploaded to the sharing site.

 

[wolverine2710]

To my knowledge the database does not have this information (only one trading tool actually has it AFAIK, but this one is much better for round trips).

Curious: Which one?

 

[iotzan]

I'm having an issue when trying to import the system and station files from maddavo's. Importing and rebuilding cache for just the prices file works fine.

Here's the output for pulling the stncsv & syscsv

Spoiler

trade.py import -www -vvv --plug=maddavo --option=stncsv --option=syscsv -i
# DB Cache is up to date.
# Loading data
# Connecting to DB
# Loaded 20.329 Systems
# Loaded 1.337 Stations
# Loaded 15 Ships
# Loaded 14 Categories
# 'consumertechnology' alias for #6 'Consumer Items/Consumer Technology'
# 'domesticappliances' alias for #7 'Consumer Items/Domestic Appliances'
# 'basicnarcotics' alias for #22 'Legal Drugs/Narcotics'
# 'heliostaticfurnaces' alias for #28 'Machinery/Microbial Furnaces'
# 'agriculturalmedicines' alias for #32 'Medicines/Agri-Medicines'
# 'hazardousenvironmentsuits' alias for #66 'Technology/H.E. Suits'
# 'terrainenrichmentsystems' alias for #67 'Technology/Land Enrichment Systems'
# 'nonlethalweapons' alias for #77 'Weapons/Non-Lethal Weapons'
# 'reactivearmour' alias for #79 'Weapons/Reactive Armour'
# Loaded 79 Items, 9 AltItemNames
# Max ship jump distance: Anaconda @ 19.50
Connecting to server: http://www.davek.com.au/td/System.csv
# Server: Microsoft-IIS/5.0
X-Powered-By: ASP.NET
Connection: close
Date: Thu, 18 Dec 2014 16:50:44 GMT
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Thu, 18 Dec 2014 02:22:15 GMT
ETag: "c89fae70691ad01:1b19"
Content-Length: 1393426


data/System.csv: 1.393.426/1.393.426 bytes |  91.27KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/Station.csv
# Server: Microsoft-IIS/5.0
X-Powered-By: ASP.NET
Connection: close
Date: Thu, 18 Dec 2014 16:51:01 GMT
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Mon, 15 Dec 2014 22:16:42 GMT
ETag: "a9681eceb418d01:1b19"
Content-Length: 42684


data/Station.csv: 42.684/42.684 bytes |  39.43KB/s | 100.00%
Connecting to server: http://www.davek.com.au/td/prices.asp
# Server: Microsoft-IIS/5.0
X-Powered-By: ASP.NET
Connection: close
Date: Thu, 18 Dec 2014 16:51:05 GMT
Content-Type: application/octet-stream
Accept-Ranges: bytes
Last-Modified: Thu, 18 Dec 2014 16:38:15 GMT
ETag: "25ce8e5e11ad01:1b19"
Content-Length: 3673134


import.prices: 3.673.134/3.673.134 bytes |  74.29KB/s | 100.00%
No new data - nothing to do - doing nothing.

And the error I get when attempting to rebuild cache right after this:

Spoiler

NOTE: Rebuilding cache file: this may take a moment
Traceback (most recent call last):
  File "D:\Games\TD\trade.py", line 75, in <module>
    main(sys.argv)
  File "D:\Games\TD\trade.py", line 48, in main
    tdb = tradedb.TradeDB(cmdenv, load=cmdenv.wantsTradeDB)
  File "D:\Games\TD\tradedb.py", line 335, in __init__
    self.reloadCache()
  File "D:\Games\TD\tradedb.py", line 409, in reloadCache
    cache.buildCache(self, self.tdenv)
  File "D:\Games\TD\cache.py", line 824, in buildCache
    processImportFile(tdenv, tempDB, Path(importName), importTable)
  File "D:\Games\TD\cache.py", line 721, in processImportFile
    for linein in csvin:
  File "C:\Python34\lib\codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 253: invalid continuation byte

Here's more info: https://www.dropbox.com/s/e9znqbipf8ydkhl/TD_SYSSTN_data.zip?dl=0
I have included the files that come with TD, the ones from maddavo's that I get from direct download as well as the files that are created when using the import function.
It appears that importing them with the update command will reformat them and change the encoding to Macintosh ANSI.

I'm using the latest code as of this posting. If this is a confirmed issue I will open a ticket on bitbucket, looking for confirmation first.

 

[OvrTehBottm]

In my first attempt at pulling down the tradedangerous.prices file resulted in the following error:

6 stations updated:Ekonir/Morey Vision, Ross 986/Jones Station, HILL PA HSI/Reilly Terminal, Yakabugai/Levi-Montalcini City, Apoyota/Hahn Relay, TABIT/Haber City
C:\Trade\trade.py: import.prices:7921 Invalid demand supply value: 0L. Got: Unrecognized units/level value: L, expected '-', '?', or a number followed by a level (L, M, H or ?).

I'm unsure where to go from here or if the db is fine.
.
On another note, please verify the sequence for importing station information:
1) Ensure both System and Station are found within their respective csv's.
2) If System is missing additional steps must be utilized to add System to the community project.
3) If Station is missing add appropriate lines to Station.csv (Ex. 'Secoya','Joy Port',0)
4) type trade.py import <filename> (ex. import.prices -- if using EliteOCR)
.
Please let me know if the above is correct.
.
Thank you for providing such an excellent tool, even if this particular user is just this side of clueless.

 

[kfsone]

Right now it appears that there are some problems with Maddavo's data that he is probably working on.

In the short term, I would recommend that you revert to the .csv files that come with TD and use the following command line:

trade.py import --plug=maddavo -q

This will import the data from Maddavo's .prices file, and won't annoy you with the warnings about items for stations that aren't yet in the DB.

I'll message Dave with a note about what the problems are.

 

[kfsone]

The latest code has fixes for:

- need to rebuild cache after downloading .csv files, before trying to parse the new .prices file,
- "0L" now means "0" (actually using "0L" is going to require more work),

 

[Daniel Jackson]

btw COMBAT STABILISERS are not in the GUI even using the -A command
found a station that buys them but nowhere to type in the price in the program :I

 

[maddavo]

btw COMBAT STABILISERS are not in the GUI even using the -A command
found a station that buys them but nowhere to type in the price in the program :I

Combat stabilisers were taken out when a bit of a cleanup was done on items. They'll be back in an update.

- - - - - Additional Content Posted / Auto Merge - - - - -

There is a problem with the TD import plugin for --maddavo at the moment. Downloading the shared prices, station and system files directly from the website (see sig) into the data folder works.