In-Development TradeDangerous: power-user trade optimizer

Tromador · Apr 3, 2015

Just got an X52pro. Now I've got the stuff displaying on it, but I can't figure out how to step through the checklist. Somehow the forum thinks there is no instance of X52 in this thread when I try to search - so apologies for dumb questions.

Update: I tried searching for Saitek.

I take it you've still not had anything back from them after all the foot shuffling.

kfsone · Apr 3, 2015

Code:

v6.17.0 Apr 03 2015
.(tKe) "import" behavior:
   This change makes it so that "import" will not overwrite existing
   data with older values unless the "--reset" option is used.
. (kfsone/tKe) #211 "local" command was a mess
  - Don't double up system names on station lines,
  - Trail station names with dots to make those lines more distinct,
. (tKe) Fixed a bug that caused routes to be pruned when they shouldn't
. (tKe) #145 Support for unit suffixes for credits, e.g.
    1k = 1000, 1.5m = 1,500,000, 1.2b = 1,200,000,000

Phil (tKe) has been slogging away, I've been busy moving offices at work and coping with a little medical emergency.

-Oliver

kfsone · Apr 4, 2015

Code:

v6.17.2 Apr 04 2015
. (kfsone) "buy" command
  - Show ship costs when listing multiple ships,
. (kfsone) "run" command
  - Added "--stock" option to set minimum stock level required,
  - Added extra detail to "--progress" output,
. (gazelle) Bash completition improvements

v6.17.1 Apr 04 2015
. (kfsone) Various minor tweaks
. (kfsone) Improvements to presentation of edscupdate
. (kfsone) Added another 30 or so systems
. (kfsone) edscupdate now ignores any systems that start with "PLAA EURIK"

So, for example, if you use

trade.py run ... --cap 120 --stock 1000 ...

It will only consider stations that have 1000 or more units to sell.

CAVEAT: It will ignore any stations that have "?" as their listed quantity. I just stumbled across a station that was selling Paladium at 12,493 and had 117,000 units. The price was in the EDDB/Maddavo/TD database aged 101 days, the stock was "?". It looks like nobody's been here. Excuse me while I go back to selling every last unit I can haul out of here

Serpentstar · Apr 5, 2015

@Kfsone, Much appreciated. It has made finding better routes much more efficient. I try to update them as often as I see old ones. It is kind of strange that so many stations have old data. Is there a way to look for stations that haven't been specifically updated for months?

kfsone · Apr 6, 2015

Serpentstar said:
@Kfsone, Much appreciated. It has made finding better routes much more efficient. I try to update them as often as I see old ones. It is kind of strange that so many stations have old data. Is there a way to look for stations that haven't been specifically updated for months?

Code:

trade.py old --limit 25
trade.py old --near ross209 --ly 100 --min 90 --limit 25

Serpentstar · Apr 6, 2015

@kfsone So... How difficult would it be to add a demand filter (same as the supply)? heheheh FYI, your supply filter is pretty much dead on. Fantastic addition.

kfsone · Apr 6, 2015

Those of you who are using Maddavo's can use the "local" command to check for stations which have missing data and then use Maddavo's "edit" option to fill out the data.

Example:

Code:

$ trade.py local Ross209 --ly 20 --stations -vv

This will show you stations with missing StnLS etc data, then you can open the "Galaxy Map" and use the "Systems" view to find the distance and pad size (we're assuming that outposts are medium and ring/coriolis are large). Go to mad's site, click "Edit Stations" and enter the offending system, update the details you have, and you're done.

You can sometimes narrow it down by looking for stations which have a '?' pad:

Code:

$ trade.py local Ross209 --ly 20 --stations -vv --pad ?

The "--stations" option here is telling it to only list systems for which it found stations, so empty systems and systems with populated station data don't get listed, leaving you to see stuff that matters.

maddavo · Apr 7, 2015

bwcawman said:
Love your plugin for TradeDangerous. Just one question. It appears that if I want the station/system data I have to also get the price data. Is that true or am I missing something? I like the feel of discovering my own trade information but don't feel the need to have to also add every station in the game as well.

Hi @bwcawman, I tried PM to you but you're at your quota. I'm guessing that you might subscribe to this thread.

I didn't write the plugin, @kfsone wrote it. It is way beyond my Python abilities. To get help on the plugin, use the following command:

Code:

trade import --plug maddavo --opt help

Looking at the help I think there is an option that does what you want. Try:

Code:

trade import --plug maddavo --opt systems,stations,csvonly

Cheers,
Maddavo

kfsone · Apr 7, 2015

maddavo said:
Hi @bwcawman, I tried PM to you but you're at your quota. I'm guessing that you might subscribe to this thread.

I didn't write the plugin, @kfsone wrote it. It is way beyond my Python abilities. To get help on the plugin, use the following command:

Code:

trade import --plug maddavo --opt help

Looking at the help I think there is an option that does what you want. Try:

Code:

trade import --plug maddavo --opt systems,stations,csvonly

Cheers,
Maddavo

Shortcut for --opt is -O

Code:

trade.py import --help
./trade.py: TradeDangerous help

usage: trade.py import [--plug PLUG] [--url URL] [--download]
                       [--ignore-unknown] [--option PLUGINOPTIONS] [--reset]
                       [-h] [--debug] [--detail] [--quiet] [--db DBFILENAME]
                       [--cwd CWD] [--link-ly MAXSYSTEMLINKLY]
                       [filename]

Optional Switches:
  filename              Name of the file to read.
  --plug PLUG, -P PLUG  Use the specified import plugin.
  --url URL             Name of the file to read.
  --download            Stop after downloading.
  --ignore-unknown, -i  Data for systems, stations and items that are not
                        recognized is reported as warning but skipped.
  --option PLUGINOPTIONS, -O PLUGINOPTIONS
                        Provides a way to pass additional arguments to
                        plugins.
  --reset               Clear the database before importing.

Common Switches:
  -h, --help            Show this help message and exit.
  --debug, -w           Enable/raise level of diagnostic output.
  --detail, -v          Increase level of detail in output.
  --quiet, -q           Reduce level of detail in output.
  --db DBFILENAME       Specify location of the SQLite database.
  --cwd CWD, -C CWD     Change the working directory file accesses are made
                        from.
  --link-ly MAXSYSTEMLINKLY, -L MAXSYSTEMLINKLY
                        Maximum lightyears between systems to be considered
                        linked.

This sub-command provides a plugin infrastructure, and comes with a module to
import data from Maddavo's Market Share (http://www.davek.com.au/td/). See
"import --plug=maddavo --opt=help" for more help.

It says, right there at the end how to get help. With the current version you can short-cut --plug as '-P' and --opt as '-O' so:

Code:

$ trade.py import -P maddavo -O help
Plugin that downloads data from maddavo's site.

Options supported by this plugin:
--opt=corrections   Merge Corrections data into local db.
--opt=csvonly       Stop after csv work, don't import prices
--opt=csvs          Merge Corrections, System, Station and ShipVendor data into the local db.
--opt=exportcsv     Regenerate System and Station .csv files after merging System/Station data.
--opt=force         Process prices even if timestamps suggest there is no new data.
--opt=shipvendors   Merge ShipVendor data into local db.
--opt=skipdl        Skip doing any downloads.
--opt=stations      Merge Station data into local db.
--opt=systems       Merge System data into local db.
--opt=use2d         Force download of the 2-days .prices file.
--opt=use3h         Force download of the 3-hours .prices file.
--opt=usefull       Force download of the full .prices file.

You can also chain options together, e.g.:
  --opt=systems,stations,csvonly

So what you're looking for is

Code:

$ trade.py import -P maddavo -O csvs,csvonly

That'll pull all of the CSVs and none of the price data.

... BTW ... Mad - good to see you back mate - glad the spiders didn't get you

If you get chance, take a look at my last-but-one comment on the incremental import issue (https://bitbucket.org/kfsone/tradedangerous/issue/83/support-incremental-imports).

DRY411S · Apr 7, 2015

Two Questions about systems

Two questions.

1. I've just downloaded the latest version and then done an immediate import --plug=maddavo --opt=systems,stations,shipvendors

I'm getting several errors about the same system. Example:

Code:

NOTE: Unrecognized system for station MV URSAE MAJORIS/Sagan Port

So somehow there is data about a station in a system that does not exist in system.csv.

2. Since 27th March, I have discovered 8 systems not in system.csv. I've got the co-ordinates validated at EDSTarCoordinator an added them by hand to my system.csv. What do I need to do to make sure they persist in maddavo's copy so they don't get lost when I download? For the record, here are my hand made entries:

Code:

'ICZ GW-W C1-20',23.3125, -124.78125, 42.15625,'Release 1.00-EDStar','2015-04-06 15:40:32'
'COL 285 SECTOR AX-L A37-0',36.9375, -43.75, 47.53125,'Release 1.00-EDStar','2015-04-05 16:30:24'
'ICZ KX-T B3-3',45.125, -108.09375, 49.09375,'Release 1.00-EDStar','2015-04-01 23:15:32'
'ICZ KC-V C2-22',31.9375, -112.90625, 55.5625,'Release 1.00-EDStar','2015-04-01 20:30:32'
'ICZ JX-T B3-5',30.78125, -113.71875, 49.5,'Release 1.00-EDStar','2015-04-01 20:21:32'
'ICZ DB-X C1-22',12.0625, -103.96875, 41.28125,'Release 1.00-EDStar','2015-03-27 08:48:34'
'ICZ CW-V B2-6',12.3125, -99.34375, 31.0625,'Release 1.00-EDStar','2015-03-27 08:43:34'
'ICZ DW-V B2-2',17.90625, -99, 18.65625,'Release 1.00-EDStar','2015-03-27 08:25:34'

kfsone · Apr 7, 2015

DRY411S said:
2. Since 27th March, I have discovered 8 systems not in system.csv. I've got the co-ordinates validated at EDSTarCoordinator an added them by hand to my system.csv. What do I need to do to make sure they persist in maddavo's copy so they don't get lost when I download? For the record, here are my hand made entries:

Code:

'ICZ GW-W C1-20',23.3125, -124.78125, 42.15625,'Release 1.00-EDStar','2015-04-06 15:40:32' 'COL 285 SECTOR AX-L A37-0',36.9375, -43.75, 47.53125,'Release 1.00-EDStar','2015-04-05 16:30:24' 'ICZ KX-T B3-3',45.125, -108.09375, 49.09375,'Release 1.00-EDStar','2015-04-01 23:15:32' 'ICZ KC-V C2-22',31.9375, -112.90625, 55.5625,'Release 1.00-EDStar','2015-04-01 20:30:32' 'ICZ JX-T B3-5',30.78125, -113.71875, 49.5,'Release 1.00-EDStar','2015-04-01 20:21:32' 'ICZ DB-X C1-22',12.0625, -103.96875, 41.28125,'Release 1.00-EDStar','2015-03-27 08:48:34' 'ICZ CW-V B2-6',12.3125, -99.34375, 31.0625,'Release 1.00-EDStar','2015-03-27 08:43:34' 'ICZ DW-V B2-2',17.90625, -99, 18.65625,'Release 1.00-EDStar','2015-03-27 08:25:34'

[/QUOTE]

Don't download, use the plugin. The plugin performs a merge.

Code:

trade.py import -P maddavo -O csvs
trade.py import -P maddavo -O csvs,csvonly
trade.py import -P maddavo -O help

DRY411S · Apr 7, 2015

Don't download, use the plugin. The plugin performs a merge.

Code:

trade.py import -P maddavo -O csvs
trade.py import -P maddavo -O csvs,csvonly
trade.py import -P maddavo -O help

[/QUOTE]

Code:

trade.py import -P maddavo -O help

trade.py: ERROR: unrecognized arguments: -P


usage: trade.py {import} ...

I think I know what I've done here. I used the system.csv that you include with the distribution, then 'downloaded' (wrong word sorry, I used the import command) madddavo's. Does that mean that the systems are still in the local db?

hitchhiker54 · Apr 7, 2015

Yeah they should still be there

If you like you can use a sqlite browser like this one for windows (I use this myself works fine on win 7 + win 10 tech preview) http://sourceforge.net/projects/sqlitebrowser/ and check the system names in your local db.

Tromador · Apr 7, 2015

This is a silly error:

D:\Gadgets\TradeDangerous>trade.py buy --near cemiess nonlethal
D:\Gadgets\TradeDangerous\trade.py: Item "nonlethal" could match nonlethalweapons or Non-Lethal Weapons

Kerodan · Apr 8, 2015

Hello everyone,

I have a short question to which I could not really find an answer...
Is there a way to convert *.csv trading data (exported by EliteOCR) to *.prices?
Would love to transfer all my collected data over to the .prices format but have no idea how to do that without typing all that stuff manually.

All the best and keep up the great work!

gazelle · Apr 8, 2015

Kerodan said:
Is there a way to convert *.csv trading data (exported by EliteOCR) to *.prices?

If you know phyton a little bit it's quite easy to use the TD_Export.py source to make a converter.

Kerodan · Apr 8, 2015

gazelle said:
If you know phyton a little bit it's quite easy to use the TD_Export.py source to make a converter.

I actually considered that and looked into it but my knowledge of python is close to non-existant, wherefore, I did not really achieve much by looking at your nice plugin...

But thanks for your reply! Now I know that I did not miss something.

gazelle · Apr 8, 2015

Add this to the end:

Code:

if __name__ == "__main__":
    import sys
    import csv

    tdNew = TD_Export(parent=None, path=None, debug=True)
    with open(sys.argv[1]) as csvFile:
        cvsReader = csv.reader(csvFile, delimiter=';')
        tdNew.run(cvsReader)

If you're using python 3 you also have to change line 103 to:

Code:

self.timeStamp = datetime.datetime.strptime(timeStamp, "%Y-%m-%dT%H:%M:%S+00:00").isoformat(" ")

Now you can just:

Code:

$ python TD_Export.py "Acharya Station.2015-04-08T19.26.19.csv"
system changed to !Unknown!
new system: !Unknown!
station changed to Acharya Station
new station: Acharya Station

Be aware that the "import.prices" file will be created in the current directory and will be overwritten by each call of the script. You can just generate one big csv file first or copy/rename the prices file before each call.

Kerodan · Apr 8, 2015

Oh my god, thank you so much!
I will give it a try as soon as possible.

And again because once was not enough: Thank you

Coyote21 · Apr 8, 2015

Not sure when this began, but when I import EliteOCR .prices files the Age/days field no longer updates.

Example for BD+72 545/Varthema Enterprise

Code:

trade.py local bd+72545 --ly 0 -vv
System       Dist
  /  Station................ StnLs Age/days Mkt BMk Shp Out Arm Ref Rep Pad Itms
--------------------------------------------------------------------------------
BD+72 545    0.00
  /  Gardner Platform....... 2.52K    88.37 Yes  No  No Yes Yes Yes Yes Med   73
  /  Rangarajan Enterprise.. 1.02K     0.88 Yes  No Yes Yes Yes Yes Yes Lrg   73
  /  Varthema Enterprise.... 1.41K    88.38 Yes Yes  No Yes Yes Yes  No Med   73

trade.py import path_obscured\import.prices
NOTE: Import complete: 69 updated items (69 buy, 15 sell) for 1 stations in 1 systems

trade.py local bd+72545 --ly 0 -vv
System       Dist
  /  Station................ StnLs Age/days Mkt BMk Shp Out Arm Ref Rep Pad Itms
--------------------------------------------------------------------------------
BD+72 545    0.00
  /  Gardner Platform....... 2.52K    88.38 Yes  No  No Yes Yes Yes Yes Med   73
  /  Rangarajan Enterprise.. 1.02K     0.89 Yes  No Yes Yes Yes Yes Yes Lrg   73
  /  Varthema Enterprise.... 1.41K    88.39 Yes Yes  No Yes Yes Yes  No Med   73

And before any one asks, the import.prices file is definitely for Varthema Enterprise.

Code:

#! trade.py import -

# TradeDangerous prices created by EliteOCR from seeebek
# see http://forums.frontier.co.uk/showthread.php?t=68771

# File syntax:
# <item name> <sell> <buy> [<demand> <stock> [<timestamp>]]
#   Use '?' for demand/stock when you don't know/care,
#   Use '-' for demand/stock to indicate unavailable,
#   Otherwise use a number followed by L, M or H, e.g.
#     1L, 23M or 30000H

#     Item Name                SellCr   BuyCr     Demand     Stock  Timestamp


@ BD+72 545/Varthema Enterprise
   + Chemicals
      Explosives                  206     221          ?    19175M  2015-04-08 20:58:44
      Hydrogen Fuel               106     111          ?    70210M  2015-04-08 20:58:44
      Mineral Oil                 318       0    433958H         -  2015-04-08 20:58:44
   + Consumer Items
      Clothing                    438       0     12220M         -  2015-04-08 20:58:44
      Consumer Technology        7495       0      5072H         -  2015-04-08 20:58:44
      Domestic Appliances         527       0       988L         -  2015-04-08 20:58:44
   + Foods
      Algae                       258       0     82179H         -  2015-04-08 20:58:44
      Animal Meat                1505       0      7122M         -  2015-04-08 20:58:44
      Coffee                     1286       0       695L         -  2015-04-08 20:58:44
      Fish                        577       0     27224H         -  2015-04-08 20:58:44
      Food Cartridges              34      44          ?    42105H  2015-04-08 20:58:44
      Fruit And Vegetables        315       0      2507L         -  2015-04-08 20:58:44
      Grain                       224       0     22209L         -  2015-04-08 20:58:44
      Synthetic Meat              396       0      6477H         -  2015-04-08 20:58:44
      Tea                        1458       0      2231L         -  2015-04-08 20:58:44
   + Industrial Materials
      Polymers                    250       0    543781M         -  2015-04-08 20:58:54
      Semiconductors             1154       0    373632H         -  2015-04-08 20:58:54
      Superconductors            6548    6627          ?    24803M  2015-04-08 20:58:54
   + Legal Drugs
      Beer                        295       0     37908H         -  2015-04-08 20:58:54
      Liquor                      848       0       845H         -  2015-04-08 20:58:54
      Tobacco                    5361       0      3360M         -  2015-04-08 20:58:54
      Wine                        344       0     33576M         -  2015-04-08 20:58:54
   + Machinery
      Atmospheric Processors      304     318          ?    14335M  2015-04-08 20:58:54
      Crop Harvesters            1994    2020          ?    10157M  2015-04-08 20:58:54
      Marine Equipment           3939    3988          ?     6190M  2015-04-08 20:58:54
      Microbial Furnaces          333       0    429027H         -  2015-04-08 20:58:54
      Mineral Extractors          407     425          ?    53556H  2015-04-08 20:58:54
      Power Generators            647       0     23285M         -  2015-04-08 20:58:54
      Water Purifiers             168     181          ?   516393H  2015-04-08 20:58:54
   + Medicines
      Basic Medicines             221     237          ?    18787M  2015-04-08 20:58:54
      Performance Enhancers      7495       0      6845H         -  2015-04-08 20:59:04
      Progenitor Cells           7495       0      2668H         -  2015-04-08 20:59:04
   + Metals
      Aluminium                   390       0    439801M         -  2015-04-08 20:59:04
      Beryllium                  8451       0     32374M         -  2015-04-08 20:59:04
      Cobalt                      847       0     44400M         -  2015-04-08 20:59:04
      Copper                      554       0    326543M         -  2015-04-08 20:59:04
      Gallium                    5355       0     46449M         -  2015-04-08 20:59:04
      Gold                      10232       0     58680H         -  2015-04-08 20:59:04
      Indium                     5808    5878          ?    30374M  2015-04-08 20:59:04
      Lithium                    1715       0    117108M         -  2015-04-08 20:59:04
      Palladium                 14229       0     45621H         -  2015-04-08 20:59:04
      Platinum                  19677       0      3401H         -  2015-04-08 20:59:04
      Silver                     5426       0     98285H         -  2015-04-08 20:59:04
      Tantalum                   3830    3878          ?    41271M  2015-04-08 20:59:04
      Titanium                   1114       0    157536M         -  2015-04-08 20:59:04
      Uranium                    2822       0     77696M         -  2015-04-08 20:59:04
   + Minerals
      Bauxite                     277       0    672457H         -  2015-04-08 20:59:18
      Bertrandite                2781       0     61666M         -  2015-04-08 20:59:18
      Coltan                     1700       0    120840H         -  2015-04-08 20:59:18
      Gallite                    2033       0     51794M         -  2015-04-08 20:59:18
      Indite                     2358       0      8182M         -  2015-04-08 20:59:18
      Lepidolite                  589       0      9874L         -  2015-04-08 20:59:18
      Painite                   35973       0     21162H         -  2015-04-08 20:59:18
      Rutile                      467       0    204026M         -  2015-04-08 20:59:18
      Uraninite                  1016       0     16585M         -  2015-04-08 20:59:18
   + Technology
      Advanced Catalysers        3329       0     73134H         -  2015-04-08 20:59:18
      Auto-Fabricators           4258       0     12186H         -  2015-04-08 20:59:18
      Computer Components         353     370          ?     1984H  2015-04-08 20:59:18
      H.E. Suits                  424       0    201027H         -  2015-04-08 20:59:18
      Resonating Separators      6603       0     41592H         -  2015-04-08 20:59:18
      Robotics                   2184       0     30551H         -  2015-04-08 20:59:18
   + Textiles
      Leather                     301       0   1048938H         -  2015-04-08 20:59:18
      Natural Fabrics             571       0    145953H         -  2015-04-08 20:59:33
      Synthetic Fabrics           289       0    385307M         -  2015-04-08 20:59:33
   + Waste
      Biowaste                     15      20          ?     3464M  2015-04-08 20:59:33
      Chemical Waste              107       0     12605H         -  2015-04-08 20:59:33
      Scrap                        31      36          ?    50583M  2015-04-08 20:59:33
   + Weapons
      Non-Lethal Weapons         2080       0       239H         -  2015-04-08 20:59:33
      Reactive Armour            2460       0       457H         -  2015-04-08 20:59:33

Strange thing is that using the new (and very cool) market command I can see that the data is properly updated just not the modified date.

Any ideas anyone? Should I post this on Git Hub as a bug?