In-Development TradeDangerous: power-user trade optimizer

maddavo · Jan 25, 2015

Hi all,

Dealing with a few server issues today. Sorry it's taking me so long but also had visitors for Sunday BBQ.

I've think I've introduced a bug somewhere that is allowing multiple stations. 39 TAURI/Porta had a zillion entries.

Also, there was a problem with the file generation and the file pages - this is now fixed, but the prices files have borked data until I get rid of the 39 TAURI/Porta problem.

I'll get back here when fixed.

Maddavo

SnotFunk · Jan 25, 2015

Hmm I downloaded and setup the tool today but I am getting this error:

C:\trade>trade.py run --fr=ekuru/dana --credits 20000 --capacity 6 --ly-per 13.0
5
C:\trade\trade.py: Error: There is no trading data for ANY station in the local
database. Please enter or import price data.

This could be due to a lack of price or station data. You
may want to consult the "local -vv" sub-command to see if
there are stations in the area with price data.

I have ran this command but no change:

trade.py import --plug=maddavo --option=stncsv --option=syscsv

Can anyone help or is the update above from maddavo the cause of this?

Fihnakis · Jan 25, 2015

Im guess the issue above. Here's what it returned for me. Haven't done an update yet so should still be clean prices.

Code:

D:\Games\Frontier\trade>trade.py run --fr=ekuru/dana --credits 20000 --capacity 6 --ly-per 13.0
EKURU/Dana Platform -> LTT 15449/Reilly Dock
  EKURU/Dana Platform: 5 x Tantalum, 1 x Cobalt,
  GD 219/McKee Ring: 5 x Land Enrichment Systems, 1 x Non-Lethal Weapons,
  LTT 15449/Reilly Dock +9,836cr

If I can figure it out I'll PM you the prices and csv files if you'd like

SnotFunk · Jan 25, 2015

Hey many thanks for that!

I managed to fix with with a system reboot(originally had python issues) and a rebuild of the cache.. or maybe what ever issues maddavo had identified were fixed, either way its all working now!

But yeah totally awesome that you went to the trouble of doing that for me, it's majorly appreciated!

maddavo · Jan 25, 2015

The server for Maddavo's Market Share is back up and prices files are being generated correctly now (AFAIK). The backlog of uploaded data has pretty much been processed now.

5:45am now - off to bed.

bdragon9222 · Jan 25, 2015

maddavo said:
The server for Maddavo's Market Share is back up and prices files are being generated correctly now (AFAIK). The backlog of uploaded data has pretty much been processed now.

5:45am now - off to bed.

Thx for your hard work!

Also thanks to kfsone for all of his efforts, that are making our gameplay so much more fun and easier!

Snake Man · Jan 26, 2015

I believe the two last releases require some "Tk" module and will not work in my linux setup. I am not complaining, just pointing it out. Will stick with f948b9be5529 release which runs.

kfsone · Jan 26, 2015

Snake Man said:
I believe the two last releases require some "Tk" module and will not work in my linux setup. I am not complaining, just pointing it out. Will stick with f948b9be5529 release which runs.

The "Tk" package is "tkinter" - it's the "batteries included" gui that comes with Python. If it's not working on your system it's because you've mixed 32- and 64-bit pythons at some point and you have the wrong bitness of tk package installed. Google "python 3.4 tk fails" and the error you're getting and you'll find there are several solutions on how to fix it.

You also might want to try "pip list" and then "pip install --upgrade" each of the packages listed.

Tk is a total pain in the ass to work with, the only reason I'm using it is that it works on all the major platforms, and I've tested it on Win, Mac, Mint (Debian/Ubuntu based) and CentOS.

kfsone · Jan 26, 2015

Code:

v6.8.1 Jan 25 2015
. (kfsone) Additional rules to prevent data pollution (ocr derp protection),
. (kfsone) Support for maddavo's "3 hour" prices file,
. (kfsone) Changed how downloads work to massively improve download speeds,
. (kfsone) Improvements to the EDSC tools,
. (kfsone) "First time" banner explaining the maddavo import plugin,
. (kfsone) Fixed "shipvendor" command doing a poor job of finding stations,
. (kfsone) "submit-distances" will now prompt you to correct distances that
        EDSC reports as needing confirmation,
. (kfsone) Fixed a crash when data was > 1 month old,
. (kfsone) (API) Added a helper for clipboarding data,
. (kfsone) (API) Added misc.edsc.StarSubmissionResult,
. (kfsone) (API) Cleaned up various pieces of code documentation,
+ Data: Systems, Stations, Ships, lots of corrections/deletions too,

Tromador · Jan 26, 2015

You've a tiny typo @ line 136 in maddavo_plug.py

And I learned quite a lot about bitbucket trying to fix it for you...

Tromador · Jan 26, 2015

Currently there is a problem downloading the 3h price list - I think it's at Maddavo's end and have sent him a message - but FYI

Edit: It's definitely his end.

Tromador · Jan 26, 2015

Ignoring stations for the moment...

Code:

D:\Gadgets\TradeDangerous>trade.py import --plug=maddavo
NOTE: Stale/missing local copy, downloading full .prices file.
Connecting to server: http://www.davek.com.au/td/prices.asp
import.prices: 14,711,973/14,711,973 bytes |  79.93KB/s | 100.00%
NOTE: Rebuilding cache file: this may take a moment.
NOTE: Missing "D:\Gadgets\TradeDangerous\data\TradeDangerous.prices" file - no price data.
NOTE: No new price entries found.

This is with 6.8.1

Update:

And with 6.8.0 so something must be badly broken with maddavo.

Though I'm not sure why TD doesn't seem to like this line from Maddavo

Code:

      Explosives                  399       0    75124M         -  2014-12-26 01:00

but is perfectly happy with this line from EliteOCR

Code:

      Hydrogen Fuel               107     111          ?    10900M  2015-01-26 12:43:27

But something isn't right... though looking at them together like that. Timestamp (00:00 vs 00:00:00) ??

Tromador · Jan 26, 2015

Tested with

/

[0-9][0-9])$/\1:01/

On Maddavo's prices file... And by jingo it works.

So - if I manually craft a mixture of the supplied stations, my stations and Maddavo's stations (sanitised for derps), then fix up the pricing data so it imports... I am in a position to gather more data.

I wouldn't mind, but I'm doing missions and couldn't care less about the trade data atm, but feel bad about not expanding the knowledge base.

Snake Man · Jan 26, 2015

A GUI, interesting. The whole reason I like TD besides that it produces good trade results is that I run it on console on my linux shell (old school, no monitor, no nothing, only ssh connection). If I wanted a graphical user interface I might as well install python to windows then.

Have you reconsidered coding TD with C++ to make simple .exe with GUI that all users (all elite users, ie windows user) can use?

Tromador · Jan 26, 2015

Snake Man said:
A GUI, interesting. The whole reason I like TD besides that it produces good trade results is that I run it on console on my linux shell (old school, no monitor, no nothing, only ssh connection). If I wanted a graphical user interface I might as well install python to windows then.

Have you reconsidered coding TD with C++ to make simple .exe with GUI that all users (all elite users, ie windows user) can use?

Old school is a Wyse terminal over serial, X25 PAD if you are lucky and you are lucky if you can even telnet off localhost. SSH is pretty new. (in my eyes

)

tkinter is just a python interface to tk library, which in turn is an extension of tcl. If you are on a Linux box and you have mishmashed bitness in these libraries, you could have some other underlying issues. Other commands (e.g. expect) rely on tcl and you need that whole toolchain to be correct. You might want to ensure that all of those libraries and your python match the bitness of the system so you know it's clean.

Chem · Jan 26, 2015

Another plugin to this excellent program

TradeDangerousHi, Commanders.

Can I propose for the use my own plugin for this excellent program?
It can help you to find two point circle routes from the gathered by this program data.
Execution params:

Code:

python trade.py import --plug maxprofit --opt=help


Plugin that find max profitable two point cyrcle routes depended on different params.     Author Chem @ Elite-Games.Ru.


Options supported by this plugin:
--opt=1d            Process data one day old only . Default.
--opt=2d            Process data two day old only.
--opt=3d            Process data two day old only.
--opt=5d            Process data two day old only.
--opt=any           Analyze any available commodities.
--opt=mpsl          Process data from stations, which has Large pad .
--opt=mpsm          Process data from stations, which has at least Medium pad . Default.
--opt=pgbctpepc     Analyze only Pall.  Gold Beryll.  vs C.Tech.  P.Enhan.  P.Cells.  Default.
--opt=sd100th       Process data with stock/demand above 100 thousands.
--opt=sd10th        Process data with stock/demand above 10 thousands . Default.
--opt=sd1th         Process data with stock/demand above 1 thousand.
--opt=sd50th        Process data with stock/demand above 50 thousands.
--opt=stdst100      Process data from stations with distance to star <= 100 ls.
--opt=stdst1000     Process data from stations with distance to star <= 1000 ls . Default.
--opt=stdst10000    Process data from stations with distance to star <= 10000 ls.
--opt=stdst20       Process data from stations with distance to star <= 20 ls.
--opt=stdst50       Process data from stations with distance to star <= 50 ls.
--opt=stdst500      Process data from stations with distance to star <= 500 ls.
--opt=sysdst10      Process data from stations with dist betw systems <= 10 ly.
--opt=sysdst20      Process data from stations with dist betw systems <= 20 ly.
--opt=sysdst30      Process data from stations with dist betw systems <= 30 ly . Default.
--opt=sysdst40      Process data from stations with dist betw systems <= 40 ly.
--opt=sysdst50      Process data from stations with dist betw systems <= 50 ly.

Result - csv file, which can be analyzed in Excel.
Example:

Code:

src_system_name;src_station_name;src_station_dist;src_station_max_pad_size;dst_system_name;dst_station_name;dst_station_dist;dst_station_max_pad_size;item1_name;item1_buy;item1_sell;item2_name;item2_buy;item2_sell;item1_stock_units;item1_stock_level;item1_demand_units;item1_demand_level;item2_stock_units;item2_stock_level;item2_demand_units;item2_demand_level;item1_profit;item2_profit;total_profit;src_age;dst_age;syst_dist
PARUTIS;Teng-Hui Hub;505;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Consumer Technology;6393;7510;747705;-1;239487;-1;38889;-1;46881;-1;1117;1239;2356;1 day, 2:15:23;1 day, 18:42:43;32.295298854864306
GLIESE 868;Braun Station;78;L;TSIM BINBA;Kopal Orbital;10;L;Gold;8814;9870;Consumer Technology;6251;7518;131174;-1;113946;-1;27973;-1;65225;-1;1267;1056;2323;1 day, 3:41:00;1 day, 2:36:25;49.135306250317605
PARUTIS;Teng-Hui Hub;505;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Progenitor Cells;6370;7451;747705;-1;239487;-1;76078;-1;24936;-1;1081;1239;2320;1 day, 2:15:23;1 day, 18:42:37;32.295298854864306
SHAPSUGABUS;Balog Dock;30;L;SEGOVEDUWA;Mason Hub;332;L;Gold;8881;10037;Progenitor Cells;6369;7527;72717;-1;121484;-1;43063;-1;18775;-1;1158;1156;2314;1 day, 4:37:27;1 day, 7:27:26;34.84991537382121
SIFJAR;Herschel Landing;41.4;L;WURU;Schuster Hub;24;L;Gold;9235;10154;Consumer Technology;6157;7540;1777;-1;186590;-1;71138;-1;1384;-1;1383;919;2302;1 day, 22:06:39;22:34:16;47.87706017107776
PARUTIS;Evans Port;299;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Progenitor Cells;6370;7430;747705;-1;239487;-1;76078;-1;24421;-1;1060;1239;2299;1 day, 18:06:51;1 day, 18:42:37;32.295298854864306
PARUTIS;Evans Port;299;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Consumer Technology;6393;7448;747705;-1;239487;-1;38889;-1;44901;-1;1055;1239;2294;1 day, 18:06:51;1 day, 18:42:43;32.295298854864306
SHAPSUGABUS;Balog Dock;30;L;SEGOVEDUWA;Mason Hub;332;L;Gold;8881;10037;Consumer Technology;6409;7527;72717;-1;121484;-1;22332;-1;42871;-1;1118;1156;2274;1 day, 4:37:27;1 day, 7:27:30;34.84991537382121
GLIESE 868;Braun Station;78;L;LTT 377;Zoline Port;645;L;Gold;8814;9983;Consumer Technology;6424;7518;131174;-1;114223;-1;22245;-1;65225;-1;1094;1169;2263;1 day, 3:41:00;1 day, 14:52:00;30.755001498740004

And plugin itself (Unpack it into plugins directory of the TradeDangerous) is View attachment 10845

Fihnakis · Jan 27, 2015

Would be great if I knew how to read all of that

kfsone · Jan 27, 2015

Chem said:

TradeDangerousHi, Commanders.

Can I propose for the use my own plugin for this excellent program?
It can help you to find two point circle routes from the gathered by this program data.
Execution params:

Code:

python trade.py import --plug maxprofit --opt=help


Plugin that find max profitable two point cyrcle routes depended on different params.     Author Chem @ Elite-Games.Ru.


Options supported by this plugin:
--opt=1d            Process data one day old only . Default.
--opt=2d            Process data two day old only.
--opt=3d            Process data two day old only.
--opt=5d            Process data two day old only.
--opt=any           Analyze any available commodities.
--opt=mpsl          Process data from stations, which has Large pad .
--opt=mpsm          Process data from stations, which has at least Medium pad . Default.
--opt=pgbctpepc     Analyze only Pall.  Gold Beryll.  vs C.Tech.  P.Enhan.  P.Cells.  Default.
--opt=sd100th       Process data with stock/demand above 100 thousands.
--opt=sd10th        Process data with stock/demand above 10 thousands . Default.
--opt=sd1th         Process data with stock/demand above 1 thousand.
--opt=sd50th        Process data with stock/demand above 50 thousands.
--opt=stdst100      Process data from stations with distance to star <= 100 ls.
--opt=stdst1000     Process data from stations with distance to star <= 1000 ls . Default.
--opt=stdst10000    Process data from stations with distance to star <= 10000 ls.
--opt=stdst20       Process data from stations with distance to star <= 20 ls.
--opt=stdst50       Process data from stations with distance to star <= 50 ls.
--opt=stdst500      Process data from stations with distance to star <= 500 ls.
--opt=sysdst10      Process data from stations with dist betw systems <= 10 ly.
--opt=sysdst20      Process data from stations with dist betw systems <= 20 ly.
--opt=sysdst30      Process data from stations with dist betw systems <= 30 ly . Default.
--opt=sysdst40      Process data from stations with dist betw systems <= 40 ly.
--opt=sysdst50      Process data from stations with dist betw systems <= 50 ly.

Result - csv file, which can be analyzed in Excel.
Example:

Code:

src_system_name;src_station_name;src_station_dist;src_station_max_pad_size;dst_system_name;dst_station_name;dst_station_dist;dst_station_max_pad_size;item1_name;item1_buy;item1_sell;item2_name;item2_buy;item2_sell;item1_stock_units;item1_stock_level;item1_demand_units;item1_demand_level;item2_stock_units;item2_stock_level;item2_demand_units;item2_demand_level;item1_profit;item2_profit;total_profit;src_age;dst_age;syst_dist
PARUTIS;Teng-Hui Hub;505;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Consumer Technology;6393;7510;747705;-1;239487;-1;38889;-1;46881;-1;1117;1239;2356;1 day, 2:15:23;1 day, 18:42:43;32.295298854864306
GLIESE 868;Braun Station;78;L;TSIM BINBA;Kopal Orbital;10;L;Gold;8814;9870;Consumer Technology;6251;7518;131174;-1;113946;-1;27973;-1;65225;-1;1267;1056;2323;1 day, 3:41:00;1 day, 2:36:25;49.135306250317605
PARUTIS;Teng-Hui Hub;505;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Progenitor Cells;6370;7451;747705;-1;239487;-1;76078;-1;24936;-1;1081;1239;2320;1 day, 2:15:23;1 day, 18:42:37;32.295298854864306
SHAPSUGABUS;Balog Dock;30;L;SEGOVEDUWA;Mason Hub;332;L;Gold;8881;10037;Progenitor Cells;6369;7527;72717;-1;121484;-1;43063;-1;18775;-1;1158;1156;2314;1 day, 4:37:27;1 day, 7:27:26;34.84991537382121
SIFJAR;Herschel Landing;41.4;L;WURU;Schuster Hub;24;L;Gold;9235;10154;Consumer Technology;6157;7540;1777;-1;186590;-1;71138;-1;1384;-1;1383;919;2302;1 day, 22:06:39;22:34:16;47.87706017107776
PARUTIS;Evans Port;299;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Progenitor Cells;6370;7430;747705;-1;239487;-1;76078;-1;24421;-1;1060;1239;2299;1 day, 18:06:51;1 day, 18:42:37;32.295298854864306
PARUTIS;Evans Port;299;L;HILL PA HSI;Curie Gateway;23;L;Gold;8921;10160;Consumer Technology;6393;7448;747705;-1;239487;-1;38889;-1;44901;-1;1055;1239;2294;1 day, 18:06:51;1 day, 18:42:43;32.295298854864306
SHAPSUGABUS;Balog Dock;30;L;SEGOVEDUWA;Mason Hub;332;L;Gold;8881;10037;Consumer Technology;6409;7527;72717;-1;121484;-1;22332;-1;42871;-1;1118;1156;2274;1 day, 4:37:27;1 day, 7:27:30;34.84991537382121
GLIESE 868;Braun Station;78;L;LTT 377;Zoline Port;645;L;Gold;8814;9983;Consumer Technology;6424;7518;131174;-1;114223;-1;22245;-1;65225;-1;1094;1169;2263;1 day, 3:41:00;1 day, 14:52:00;30.755001498740004

And plugin itself (Unpack it into plugins directory of the TradeDangerous) is View attachment 10845

Awesome - I'll need to:

a/ Provide you with a better way to do this; you could make a command for it if you prefer and we can check it into the repos, the only problem is you lose some control that way,
b/ Support plugins other than "import"

,
c/ Figure out how to provide you exposure to other TD users via the documentation/etc,

- - - - - Additional Content Posted / Auto Merge - - - - -

Code:

v6.8.2 Jan 27 2015
. (kfsone) Issue #148 Don't blow up horribly when tkinter is missing,
. (kfsone) Issue #149 "nav" with --via was broken,
. (kfsone) Typo correction in the maddavo splash (thanks, Stefan),

kfsone · Jan 27, 2015

Snake Man said:
A GUI, interesting. The whole reason I like TD besides that it produces good trade results is that I run it on console on my linux shell (old school, no monitor, no nothing, only ssh connection). If I wanted a graphical user interface I might as well install python to windows then.

Have you reconsidered coding TD with C++ to make simple .exe with GUI that all users (all elite users, ie windows user) can use?

I've wrapped the tkinter use so that it shouldn't interfere with pure command line usage. Chalk up one more "seriously, python, wth?" for my book of black marks against Python.

I'd probably be more likely to write it in C# if I'm worrying about GUI because, well, Mono. As I've said before, I'm consdering rewriting the back-end in something more performant and threadable than Python and simply layering a GUI over it in Python or Go or something. I spent this evening learning Cython and rewriting my "prices to json" converter with it, and getting the performance from 27s to in pure python to convert mad's "all time" prices file to 780ms (that's 27,000 -> 780). But converting the bulk of tradedb.py and tradecalc.py stretches my definition of 'recreational coding'

Chem · Jan 27, 2015

Max profitable routes find command for TradeDangerous

kfsone said:
Awesome - I'll need to:

a/ Provide you with a better way to do this; you could make a command for it if you prefer and we can check it into the repos, the only problem is you lose some control that way,
b/ Support plugins other than "import" ,
c/ Figure out how to provide you exposure to other TD users via the documentation/etc,

Hi.

Thanks.
Option a) - fully acceptable.

I've modified it to work as command. But it require to modify __init__.py in commands directory.

Now I've redefined one object - class ProfitRoute(Route)
and emulate result of TradeCalc for displaying found routes...

By default - it's display it in format, similar to command run

Now, when you run it with, for example for my T7:

Code:

python trade.py profit  --capacity 212 --ly-per 16 --age 5 --pad-size L --ls-max 100 --ly 50 --out-limit 3 --distance-sort

Result will looks like:

Code:

Circle: JUNGA/Curtiss Station <-33.74 Ly-> ORCUS/Tombaugh Station
  JUNGA/Curtiss Station: 212 x Gold,
  ORCUS/Tombaugh Station: 212 x Performance Enhancers,
  JUNGA/Curtiss Station +460 040cr


Circle: GLIESE 868/Braun Station <-38.26 Ly-> V374 PEGASI/Mackellar Hub
  GLIESE 868/Braun Station: 212 x Gold,
  V374 PEGASI/Mackellar Hub: 212 x Consumer Technology,
  GLIESE 868/Braun Station +502 228cr


Circle: GLIESE 868/Braun Station <-38.26 Ly-> V374 PEGASI/Mackellar Hub
  GLIESE 868/Braun Station: 212 x Gold,
  V374 PEGASI/Mackellar Hub: 212 x Consumer Technology,
  GLIESE 868/Braun Station +502 228cr

With option -vvv it will look (only one result displayed):

Code:

Circle: JUNGA/Curtiss Station <-33.74 Ly-> ORCUS/Tombaugh Station (score: 33.743301)
Start CR:  1 872 384
Hops    :          2
Jumps   :          8
Gain CR :    460 040
Gain/Hop:    230 020
Final CR:  2 332 424


  Load from JUNGA/Curtiss Station (32ls/star, No/bm, Lrg/pad):
      212 x Gold                       8 832cr each,  1 872 384cr total,  data from 2 days vs 3 days
  Jump JUNGA -> HR 8031 -> SENGU -> NYAMINAWE -> ORCUS
  Unload at ORCUS/Tombaugh Station (12ls/star, No/bm, Lrg/pad) => Gain 216 664cr (1 022cr/ton) => 2 089 048cr
  Load from ORCUS/Tombaugh Station (12ls/star, No/bm, Lrg/pad):
      212 x Performance Enhancers      6 385cr each,  1 353 620cr total,  data from 3 days vs 2 days
  Jump ORCUS -> NYAMINAWE -> SENGU -> HR 8031 -> JUNGA
  Unload at JUNGA/Curtiss Station (32ls/star, No/bm, Lrg/pad) => Gain 243 376cr (1 148cr/ton) => 2 332 424cr
  ----------------------------------------------------------------------------
  Finish at JUNGA/Curtiss Station (32ls/star, No/bm, Lrg/pad) gaining 460 040cr => est 2 332 424cr total

Functionality to make csv still exists via --to-csv option.

Final version of command in attach:

For possibility to use it require to add one line in
commands\__init__.py
instead of

commands\__init__.py said:
import commands.update_cmd

commandIndex = {

should be

commands\__init__.py said:
...
import commands.update_cmd
import commands.profit_cmd

commandIndex = {
...

Fixed:
Replaced error, when nothing found by text message.