TradeDangerous: power-user trade optimizer

[kfsone]

TradeDangerous is a profit optimizer that runs locally on your computer, using a local database of price data you enter as you fly (or by importing data from other commanders). It is an Open Source project to which a number of commanders have now contributed (thanks esp to ShadowGar, Gazelle, Smacker65)

Back in 1984 I scribbled hundreds of pages of price notes. Naturally enough, when I started playing ED I pulled up notepad and started typing out interesting prices... It quickly became unwieldy and so I started knocking out this little python script to figure out the best trades.

By best trades I mean much more than just which item has the most markup from this station.

TradeDangerous is designed to do everything from simply calculating what's hot at Dahan to very efficiently solving the "NP" problem of "exactly how rich can I get in 10 stops in a type 6 with a 12 ly jump limit starting with 42 credits?"

For complicated routes, a "checklist" mode will walk you through each step (dock at Dahan, buy 100 fish, jump to X, jump to Y, dock at station, sell, refuel, buy ...). And for X52 users it will even display the steps on your MFD as you go.

More importantly, TradeDangerous is Open Source - I've written it so that other developers can make use of the extremely fast and efficient calculation engine to build tools.

==== CHANGES ====
(New changes occur frequently, I only update with significant releases here)
v7.3.0 Jul 05 2015:
. (ahamid/kfsone) "fromfile" support for command options:
 - All commands will now look for a "fromfile" specifying default arguments
   for that command .tdrc_{cmd} e.g .tdrc_nav or .tdrc_shipvendor,
 - One argument-position per line, "--ly 20" is two lines but "--ly=20" is one,
 - The fromfile argument is replaced with the arguments in the file,
 - Looks in the current directory and then your user directory,
 - You can also use your own fromfile as an argument prefixed with a '+':
     trade.py nav +myfrom.txt
. (Fred Deschenes) "buy" now supports categories e.g.
     trade.py buy --near sol metals,food
 - to find food cartridges, use foodc or "food cartridges",

Jun 20 2015:
. (kfsone) Lots of new systems (until EDSC broke),
. (kfsone) Minor optimizations,
. (kfsone) Tweaks to edscupdate,

v7.2.0 Jun 14 2015:
. (Orphu) Created an import plugin that uses Frontier's Mobile API,
    - See README.md for more help,
    - Use "trade.py import -P edapi" to get started,
    - Able to optionally upload prices it pulls to EDDN:
        trade.py imp -P edapi -O eddn
. (maddavo) Fixed/merged lots of new systems,

v7.1.2 Jun 6 2015:
. (Dry411s) Diamondback prices
. (Orphu) Imperial courier price
. (kfsone) Maddavo Data import/refresh

v7.1.1 May 25 2015
. (kfsone) Added "--distance" option to "edscupdate" which shows the 10
      closest candidates.
+ (kfsone) Over 500 new core systems

v7.1.0 May 11 2015
. (kfsone) #236 Unicorn (erh, Unicode)
    - Added code to trade.py to catch wild unicode errors, esp under windows:
      Windows console is NOT utf-8 based, which means that when TD tries to
      print a unicode character, an error occurs. I've added code to catch
      this early (TradeEnv.uprint) and code to catch it late (trade.py),
    - Changed various file openings to "utf-8",
. (gazelle) Changed display of "adding/removing ships" in shipvendor sub-command
. (kfsone) #234 Change display of "adding ship" in maddavo plugin
. (kfsone) Fix submit-distances/edscupdate generating duplicate extra-stars.

v7.0.3 May 09 2015
. (kfsone) "run" command:
    - Added "--show-jumps" (aka -J)
    - Jumps are no-longer shown by default,
    - Request #233 Jumps now include distance
    - If start and end station of a hop are in the same system,
      display "Supercruise to ..." instead of a jump
    - When a hop involves multiple jumps using --show-jumps, it will
      tell you the direct and total distances,
. (kfsone) Revamped the intro of the README.md (http://kfs.org/td/source)
. (gazelle) Corrected unicode system names (Argetlamh etc)

v7.0.2 May 05 2015
. (kfsone) Added "--demand" option to "run", filters based on demand.
      CAUTION: Items with "?" demand will be ignored when using --demand.
      If you want to enter demand values in the update tool, you will
      need to use the "--use-demand" (-D) option of "update".
. (kfsone) Fixed #231 Exception when using --quantity in buy/sell
. (kfsone) --supply is now an alias for --quantity in "buy"
. (kfsone) --demand is now an alias for --quantity in "sell"
. (kfsone) Fixed #232 'supply' values ignored by "run"

v7.0.1 May 04 2015
. (kfsone) Minor fixes from forums
. (kfsone) Fixed issue causing some users to get a numpy import error
+ Extra data and Rares

v7.0.0 May 03 2015
CACHE BUILD REQUIRED (trade.py buildcache -fi)
"requests" packages is now required if you want to use "maddavO"s plugin
. (Avi) README.txt has been replaced with README.md
. (kfsone) "import" command (non-plugin behavior)
    - Restored the default behavior of forcefully overwriting existing data
      with values from the .prices file and deleting items that aren't listed
      in the import.
    - Added "--merge-import" option: only imports entries that are newer than
      existing local data and only removes entries when there is an explicit
      entry for it with 0 prices.
    - Changed "--reset" to "--reset-all" because it's a scary command,
. (kfsone) Consistency of various commands:
    - "demand" refers to what a station will buy,
    - "supply" refers to what a station is selling,
    - Changed several command options from "--stock" to "--supply",
    - "--black-market" is now consistently spelled with a hyphen
       (or just --black for short),
    - "--bm" now has two hyphens in all uses (it was -bm in some cases),
. (kfsone) "market" command:
    - Only show the age of items once,
. (kfsone) "nav" command:
    - Fixed --refuel-jumps
    - Added --pad-size to limit which stations will be listed/refuelled at,
. (kfsone) "buy" and "sell" commands:
    - Added "--black-market" (--bm) option,
    - Added "--avoid" option (accepts station and/or system names)
. (kfsone) Performance:
    - Re-unified the StationBuying/StationSelling tables into StationItem,
    - Added StationBuying and StationSelling views,
    - Minor refactor of getTrades to reduce the pathalogical near O(n^2)
      behavior it used to match buyers to sellers,
    - Improved performance of nav command slightly,
    - "import" plugin now uses the "requests" module, which means that
      it will automatically download/use compressed .prices files when
      they are made available. E.g. maddavo's .prices files now require
      upto 5x less bandwidth,
    - Added code to allow experimental use of numpy,

See the "CHANGES.txt" file for more updates.

==== SUPPORT / LINKS ====

Source | Issue Tracker | Developer Facebook Group | Thread

==== DESCRIPTION ====

TD is a best profit calculation system that runs locally on your computer, designed to enable other programmers/power-users to easily build on it.

Running locally allows it can do much more work finding you that best trade. It can do much more than just tell you the highest value trade between two stations (which you might not even be able to afford!) It will find the load that makes the absolute best use of your cargo space and credits.

Better still - it can project that forward and calculate many hops for you, and then pick the best sequence: It may be most profitable to take 100 Gold from A->B, but if there's nothing worth buying at B, your next hop is a waste of time. And it takes the money you're going to make into account. When you have only 10,000 credits, it might be lucrative to Clothing A->B and Grain B->A, but if you've made 5,000 credits already, why aren't you picking up Fish B->C and then using the additional credits to allow you to start running Dom. Appliances C->A etc.

Because TD runs locally, it can figure out the full deal for you.

All of this is built in a modular way so that if you can easily pick up the TradeDB or TradeCalc modules and make your own front end to the data and to the calculation system.

OBTAINING:

TradeDangerous doesn't come with an installer - so it's not as easily accessible as it could be, yet. To get it running you'll need Python 3.4.1 installed.

Then, grab the latest code and data from BitBucket (https://bitbucket.org/kfsone/tradedangerous or for short: http://kfs.org/td/source).

Alternatively, you can use "git" to fetch and update it. Command line users can use "git clone git@bitbucket.org:kfsone/tradedangerous.git", or if you're not that adventurous, you could download Atlassian's "SourceTree" GUI and use that.

USAGE:

For more setup instructions, user guides, etc, please see our wiki: http://kfs.org/td/wiki


Enjoy.

-Oliver

[wolverine2710]

Sweet. A nice fresh and different approach to what is out there atm. Certainly very useful and it has functionality others atm lack. Also thanks for sharing the code. Not sure if you can also exclude a certain station from hops. That is something BPC users have asked slopey.

You lost me a bit with the access stuff. Does TD gets it data from a central database which one is connecting to or is it a local access db - which you provide and users can download a new from you side. Or that users somehow fill them self.

Something to consider. Andreas of marketdump is setting up EMDN (based upon EVE's EMDR) , see his thread. BPC is going to use that also in the future if all is stable. As far as I understand EMDN gets it data from users who have marketdump installed - included with BPC - stores it and makes it available to other clients. So its a producer of data and a consumer client like BPC or TD which are consumers can get that data. If TD or a separate program can optionally connect to EMDN it means users get up to date data which you could store in the local access db and then TD can use that data. Perhaps worthwhile to contact Andreas to see if you can work something out. Like a (json) data format which is future proof and can by used by TD, BPC and hopefully many more other authors/clients. That way you can concentrate on TD and don't have to worry about out of date data. I hope I'm making sense ;-)

Noticed this thread is in the Elite: Dangerous Fan Creations foum of ED. Most other tools are afaik discussed/published in the "Beta Discussion Forum". Normally I don't visit this thread often and I can imagine that others do the same. You might consider moving or having it moved by moderators to that thread. That way this gem
can far more exposure. Just my 2 euro cent.

[Toumal]

This could be awesome for sim pit builders who might want to pull up trade info on an auxiliary screen!

[wolverine2710]

This could be awesome for sim pit builders who might want to pull up trade info on an auxiliary screen!

Excuse me. What exactly is a 'sim pit builder'?

[Toumal]

Excuse me. What exactly is a 'sim pit builder'?

These guys:

[wolverine2710]

Wow. These guys/galls are very serious about their hobby ;-)

[wolverine2710]

@kfsone. Yesterday someone posted/released the exact system coordinates for SB1. Are you using that data to feed your SMT algorithm?

[kfsone]

The stuff about Access doesn't apply to version 3.0 and above

Sweet. A nice fresh and different approach to what is out there atm. Certainly very useful and it has functionality others atm lack. Also thanks for sharing the code. Not sure if you can also exclude a certain station from hops. That is something BPC users have asked slopey.

I'll make "--avoid" take both station and item names, and fix how avoid matches names (right now it requires an exact match, which sucks).

You lost me a bit with the access stuff. Does TD gets it data from a central database which one is connecting to or is it a local access db - which you provide and users can download a new from you side. Or that users somehow fill them self.

"Access" is part of Office, it's the database front end. For now, it's just a means to make the data easier to see / play with. When you tell it how data is related, it starts doing neat things like giving you [+] expanders to see the related data - e.g. prices for items at a given station.

You don't have to have MS Access to use the tool, but if you do have it, you can use it as a nice GUI for editing the data. There's a tool included with the code that will take my digital equivalent of 1984 scribbled notes on star systems and trades and turn them into the data it needs.

Meanwhile, I'm in the process of decoupling TD from MS Access and using something discrete/portable like SQL Lite, BDB or just good old text files. Work is giving me an Apple laptop and I imagine that would suddenly make me care about supporting the mac.

Something to consider. Andreas of marketdump is setting up EMDN (based upon EVE's EMDR) , see his thread. BPC is going to use that also in the future if all is stable. As far as I understand EMDN gets it data from users who have marketdump installed - included with BPC - stores it and makes it available to other clients. So its a producer of data and a consumer client like BPC or TD which are consumers can get that data. If TD or a separate program can optionally connect to EMDN it means users get up to date data which you could store in the local access db and then TD can use that data. Perhaps worthwhile to contact Andreas to see if you can work something out. Like a (json) data format which is future proof and can by used by TD, BPC and hopefully many more other authors/clients. That way you can concentrate on TD and don't have to worry about out of date data. I hope I'm making sense ;-)

The code is modular, the DB and the Calc are separate, so that if someone wants to do that, they can - but for me, based on past experience with things like the X universe (I wrote various plugins/modules for several X games) having it too easy can kill some of the fun.

That is - I'm entering all the data manually. You *can* just periodically pull my latest data, or you can manage your own, whichever you prefer. Someone can also easily fork the project and add a plugin that pulls down EMDN data and use that instead of using a local source.

Just my 2 euro cent.

=) I'm a brit-expat, so technically that's what my cents are, too :)

-Oliver

[kfsone]

@kfsone. Yesterday someone posted/released the exact system coordinates for SB1. Are you using that data to feed your SMT algorithm?

Can you link that? I've been manually entering star distances by using the galaxy map to hunt around (TD comes with a crap little script called 'import.py' which lets you provide a list of stars you want added and how far apart they are).

I'm toying with the idea of a triangulator so that you can plug in distances between stars and it works out the 3d map. Obviously, I'll keep it modular so that it can either read an existing stardb or you can role-play the explorer and build your own (tbh, part of the fun of writing TD has been punching in the data and gradually seeing it grow; every now and again it will suggest a daft route to get from A->B and I'll catch it before I look at the galaxy map, and that makes me feel like I'm still playing the game to some degree as opposed to just driving a train. I know that's not everyone's idea of fun, but I'm taking the grognard approach while making it possible for people to use the tool without having to be a grognard

[CMDR Moortz]

Well now, there's a name from a familiar previous WWII battleground experience.

[kfsone]

This could be awesome for sim pit builders who might want to pull up trade info on an auxiliary screen!

Python is not my forte, I picked it because I figured I could do this in a modular way and Python would offer the largest group of people the ability to leverage it for their own ends.

I also needed the practice. Starting a new job soon and they do a lot of Python.

It would certainly be easy to integrate this into something that you drive through a sim pit - it was something I had vaguely in the back of my mind while creating it.

[kfsone]

Well now, there's a name from a familiar previous WWII battleground experience.

s! =) What do you mean 'previous', though, they're still running ;-P Doc's not dead yet.

[wolverine2710]

Can you link that? I've been manually entering star distances by using the galaxy map to hunt around (TD comes with a crap little script called 'import.py' which lets you provide a list of stars you want added and how far apart they are).

I'm toying with the idea of a triangulator so that you can plug in distances between stars and it works out the 3d map. Obviously, I'll keep it modular so that it can either read an existing stardb or you can role-play the explorer and build your own (tbh, part of the fun of writing TD has been punching in the data and gradually seeing it grow; every now and again it will suggest a daft route to get from A->B and I'll catch it before I look at the galaxy map, and that makes me feel like I'm still playing the game to some degree as opposed to just driving a train. I know that's not everyone's idea of fun, but I'm taking the grognard approach while making it possible for people to use the tool without having to be a grognard

No problem.

Thread: Beta System Coordinates

This is from his OP:
Here's a list of the coordinates of every system in the beta (as of v1.03). These are exactly as used by ED itself, and are posted with the permission of Michael Brookes.

Also as you mentioned 3D map: External Galaxy Map. Please read the whole thread, useful info there.

I will reread your posts later and I probably have some response to them ;-)

It seems as if currently quite a few authors are releasing VERY interesting projects ;-)

Last weekend using crowd sourcing volunteers provided system coordinates for BPC AND I've done some work also. I know how much work that is, so kudo to you. Rep +1.

[kfsone]

No problem.

Thread: Beta System Coordinates

Thanks... Wife permitting, I'll play with this today or this week. I'm in the middle of moving so my computer-access-allowance is being crimped. [*shakes mock fist*]

[kfsone]

Sweet. A nice fresh and different approach to what is out there atm. Certainly very useful and it has functionality others atm lack. Also thanks for sharing the code. Not sure if you can also exclude a certain station from hops.

Implemented in 2.01 which I just checked in. https://bitbucket.org/kfsone/tradedangerous/overview

"--avoid" now matches against items, systems or stations.

[Aside for anyone looking at the code: I originally treated systems and stations as the same thing, I'm gradually transitioning them into separate entities. If it's a little confusing that's because I'm doing it organically]