In-Development TradeDangerous: power-user trade optimizer

[kfsone]

To fix this I edited the following entry in the file:

dll_path="C:\\Program Files (x86)\\Saitek\\DirectOutput\\DirectOutput.dll

to

dll_path="C:\\Program Files\\Saitek\\DirectOutput\\DirectOutput.dll

Any now it works.

I'm working on removing the MS Access dependency right now (and then moving house tomorrow so there may be a short delay until I follow up again), but I'll do something about that - at least provide better feedback. Thanks for posting the detail.

Now I have couple of other questions.

Do you need to use --checklist to make useful in the MFD?

Short answer: Yes.

FYI - all of the commands can be abbreviated to their least-ambiguous abbreviation. "--cr" for credits, "--ch" for "checklist", "--x" for x52.

I'll make it tell you that you need to supply an interactive command, such as --checklist, for it to be useful.

If I dont use --checklist it shows something in MFD for a second and it disappears.

It's saying "TradeDangerous" "INITIALIZING" lol.

Only way its useful is if I run --checklist then it steps you through it but You have to ALT-TAB to go to the next step. I know you have contacted Saitek about it, have you had a response?

Yes. They said "I'll ask one of the developers". I'm in temporary housing right now, on my way to a shiny new job at Facebook, and I'm moving into my permanent housing tomorrow and Tuesday. When I'm done with that and the wife lets me near a computer again, I'll chase them.

is it possible to display something like below without using --checklist and --detail

I BOOTIS Chango Dock -> DAHAN Gateway:
>-> At I BOOTIS/Chango Dock, Buy: 16 x Fish,
+ At LP 98-132/Freeport, Buy: 16 x Coltan,
<-< DAHAN Gateway gaining 9,856cr => 29,856cr total

Im not sure if the MFD is big enough to display all this info, but if you can get it to dot it it will be very useful as it has all the info you need really on the screen and you don't need to alt + tab out.

I'm still churning ideas for displays; if I hadn't left my G19 at Blizz I would have added a driver for that too.

But the idea is that you'll be able to use the buttons on the stick below the MFD to go through steps. All I need is for Saitek to tell me why their driver isn't sending me those key presses. (I tried writing the app in C# and C++ to see if I could get it to work, and it didn't work there either, so it could just be that the X52 Pro drivers are shafted in perpetuity. Hopefully not, but brace yourself for the possibility)

If you get the latest TD version, you'll see I cleaned up the display yesterday (less of the frippery crap like which hop and step).

If you'd like to try out ideas on the stick itself, you can do the following (and, really, this is the reason I'm doing this project garage style like this instead of writing some super-pretty user-friendly app)

[note: I have git bash installed, so where you see '$' it's basically equivalent to the dos prompt]

$ cd WHERE_TRADE.PY_IS
$ python
>>> import trade
>>> mfd = trade.X52ProMFD()
>>> mfd.display("This is the first line", "This is the middle line", "This is the last line")

and you'll get a sense of what the constraints are.

The marquee scrolling is a bit annoying, and I'd only be able to display 2-3 hops, but that might not be terrible. I'm just not sure there's a good way to use it all for anything but the simpler routes:

$ ./trade.py --hops 2 --ship type9 --from Chango --credits 6434431 --check --x52 --compact

outputting

[Chango Dock 228 x Fish]
[Aulin Enterprise 228 x Dom.Appliances]
[Vonarburg Co-operative]

looks a bit weird, and you have to wait for the scrolling to see what you're supposed to buy or where you're supposed to be

Bear in mind: The next beta is likely to be introducing multiple stations per system (I'm incorporating that into the work I'm doing to move away from the access database right now)

So you probably want it to display system as well as station name:

[CHANGO/Chango Dock 228 x Fish]
[AULIN/Aulin Enterprise 228 x Dom.Appliances]
[WYRD/Vonarburg Co-operative. KERCHING!]

-Oliver

 

[Hondo]

Thanks Oliver for taking the time out to respond.

Good luck with your new job and hopefully you will find some time to develop this little app bit more as I think Its one of the most useful apps out there.

If we can the Saitek MFD to work properly as well as the G19 display to work that will be really handy...

I'll take a look what you have suggested.... but Im not a competent programmer at all

 

[kfsone]

Good luck with your new job and hopefully you will find some time to develop this little app bit more as I think Its one of the most useful apps out there.

It's just the moving house move the next couple of days that'll bite into my time, mostly because I might not have internet until later on Tuesday

If we can the Saitek MFD to work properly as well as the G19 display to work that will be really handy...

I'll bend their ear on Wednesday, if I haven't heard from them. And I'll see about picking up another G19 soon. If anyone else has any other MFD type displays I'll look into those down the line.

I'll take a look what you have suggested.... but Im not a competent programmer at all

Don't worry too much about programming, I just thought it might be handy to try out various display formats and see how they look on the MFD.

If you've put the trade stuff into C:\TradeDangerous\ you'd do this to give yourself a connection to the MFD to play with:

C:\Users\Hondo\> cd \tradedangerous
C:\TradeDangerous\> python
>>> import trade
>>> mfd = trade.X52ProMFD()

and then just use change the stuff between quotes to try out whatever format you think might work:

>>> mfd.display("First thing", "Second Thing", "Third Thing")

Given that, it'd put "First thing" on the top line, second on the middle and third on the bottom. You only get 3 lines on these MFDs (because they just couldn't waste enough space)

So you could do:

>>> mfd.display("Trade 200 x Fish at CHANGO / Chango Dock", "Trade 200 x Dom.Appliances at AULIN / Aulin Enterprise", "FINISH at WYRD / Romanburg Convoluted-Name Place")

and see how that looks on your MFD - I think you'll probably find the scrolling is really annoying

I can write code, but I suck at visuals and layouts so - I welcome anyone's input on formatting/layout for the MFD

-Oliver

 

[kfsone]

Thanks Oliver for taking the time out to respond.

Good luck with your new job and hopefully you will find some time to develop this little app bit more as I think Its one of the most useful apps out there.

If we can the Saitek MFD to work properly as well as the G19 display to work that will be really handy...

I'll take a look what you have suggested.... but Im not a competent programmer at all

Oh, and if you get tired of having to do manual downloads, you can use "git" to pull the latest versions.

Option 1: Install "Source Tree" from Atlassian. Then it's a gui and fairly self explanatory. Just do a "pull" periodically. Might be a bit excessive if you're only using it to update TradeDangerous, lol.

Option 2: Install "git" http://git-scm.com/download/win
Figure out where you want to put TD, e.g. you could put it in "trade" under your windows home directory... Open a command prompt and use the command below:

C:\Users\Hondo\> git clone git@bitbucket.org:kfsone/tradedangerous.git tradedangerous

If you don't want to have to be typing 'tradedangerous' all the time

C:\Users\Hondo\> git clone git@bitbucket.org:kfsone/tradedangerous.git td

will extract it into a directory called 'td' instead.

If you want it somewhere else, say C:\td

C:\Users\Hondo\> cd \
C:\> git clone git@bitbucket.org:kfsone/tradedangerous.git td

To use it.. just cd into the directory

C:\Users\Hondo\> cd tradedangerous
C:\Users\Hondo\tradedangerous\> trade.py ...
or
C:\Users\Hondo\> cd td
C:\Users\Hondo\td\> trade.py ...
or
C:\Users\Hondo\> cd \td
C:\td\> trade.py ...

depending where you put it

To refresh with the latest code:

C:\Wherever\td\> git pull

and it'll update itself.

 

[Crank Larson]

I'd like to add my thanks for this script too. It's excellent, and has rescued me from my spreadsheet so I can now concentrate on flying instead of working out routes... I had tried some of the other trading applications too, but I prefer this by a long shot.

 

[kfsone]

I've just published the first alpha of version 3.0:

- Switched from the Microsoft Access DB to using text files as source and an SQLite database backend as a cache
-- Laymans terms:
Faster, easier to use, simpler to install,
Edit "data/TradeDangerous.prices" with pricing updates,
-- Dev notes:
The .db file gets rebuilt if it's missing or if the .sql or .prices file is updated.

- Should be possible to run on a Mac or a Linux box now (I haven't tested yet, will do that when my wife isn't wondering why I'm spending our first anniversary not even playing ED

- Lots of cleanup,

- Lots of speed improvements,


If you're feeling frisky and want to give it a try, it's still in its own branch:

https://bitbucket.org/kfsone/tradedangerous/branch/sqlite


Let me know if you run into any problems.

Still haven't heard back from Saitek about driving the soft-buttons on the X52 through Python.

-Oliver

 

[kfsone]

Ok, version 3.0 is live in the main branch now.

It no-longer uses MS Access, it uses the batteries-included Python sqlite support instead. It builds the database from files in the "data" directory. If you want to update your local prices, just edit "TradeDangerous.prices" (should be self-explanatory) and next time you run the tool it will incorporate the data.

A few of the command line options have been shuffled around, mostly to make it so that you can use abbreviations ("--cr" instead of "--credits", etc) and it has slightly more ship information.

In the next couple of versions I'll be improving the checklist system to be more like a console that tracks where you're at, how much money you have, etc.

I'm also planning a "--goal" option to let you say 'tell me the runs I need to make to get this much money', to make the MFD/Checklist more interactive (so if you have to correct prices after 2 hops, you can ask it to recalculate your route for you).

Let me know if you have any problems.

 

[Lazerguns]

Awesome work! To be honest I don't understand how this algorithm is so fast compared to some other trade route finders

I'm in the process of writing a tool to pull data down from EMDN and loading it into tradedangerous.

I started to pipe fh.exe's output (from Andreas' marketdump package) to my tool, parsing the json and producing your import format.

As you switched to sqlite now, I think it would be more efficient to write to the db directly, so I would start again (and directly connecting to 0mq instead of piping from fh.exe). It is written in go, as I don't do python (I'm sorry...)

I have the problem that Station and Item ID's do not match your schema because they are getting it directly from the engine, and I reckon the display names in the market screen differ sometimes. For example, Hel-Static Furnaces => heliostaticfurnaces.

I think the best way to tackle this would be an extra column in the schema (i.e. emdn_name) for Items and Station/System Names. Would you take a patch for this? It would spare me from mainaining a fork

Thanks again,

CMDR Lazerguns

 

[kfsone]

Awesome work! To be honest I don't understand how this algorithm is so fast compared to some other trade route finders

I'm in the process of writing a tool to pull data down from EMDN and loading it into tradedangerous.

I started to pipe fh.exe's output (from Andreas' marketdump package) to my tool, parsing the json and producing your import format.

As you switched to sqlite now, I think it would be more efficient to write to the db directly, so I would start again (and directly connecting to 0mq instead of piping from fh.exe). It is written in go, as I don't do python (I'm sorry...)

I have the problem that Station and Item ID's do not match your schema because they are getting it directly from the engine, and I reckon the display names in the market screen differ sometimes. For example, Hel-Static Furnaces => heliostaticfurnaces.

I think the best way to tackle this would be an extra column in the schema (i.e. emdn_name) for Items and Station/System Names. Would you take a patch for this? It would spare me from mainaining a fork

Thanks again,

CMDR Lazerguns

Oh, absolutely; grab the latest 3.1, and basically edit the data/tradedangerous.sql file and mail me a diff (oliver@kfs.org).

3.x has some significant changes, and 3.1 more so. Really all your version needs is a variant of the cache populate function.

And with regards to speed, it's lacking there, I need to do another optimize pass. I'm still learning python, and while I was able to use some nice tricks to stop python making it show, this is a pretty simple operation and very well suited to modern hardware. It really ought to be able to do several hundred hops per second rather than 10-20. But I'm still w writing my python like a c programmer

 

[Hondo]

Hey Mate,

I thought I try out the version 3.0 but having some issues. I tried running a simple command.

trade.py --from Chango --credits 20000 --hops 2

And i get the following.

usage: trade.py [-h] [--debug] [--db DB] {run,update} ...
trade.py: error: argument subparser: invalid choice: 'Chango' (choose from 'run', 'update')

if I just run trade.py, i get the following:

ERROR: Specified file, "data\TradeDangerous.db", does not exist.

Help?

Thanks,
Hondo

 

[kfsone]

Hey, Hondo,

The command line has been expanded, as a result you have to tell it you want to do a trade run -- you'll see why this inconvenience in a version or two

So instead of just "./trade.py ..." you have to say "./trade.py run".

Assuming you have a Lakon Type 6, you can do:

./trade.py run/ --fr chango --cr 20000 --ship type6

Things to note:

. If in doubt, add "-h" to the line, e.g. "./trade.py -h" or "./trade.py run -h".

. You don't *have* to type out the full form of each option - e.g. --fr instead of --from, --cr instead of --credits.

. 2 is the default number of hops,

. You need to specify your ship so it can approximate things like how far you can jump, ship names use the same matching code so type6 or "lakon type 6" or "pe 6" will match.

. The only other command at the moment is "update", although I've only half implemented it.

. The "db error" is my bad, it's supposed to tell you that you didn't select a command. I've checked in a fix for it.

 

[kfsone]

I've updated the README.

 

[kfsone]

Other sub-commands I have in-mind:

nav[igate] stationA stationB [--via ...] [--avoid ...]
Plan jumps for me to get from A to B

find ...
Tell me the best closest or best place to buy items,
ships or upgrades from where I am; or find the best
place to sell a cargo I have.

 

[Hondo]

Thanks Mate,

It works like a charm.

I got around the Saitek MFD using autohotkey (http://www.autohotkey.com/)

With that app and help form their members on the forums I now use a script that when I press enter it send the enter command to the cmd and I can step through the every step in the MFD without having to all tab out. Its only Temp fix but it works.... hopefully you will hear back from saitek so there can be a permanent fix.

 

[kfsone]

Thanks Mate,

It works like a charm.

I got around the Saitek MFD using autohotkey (http://www.autohotkey.com/)

With that app and help form their members on the forums I now use a script that when I press enter it send the enter command to the cmd and I can step through the every step in the MFD without having to all tab out. Its only Temp fix but it works.... hopefully you will hear back from saitek so there can be a permanent fix.

Apparently it's "complex", and it may take them some time to get an answer (their words).

 

[kfsone]

I'm in the process of writing a tool to pull data down from EMDN and loading it into tradedangerous.

I knocked up a small EMDN module in python to do this, grab version 3.3. I've also added support for the internal game names (table is AltItemName in the DB).

-Oliver

 

[wolverine2710]

I haven't used your tool, but after a 2 week vacation going to use/test it. Very curious how it compares to BPC wrt speed. If I've interpreted it correctly it now uses sqllite to read its data instead of Access. Cool. I plan to install it on my 2nd PC which is on 24/7. Is it possible to run the tool on my ED PC and get the data from my 2nd PC?

Not tested it, but is TD multi-threaded?
If not, is that something your are planning to add?
You have a navigation system in place (possible using Dykstra or an A* search. Atm there are two navigation systems out there which also take fuel consumption into account. Is that something you are planning to integrate into TD?

Very curious what your tool will contain when I'm back in two weeks ;-)
I'm a CLI person, so I very much like this tool. Also I happen to have a X52!!!

Edit: Out of your hands but something for Andeas. Because of missing securirty setups in the ElasticSearch Andreas can't have users accessing it directly. The firehose works fine when you contact to a it for a longer period. Would be nice if Anderas would have a web api which gives back the latest prices for a station, multiple stations, or even all stations. That preserves band with on his side I think and it would be possible to directly have the latest prices, for for example TD. That EMDN of his works great and enables authors to have data without setting up hosted databases themselves. Tip of the iceberg I think of what would be possible!!
Even better if Andreas somehow finds a way to protect ES and have it read-only open to the world so that ES-queries can be made directly. There are tools for this I believe....

Edit: Good luck with your new home ;-)

 

[Net-burst]

Abount EMDN. Kfsone, do you plan to integrate price update from EMDN into TD? I tried to somehow do it myself, but wasn't able to

 

[wolverine2710]

Abount EMDN. Kfsone, do you plan to integrate price update from EMDN into TD? I tried to somehow do it myself, but wasn't able to

From the OP: "v3.1 Sep 01/2014
Introduced sub-commands:
- "run" command provides the old default behavior of calculating a run,
- "update" command provides ways to update price database easily,"

Also from Andreas marketdump thread he post this. So it looks to be already in TD - to some extend. Though its not really documented. Tbh I've tried it but got an error about zeroMQ missing. Could be doing something wrong though.

Perhaps in the future he's going to do it like Andreas, an exe with Python embedded into it. TD is blindingly fast. I can see this becoming an engine being used by others. As in others who create a GUI, call TD and then display the results in their GUI. Only thing missing for that is some optional JSON and/or XML output which can be easily parsed.

 

[kfsone]

I haven't used your tool, but after a 2 week vacation going to use/test it....

No, TD is not multi-threaded, there's really no reason for it to be, it's pretty fast... Here it's calculating a 30 hop run for a type 6 starting off with 600 credits all the way to 2.3 million:

$ time ./trade.py run --hops 30 --ship type6 --credits 600
unspecified hops 31, numHops 30, viaStations 0
LHS 417 Gernhardt Camp -> BOLG Moxon's Mojo:
 >-> At LHS 417/Gernhardt Camp, Buy: 54 x Biowaste,
  +  At WYRD/Vonarburg Co-operative, Buy: 96 x Leather,
  +  At TILIAN/Maunder's Hope, Buy: 100 x Basic Medicines,
  +  At BOLG/Moxon's Mojo, Buy: 100 x Mineral Extractors,
  +  At ITHACA/Hume Depot, Buy: 100 x Uraninite,
  +  At ACIHAUT/Cuffey Plant, Buy: 87 x Lithium, 13 x Hydrogen Fuels,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 99 x Reactive Armor, 1 x H.E. Suits,
  +  At LFT 880/Baker Platform, Buy: 100 x Uranium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Reactive Armor,
  +  At LFT 880/Baker Platform, Buy: 100 x Uranium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 73 x Consumer Tech, 27 x H.E. Suits,
  +  At LFT 880/Baker Platform, Buy: 100 x Uranium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Consumer Tech,
  +  At LFT 880/Baker Platform, Buy: 94 x Beryllium, 2 x Uranium,
  +  At ASELLUS PRIMUS/Beagle 2 Landing, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Lithium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Consumer Tech,
  +  At LFT 880/Baker Platform, Buy: 100 x Beryllium,
  +  At ASELLUS PRIMUS/Beagle 2 Landing, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Lithium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Consumer Tech,
  +  At LFT 880/Baker Platform, Buy: 100 x Beryllium,
  +  At ASELLUS PRIMUS/Beagle 2 Landing, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Lithium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Consumer Tech,
  +  At LFT 880/Baker Platform, Buy: 100 x Beryllium,
  +  At ASELLUS PRIMUS/Beagle 2 Landing, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Lithium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Consumer Tech,
  +  At LFT 880/Baker Platform, Buy: 100 x Beryllium,
 <-< Moxon's Mojo gaining 2,385,195cr => 2,385,795cr total

drumroll:

real    0m2.800s
user    0m0.000s
sys     0m0.046s

So there's really not a lot of reason for it to be multi-threaded.

And it's been a long time since I did an optimization pass. I can probably drop 0.8-1.0s of that time if I give it an hour or so.

Despite the speed, the algorithm is accurate, not just precise. If you look at the code in tradecalc.py you'll see there are two "fit" routines. One uses a sheer, bloody minded "try EVERY combination" approach and the other is the normal algorithm.

In terms of jump distances and fuel costs, I'm already in the process of incorporating those in (check ships.py), as well as how far stations are from the landing point in a star system.

And I just committed a version with the "emdn-tap.py" tool for fetching data from EMDN into the TD database.