In-Development TradeDangerous: power-user trade optimizer

[wolverine2710]

Thanks for the detailed explanation. I´ve to reread a few times so it really sinks in though ;-) Powerfull stuff.

It looks even more impressive then BPC wrt routes and its a command prompt tool. All nice. BPC has as advantage and reason I use it that the market data is up to date all the time.

Things have changed since then. Andreas marketdump tool got a real boost with EMDN - basically EVE´s EMDR. Even BPC does not upload data anymore and MD uploads market info to EMDN. BPC has become a consumer of EMDN. I´m hoping that a future version of TD has optional support for EMDN. Either TD gets data from EMDN and dumps it in the (local) access database or when running TD pulls relevant info fom the EMDN (hence real time and up to data info). MD uses zeroMQ to upload data. You can see the real time data by starting firehose.exe. Should you be willing to consider this optional feature I´m sure Andreas can help you further. It would imho give TD the boost and broader appeal it really deserves.

PS. Newsletter 36 basically said the traing wheels be removed from the market and its gonna really dynamic.....

 

[Dirk Braben]

Brilliant. Finally a decent trade calculator

Indeed very good work.

Now take it to a new level: don't only calculate:
1. buy X, Y, Z at A
2. fly through B to C
3. sell everything
4. goto 1

but do this:
1. buy X, Y, Z at A
2. fly to B
3. sell X (leaving Y and Z in the hold), buy W
4. fly to C
5. sell Y, W (leaving Z in the hold), buy V
6. fly to D
7. sell Z (leaving V in the hold), buy U
8. [...]

So taking some items from a earlier station to a station coming later in the pipeline at a higher price and don't selling everything every time.

 

[kfsone]

Thanks for the detailed explanation. I´ve to reread a few times so it really sinks in though ;-) Powerfull stuff.

It looks even more impressive then BPC wrt routes and its a command prompt tool. All nice. BPC has as advantage and reason I use it that the market data is up to date all the time.

Things have changed since then. Andreas marketdump tool got a real boost with EMDN - basically EVE´s EMDR. Even BPC does not upload data anymore and MD uploads market info to EMDN. BPC has become a consumer of EMDN. I´m hoping that a future version of TD has optional support for EMDN. Either TD gets data from EMDN and dumps it in the (local) access database or when running TD pulls relevant info fom the EMDN (hence real time and up to data info). MD uses zeroMQ to upload data. You can see the real time data by starting firehose.exe. Should you be willing to consider this optional feature I´m sure Andreas can help you further. It would imho give TD the boost and broader appeal it really deserves.

PS. Newsletter 36 basically said the traing wheels be removed from the market and its gonna really dynamic.....

I'm fairly familiar with 0MQ (this is me).

The way TD is written, it would be fairly easy for someone to make it use a scraped database - either by pulling on the fly or by making a scraper that writes to a local DB.

But I'm stopping short of doing that myself. I think I've made it fairly easy for someone else to do (and indeed, someone has already forked TD on bitbucket and perhaps that is what they intend to do).

I'm reluctant to do it myself, based on my experience writing stuff in the X universe (e.g. x2 cargo ferries), because it would eliminate ALL interaction with the market for me. It only takes me a few seconds to update my local DB by hand using MS Access, but those few seconds actually help me get a sense of where the market is fluctuating along the routes.

 

[kfsone]

Brilliant. Finally a decent trade calculator

Indeed very good work.

Now take it to a new level: don't only calculate:
1. buy X, Y, Z at A
2. fly through B to C
3. sell everything
4. goto 1

but do this:
1. buy X, Y, Z at A
2. fly to B
3. sell X (leaving Y and Z in the hold), buy W
4. fly to C
5. sell Y, W (leaving Z in the hold), buy V
6. fly to D
7. sell Z (leaving V in the hold), buy U
8. [...]

So taking some items from a earlier station to a station coming later in the pipeline at a higher price and don't selling everything every time.

That gets very expensive to do very quickly, because you accumulate potential routes to compare with every hop - best case is logarithmic. (50 stations = 49 x (49^routes)).

But I also found that it rarely found combinations that paid off, and with the introduction of fuel costs it's going to be a very, very small margin. It might work for 4-16 capacity ships with a 2 hop limit (everything has to be gone within two hops).

Or I might be able to use shenanigans ... to do something like, say: if you can only afford 1-2 of something with a high profit threshold and it requires 2 jumps to sell, try and find secondary trades on the hops inbetween.

So: 4 capacity, 4000cr : 2 x Tea (1500cr) -> A -> [via B] -> C.
Since we cashed out, and there are 2 hops to get to C, apply a second trade calc to solve A->B->C with 2 capacity and 1000cr finds 1 food cart, 1 alga from A->B pays for the fuel for the trip, 1 bio lichen b->c adds 120 profit which is small but is a significant percentage.

Would that be sufficient?

 

[kfsone]

Just checked in 2.06:
- Adds a "--checklist" command to walk you through the steps in the route,
- Adds a "--x52-pro" command to walk you thru the checklist on your X52 Pro MFD


(click for larger view)

I'm waiting on feedback from Saitek so I can let you use the X52 MFD buttons rather than having to alt-tab out and hit enter.

 

[demiurge]

Just checked in 2.06:
- Adds a "--x52-pro" command to walk you thru the checklist on your X52 Pro MFD

I like the idea of displaying the route on the MFD. Is also displays the destination next hop?
What algorithm you applied to calculate the best routes of trade? I am interested in your solution. Since yesterday, trying to understand it.
I wrote my own program, in which I used two algorithms: Minimax to determine the best next hop, and the knapsack problem to determine the best goods from the station with adequate financial resources. But it involved a small number of stations in Beta Premium (7 stations). Now we have 41 stations, thus finding the optimal solution to this problem seems to be quite difficult.

 

[Blocks]

This looks fantastic - thank you for sharing it!

It looks like it's already really useful and will develop into an incredibly powerful tool if you keep developing it

 

[kfsone]

I like the idea of displaying the route on the MFD. Is also displays the destination next hop?

The display layer will show you the entire route and then, if you specified that you want checklist mode, walk you through it step by step:

1. Buy 100x frogs
2. Refuel
3. Jump to LHS 2887
4. Jump to Dahan
5. Dock at Dahan Gateway
6. Sell 100x frogs
7. Buy 100x chocolate
8. Refuel
9. Jump to ...

the degree of detail depends on whether or not you specify '--detail'.

At each step, it displays a summary on the MFD. It's still brand new so I'm open to suggestions. Ideally, I want you to be able to interact with the app via the MFD but Saitek's documentation is abysmal and their API is kinda crap.

What algorithm you applied to calculate the best routes of trade? I am interested in your solution. Since yesterday, trying to understand it.

A* for the routing selection and my own variant of knapsack for the cargo selection.

I wrote my own program, in which I used two algorithms: Minimax to determine the best next hop, and the knapsack problem to determine the best goods from the station with adequate financial resources. But it involved a small number of stations in Beta Premium (7 stations). Now we have 41 stations, thus finding the optimal solution to this problem seems to be quite difficult.

It's actually a fairly interesting problem, from an algorithmic point of view, if you want to solve it for yourself and get a good handle on it, grab a bunch of Monopoly money and some index cards and make yourself a small inventory to experiment with - I did this to show my wife how I was doing it (she doesn't understand programming at all).

The idea is to find simple best fits and then attempt to collapse them (sure I can make $20 from 3 of A, but it turns out with the same money I can afford 4 of B which is less profitable per, but since I can afford 4 I actually make $21). Using python's generators allowed me to do this more lazily and, because of the way I've marshaled the data and modern CPU architecture works, it's actually more efficient than taking the algorithmic approach because it gets less branch fails.

 

[Virgil Kyle]

These guys:

Spoiler

Spoiler

Spoiler

So envy. Much awesome.

Wow.

 

[kfsone]

New version, I'm working on several new features, but this version has a lot of improvements to presentation and "--detail" (or -v for short) can be specified twice to get even more data (e.g. profit/ton on each hop), the X52 Pro MFD is working nicely, and you can now specify --ship rather than having to fill out the ly-per etc fields.

Version 3.0 will work slightly differently: There'll be sub-commands so you can ask it things like "where do I buy the Anaconda" or "where can I get a C4 Cannon around this neck of the woods?" or "where can I find cheap Tobacco around here" or "what jumps do I have to make to get from chango to Morgor with this ship?".

 

[soapmak3r]

Might be a good idea to link directly to where we can download the required packages for anyone not familiar with them.


Is it "Microsoft Access Database Engine 2010 Redistributable" that I need for "Microsoft Access 2010+ drivers"?

Searching "Microsoft Access 2010+ drivers" brought up all kinds of results including CNET (likely malware/spyware infested) sponsored downloads.

The other 2 were straight forward to locate though.

Thanks for your effort.

*EDIT* Ah well, guess this is not for me. I can't make head nor tail of it (as in, I don't even know how to get anything to load or what to do with the downloaded files).

An Android app would be very handy, with a simple search function for trade routes and what is most profitable for any given route.

 

[Hondo]

Hi Mate,

This is really usefull. Thank you for taking your time out to create something like this for us traders.

I was wondering if you can help me with an error im getting with getting x52 pro MFD working.

I'm getting the following error:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 307, in __init__
self.direct_output = DirectOutput(dll_path)
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 99, in __init__
self.DirectOutputDLL = ctypes.WinDLL(dll_path)
File "F:\Program Files (x86)\Python\lib\ctypes\__init__.py", line 351, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\trade.py", line 77, in __init__
self.doObj = saitek.X52Pro.SaitekX52Pro()
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 309, in __init__
raise DLLError(e.winerror)
saitek.DirectOutput.DLLError: Unable to load DirectOutput.dll - possible 32/64 bit mismatch between Python interpreter and DLL

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\trade.py", line 382, in <module>
main()
File "F:\Downloads\Trade Dangerous\trade.py", line 329, in main
parse_command_line()
File "F:\Downloads\Trade Dangerous\trade.py", line 247, in parse_command_line
mfd = X52ProMFD()
File "F:\Downloads\Trade Dangerous\trade.py", line 79, in __init__
raise Exception('Unable to initialize the X52 Pro module. Make sure your X52 is plugged in and you have the drivers installed.')
Exception: Unable to initialize the X52 Pro module. Make sure your X52 is plugged in and you have the drivers installed.

I can confirm the X52 is plugged in and it is working on Elite. Any help is greatly appreciated.

Thanks,
Hondo

 

[kfsone]

Hi Mate,

This is really usefull. Thank you for taking your time out to create something like this for us traders.

I was wondering if you can help me with an error im getting with getting x52 pro MFD working.

I'm getting the following error:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 307, in __init__
self.direct_output = DirectOutput(dll_path)
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 99, in __init__
self.DirectOutputDLL = ctypes.WinDLL(dll_path)
File "F:\Program Files (x86)\Python\lib\ctypes\__init__.py", line 351, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\trade.py", line 77, in __init__
self.doObj = saitek.X52Pro.SaitekX52Pro()
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 309, in __init__
raise DLLError(e.winerror)
saitek.DirectOutput.DLLError: Unable to load DirectOutput.dll - possible 32/64 bit mismatch between Python interpreter and DLL

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\trade.py", line 382, in <module>
main()
File "F:\Downloads\Trade Dangerous\trade.py", line 329, in main
parse_command_line()
File "F:\Downloads\Trade Dangerous\trade.py", line 247, in parse_command_line
mfd = X52ProMFD()
File "F:\Downloads\Trade Dangerous\trade.py", line 79, in __init__
raise Exception('Unable to initialize the X52 Pro module. Make sure your X52 is plugged in and you have the drivers installed.')
Exception: Unable to initialize the X52 Pro module. Make sure your X52 is plugged in and you have the drivers installed.

I can confirm the X52 is plugged in and it is working on Elite. Any help is greatly appreciated.

Thanks,
Hondo

Hey, Hondo,

The app accesses the stick through Saiteks "DirectOutput" dll. The error you're seeing sounds like what would happen if you didn't have that, e.g. I'd you had the version of the drivers that came with the stick.

Can you confirm you have been to the Saitek website and downloaded and installed the latest version of their drivers?

Which OS are you running? Xp? Vista? 7? Are you running it through a virtual machine or wine or something?

Which version of python are you running? (Do python --version)

Lastly, try doing a git pull to make sure you don't have an old version I'd broken ; g

 

[kfsone]

Might be a good idea to link directly to where we can download the required packages for anyone not familiar with them.


Is it "Microsoft Access Database Engine 2010 Redistributable" that I need for "Microsoft Access 2010+ drivers"?

Searching "Microsoft Access 2010+ drivers" brought up all kinds of results including CNET (likely malware/spyware infested) sponsored downloads.

The other 2 were straight forward to locate though.

Thanks for your effort.

*EDIT* Ah well, guess this is not for me. I can't make head nor tail of it (as in, I don't even know how to get anything to load or what to do with the downloaded files).

An Android app would be very handy, with a simple search function for trade routes and what is most profitable for any given route.

Hi, soap.

I thought it actually did direct the user thusly, mindbomb had added some error text.

Ill be ditching the access requirement for something more pythonic in the next few days.

It's currently command line driven, so you need a command prompt or shell in the directory the trade.py script is in, but that's why I gave this the title "power user" I may eventually give it a gui or a prompt driven shell of its own, but for now I was mostly trying to provide other developers a powerful tool for doing the grunt work.

Example usage:

C:/trade/> trade.py -v -v --ship type6 --cr 40000 --from morgor --avoid Anderson --avoid gold --hops 5 --via aulin

 

[Hondo]

Hi kfsone,

Yep I can confirm I'm running version that is available on the saitek web site. I downloaded the Winodws 8.1 64bit version and installed it today.

Im running Python version 3.4.1 and no virtual machine.

Where can I find it via git. Version I download was from here:

https://bitbucket.org/kfsone/tradedangerous/downloads

Thanks,
Hondo

 

[Seonid]

It is in git form, just git clone https://bitbucket.org/kfsone/tradedangerous.git

 

[Hondo]

It is in git form, just git clone https://bitbucket.org/kfsone/tradedangerous.git

so to confirm I did download the correct one?

 

[kfsone]

so to confirm I did download the correct one?

It's a temporal question - i.e. when not where There's no auto-update, you just have to periodically get the new version, which is why doing it with git might be handy. If you don't know/have/want git, just pull the latest download periodically.

BUT ... All that aside, I missed this when I was replying on the kindle last night:

saitek.DirectOutput.DLLError: Unable to load DirectOutput.dll - possible 32/64 bit mismatch between Python interpreter and DLL

You said you have the 64bit Saitek driver installed. If your python is 32-bit you'll have to also-install the 32-bit Saitek driver for it to be able to use it.

I think.

 

[Hondo]

Hey Mate,

I think you are right. I did have python 32bit version installed. I just went ahead and installed 64bit version. But I'm still getting the same error. This what I have done so far.

1. Remove Python 32bit version and and installed Python 64bit
2. Re-installed pypyodbc and ran setup.
3. Re-isntalled Saitek x52 Pro drivers from the web site (Windows 8.1 64bit)

Python Version:

F:\Downloads\Trade Dangerous>python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

When I run the command with x52 i get the following:
Command: trade.py --ship hauler --from Chango --credits 20000 --hops 2 --x52-pro

Error:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 307, in __init__
self.direct_output = DirectOutput(dll_path)
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 99, in __init__
self.DirectOutputDLL = ctypes.WinDLL(dll_path)
File "F:\Program Files\Python\lib\ctypes\__init__.py", line 351, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\trade.py", line 84, in __init__
self.doObj = saitek.X52Pro.SaitekX52Pro()
File "F:\Downloads\Trade Dangerous\saitek\DirectOutput.py", line 309, in __init__
raise DLLError(e.winerror)
saitek.DirectOutput.DLLError: Unable to load DirectOutput.dll - possible 32/64 bit mismatch between Python interpreter and DLL

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "F:\Downloads\Trade Dangerous\trade.py", line 396, in <module>
main()
File "F:\Downloads\Trade Dangerous\trade.py", line 343, in main
parse_command_line()
File "F:\Downloads\Trade Dangerous\trade.py", line 264, in parse_command_line
mfd = X52ProMFD()
File "F:\Downloads\Trade Dangerous\trade.py", line 86, in __init__
raise Exception('Unable to initialize the X52 Pro module. Make sure your X52 is plugged in and you have the drivers installed.')
Exception: Unable to initialize the X52 Pro module. Make sure your X52 is plugged in and you have the drivers installed.


I have tried everything I can think of

Any help is greatly appreciated.

 

[Hondo]

Hi kfsone,

I figured out what the problem was. I'm posting it here as it maybe useful to someone else.

Problem was infact using the incorrect version of the DirectOutput.dll file. I have windows 8.1 64bit and have the Saitek 64 bit drivers install and it doesnt allow me to install the 32bit drivers because im running windows 8.1 64bit version.

Problem is that DirectOutput.py is set use the 32bit version of DirectOutput.dll.

To fix this I edited the following entry in the file:

dll_path="C:\\Program Files (x86)\\Saitek\\DirectOutput\\DirectOutput.dll

to

dll_path="C:\\Program Files\\Saitek\\DirectOutput\\DirectOutput.dll

Any now it works.


Now I have couple of other questions.

Do you need to use --checklist to make useful in the MFD?

If I dont use --checklist it shows something in MFD for a second and it disappears.

Same thing if I use --detail.

Only way its useful is if I run --checklist then it steps you through it but You have to ALT-TAB to go to the next step. I know you have contacted Saitek about it, have you had a response?

is it possible to display something like below without using --checklist and --detail

I BOOTIS Chango Dock -> DAHAN Gateway:
>-> At I BOOTIS/Chango Dock, Buy: 16 x Fish,
+ At LP 98-132/Freeport, Buy: 16 x Coltan,
<-< DAHAN Gateway gaining 9,856cr => 29,856cr total

Im not sure if the MFD is big enough to display all this info, but if you can get it to dot it it will be very useful as it has all the info you need really on the screen and you don't need to alt + tab out.

Thanks,
Hondo