In-Development TradeDangerous: power-user trade optimizer

wolverine2710 · Sep 6, 2014

Thanks for the fast response, really appreciated. Played with it a bit this morning. Indeed its AMAZINGLY FAST. Much faster then the routines used by Slopey in its BPC. Don't have time to fully test it and compare it to BPC. Tomorrow I'm flying - holiday.

I've figured out how to do a ANY to ANY by leaving out from and to. I think I figured out a roundtrip like the BPC by setting --from and --to the same station. Though I get different values then with BPC - could be prices not updated yet. Haven't figured out to do an ANY to ANY with roundtrip like BPC. Am I missing something or?

With --routes you get more suggestions. But it looks as if that are suggestions for stations. Setting --from and --to different stations I get a traderoute but that is only for one commodity - best route I suspect. BPC shows multiple commodities with different profits. Is that possible with TD also?

Speaking about roundtrips. Given the amazing speed of TD. Would it be possible to find the best roundtrips, not for two but for for example 3-4 stations. So it involves buying/selling at multiple stations to get the best trade route.

About speed: Its amazing, and you stated with some tweaking it could be even be much better/faster. Should TD ever becoming not fast enough, you could always go the MT route - if you deem that needed.

As I stated before: I can see this become the engine of choice for GUI builders - if optional JSON/XML output is implemented. Is that something you would like to pursue/see happen?

wolverine2710 · Sep 6, 2014

Just tried TD 3.4 and emd-tap.py. I'm getting the following error about zeroMQ missing. I guess I can install that one myself, but would it be possible to add that one to a future TD 3.x version.

Message:
c:\Data\Games\Elite Dangerous\TD 3.4>c:\Python34\python.exe emdn-tap.py
Traceback (most recent call last):
File "c:\Data\Games\Elite Dangerous\TD 3.4\emdn\firehose.py", line 37, in <module>
import zmq
ImportError: No module named 'zmq'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "emdn-tap.py", line 29, in <module>
from emdn.firehose import Firehose
File "c:\Data\Games\Elite Dangerous\TD 3.4\emdn\firehose.py", line 39, in <module>
raise ImportError("This module requires the ZeroMQ library to be installed. The easiest way to obtain this is to type: pip install pyzmq")
ImportError: This module requires the ZeroMQ library to be installed. The easiest way to obtain this is to type: pip install pyzmq

Net-burst · Sep 6, 2014

wolverine2710, its telling you that your Python environment don't have extension to handle ZeroMQ. Just run "python pip install pyzmq" and it will install it. Its not possible to ship extensions, I think. End user must install it separately.

kfsone · Sep 6, 2014

wolverine2710 said:
Just tried TD 3.4 and emd-tap.py. I'm getting the following error about zeroMQ missing. I guess I can install that one myself, but would it be possible to add that one to a future TD 3.x version.

At the moment you're just grabbing the source, so there's no way for me to do that automatically.

Eventually, I'll use "py2exe" to make an executable release of version snapshots which will be able to do that, but right now you just have to do the "pip install pyzmq" as the error says on the last line.

If you grab the most recent version (v3.5) I'd already made that error a little more self-evident

kfsone · Sep 6, 2014

v3.5 checked in: now using the JSON compressed stream for the fire hose, makes it a little faster and a little less bandwidth intense.

wolverine2710 · Sep 6, 2014

kfsone said:
v3.5 checked in: now using the JSON compressed stream for the fire hose, makes it a little faster and a little less bandwidth intense.

Thanks again for your response. Time to go to bed and get my flight tomorrow. Cu in 2 weeks. VERY curious how TD will look by then ;-)

rikkles · Sep 8, 2014

Hello kfsone, something seems a bit off:

trade.py run --sh type6 --cr 220000 --ju 1 --hops 3 --from louis

unspecified hops 3, numHops 3, viaStations 0

Code:

LHS 3262 Louis De Lacaille Prospect -> NANG TA-KHIAN Hay Point:
 >-> At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Coltan,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Lithium,
  +  At LHS 3262/Louis De Lacaille Prospect, Buy: 100 x Auto-Fabricators,
 <-< Hay Point gaining 207,700cr => 427,700cr total

Why would I want to grab 100 coltan then come back to the origin? Plus, considering there is no coltan at Louis de Lacaille, it won't really work. The 2nd route from LHS 3262 that has auto-fabricators makes sense.

I am running the EMDN price updater in the background, so in theory it should have the latest prices. Also at the same time, Slopey's tool gives me correct answers for single routes.

And when I look at the updated prices file, in the Louis De Lacaille section I see:

Code:

      Beryllium                8025      0   2014-09-08 09:09:39
      Gallium                  4990      0   2014-09-08 09:09:39
      Coltan                    940    941   2014-09-06 08:55:02

The Coltan line is clearly old and hasn't been updated. Looks like stuff is added and updated, but not deleted.

Zillion · Sep 8, 2014

I just downloaded latest version (via download 3.4).

Code:

trade.py run --cr 1500000 --ship type6 --from Baker --hops 10 --jumps-
per 1
unspecified hops 10, numHops 10, viaStations 0
LFT 880 Baker Platform -> BOLG Moxon's Mojo:
 >-> At LFT 880/Baker Platform, Buy: 100 x Gallium,
  +  At NANG TA-KHIAN/Hay Point, Buy: 100 x Personal Weapons,
  +  At ERANIN/Azeban City, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Superconductors,
  +  At AULIN/Aulin Enterprise, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Superconductors,
  +  At AULIN/Aulin Enterprise, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Superconductors,
  +  At AULIN/Aulin Enterprise, Buy: 100 x Resonating Separators,
  +  At ACIHAUT/Cuffey Plant, Buy: 100 x Superconductors,
 <-< Moxon's Mojo gaining 1.217.500cr => 2.717.500cr total

Sounds good, but.. I cant buy Resonating Separators @ Eranin, and there are no Superconductors for sale @ Acihaut...

?

deadevilgrounds · Sep 8, 2014

I can confirm that I get the same issue with the latest version.
Products that are not sold show up in the "prices" file for multiple stations.

kfsone · Sep 9, 2014

Yeah - I'm working on cleaning those out, it appears they originate from people docking at a station with a load that the station doesn't buy. The load shows up in the commodities screen, and EMDN scrapes it and transmits it.

kfsone · Sep 9, 2014

rikkles said:
Code:

Beryllium 8025 0 2014-09-08 09:09:39 Gallium 4990 0 2014-09-08 09:09:39 Coltan 940 941 2014-09-06 08:55:02

The Coltan line is clearly old and hasn't been updated. Looks like stuff is added and updated, but not deleted.

This is correct, I don't have any kind of auto-cutoff at the moment, because up until a few days ago all the data was hand-entered. The tail end of this week I started allowing updates from EMDN. Right now I'm trying to figure out how to detect those lines where's it not a "real" entry but one of those UI artefacts.

-Oliver

kfsone · Sep 12, 2014

v3.6 should do a better job of filtering those noise items out now, resultingly the db size has dropped from 1350 items to 1240.

I've also made it take stock levels into account when they are present (so if you're hand-building your own DB, you won't be affected, if you're pulling from EMDN it'll take into account that it last saw only X of an item available).

Code:

v3.6 Sep 12/2014
  Added DB support for tracking item stock/demand levels,
  TradeCalc will now factor stock levels when present,
  Minor performance/memory tweak
  emdn-tap:
    Now accepts --warn-to argument,
    Applies filters to what data it will accept,
    Records item stock/demand levels to the DB

wolverine2710 · Sep 13, 2014

wolverine2710 said:
Thanks for the fast response, really appreciated. Played with it a bit this morning. Indeed its AMAZINGLY FAST. Much faster then the routines used by Slopey in its BPC. Don't have time to fully test it and compare it to BPC. Tomorrow I'm flying - holiday.

I've figured out how to do a ANY to ANY by leaving out from and to. I think I figured out a roundtrip like the BPC by setting --from and --to the same station. Though I get different values then with BPC - could be prices not updated yet. Haven't figured out to do an ANY to ANY with roundtrip like BPC. Am I missing something or?

With --routes you get more suggestions. But it looks as if that are suggestions for stations. Setting --from and --to different stations I get a traderoute but that is only for one commodity - best route I suspect. BPC shows multiple commodities with different profits. Is that possible with TD also?

Speaking about roundtrips. Given the amazing speed of TD. Would it be possible to find the best roundtrips, not for two but for for example 3-4 stations. So it involves buying/selling at multiple stations to get the best trade route.

About speed: Its amazing, and you stated with some tweaking it could be even be much better/faster. Should TD ever becoming not fast enough, you could always go the MT route - if you deem that needed.

As I stated before: I can see this become the engine of choice for GUI builders - if optional JSON/XML output is implemented. Is that something you would like to pursue/see happen?

Still on holiday but finally found Internet access. Will be back in a week. Perhaps you have missed my post. Would it be possible for you to answer my questions? That would be greatly appreciated.

rikkles · Sep 13, 2014

kfsone said:
v3.6 should do a better job of filtering those noise items out now, resultingly the db size has dropped from 1350 items to 1240.

I've also made it take stock levels into account when they are present (so if you're hand-building your own DB, you won't be affected, if you're pulling from EMDN it'll take into account that it last saw only X of an item available

TM is well structured, fast and the CLI is exactly what one would want, but unfortunately the underlying EMDN has a lot of garbage, making the tool relatively useless unless one updates the DB manually. I just tested again and landed in Dhezhunov(sp?), sold my fish and was looking for tea that just didn't exist. I checked EMDN directly and it exceedingly wrong for that station.

wolverine2710 · Sep 13, 2014

rikkles said:
TM is well structured, fast and the CLI is exactly what one would want, but unfortunately the underlying EMDN has a lot of garbage, making the tool relatively useless unless one updates the DB manually. I just tested again and landed in Dhezhunov(sp?), sold my fish and was looking for tea that just didn't exist. I checked EMDN directly and it exceedingly wrong for that station.

Have you checked the prices with Slopeys BPC which also uses EMDN. Curious if BPC also gives the same issues you reported?

rikkles · Sep 13, 2014

wolverine2710 said:
Have you checked the prices with Slopeys BPC which also uses EMDN. Curious if BPC also gives the same issues you reported?

Yes of course. That's why I said

I checked EMDN directly

. It's not a TD problem but an EMDN data problem.

Lazerguns · Sep 13, 2014

kfsone said:
I knocked up a small EMDN module in python to do this, grab version 3.3. I've also added support for the internal game names (table is AltItemName in the DB).

-Oliver

That's great news! Thanks.
I had prepared my patch for you for this, but I got caught on compiling zeromq for windows in cygwin and couldn't dedicated alot of time to this.
This pesky real-life things

Will check it out tomorrow.

Lazerguns · Sep 14, 2014

I notice emdn-tap.py is sitting at 100% cpu all the time (well, 25% for one core )
Is this expected?

jojje · Sep 14, 2014

Lazerguns said:
I notice emdn-tap.py is sitting at 100% cpu all the time (well, 25% for one core )
Is this expected?

Expected, yes, as that's how it's currently implemented, desirable, no.

The root cause is that the firehose code polls for zmq messages in a tight loop, meaning more aggressively than it needs to.
I can see two obvious ways to address it.

Explicitly yield the CPU when there's nothing much going on.
Use an event mechanism instead of polling, and let zmq call the message processing function when it actually has some data to be processed. I.e. leverage the IOLoop API.

A patch for the former approach:

Code:

diff --git a/emdn/firehose.py b/emdn/firehose.py
index b381f0b..2f4ade8 100644
--- a/emdn/firehose.py
+++ b/emdn/firehose.py
@@ -73,7 +73,14 @@ class Firehose(object):
         else:
             def _poll(timeout):
                 #### TODO: Use a ZMQ Poller so we can catch ctrl-c
-                return self.__socket.poll(timeout)
+                result = self.__socket.poll(timeout)
+
+                # No events were ready, so throttle down a bit and
+                # let the CPU take a breather.
+                if result == 0:
+                    time.sleep(0.01)
+                return result
+
             def _read(nonBlocking=False):
                 flags = zmq.NOBLOCK if nonBlocking else 0
                 compressed = self.__socket.recv(flags)

This fix will make the firehose code take short naps (10ms) when the message queue is empty, naps short enough to drop my CPU usage to 0 when nothing is going on.

preynolds2 · Sep 15, 2014

Do I need to be running anything other than python ?

ex.. marketdump etc? (if so.. how do I use the two together or update trade.py using marketdump?)

Also..

is there something better to use other than running a "cmd" window?

Hard to see the output.

AND..

How are the market prices updated? do I do this manually?

In-Development TradeDangerous: power-user trade optimizer

wolverine2710

Tutorial & Guide Writer

wolverine2710

Tutorial & Guide Writer

Net-burst

kfsone

kfsone

wolverine2710

Tutorial & Guide Writer

rikkles

Zillion

deadevilgrounds

kfsone

kfsone

kfsone

wolverine2710

Tutorial & Guide Writer

rikkles

wolverine2710

Tutorial & Guide Writer

rikkles

Lazerguns

Lazerguns

jojje

preynolds2