In-Development TradeDangerous: power-user trade optimizer

[jojje]

And with regards to speed, it's lacking there, I need to do another optimize pass.

Except the excessive polling, it seems plenty fast to me.

Before you waste time on pre-optimization p) take a look at the call graph for your code, after I opened the polling valve a bit with the patch above.

Polling is still where most time is spend for obvious reasons (waiting for data). The tradedb.* processing logic bit might need some looking into in the future, but for now it seems plenty fast. Not something I'd waste a second on if it'd been my project.

 

[preynolds2]

+ At KERIES/Derrickson's Escape, Buy: 100 x Bertrandite (@1878cr),
| Keries -> LHS 2887


---
there is nothing to buy at Derrickson's Escape by the name of "Bertrandite"


EDIT:
also other trade "helpers" tell me to buy commodities from this station that are not currently listed

 

[Net-burst]

+ At KERIES/Derrickson's Escape, Buy: 100 x Bertrandite (@1878cr),
| Keries -> LHS 2887


---
there is nothing to buy at Derrickson's Escape by the name of "Bertrandite"


EDIT:
also other trade "helpers" tell me to buy commodities from this station that are not currently listed

Developers messed with the market some time ago and very heavily changed profitability. They also removed goods here and there.

 

[kfsone]

That's great news! Thanks.
I had prepared my patch for you for this, but I got caught on compiling zeromq for windows in cygwin and couldn't dedicated alot of time to this.
This pesky real-life things
Will check it out tomorrow.

Eep - haven't done any small fiddlies with ZeroMQ since this, and I think the reason it's being used more widely with Python et al is that it's as simple as "pip install pyzmq".

-Oliver

 

[kfsone]

Do I need to be running anything other than python ?

ex.. marketdump etc? (if so.. how do I use the two together or update trade.py using marketdump?)


Also..

is there something better to use other than running a "cmd" window?

Hard to see the output.


AND..

How are the market prices updated? do I do this manually?

* Updating the prices:

There are two approaches you can take. You can either manually track prices as you move between stations, by using the command line:

trade.py update Aulin

and specify a way to edit the data:

trade.py update --edit "c:\program files\thingy\thingy.exe" aulin

if you want to specify an editor,

trade.py update --sublime aulin
trade.py update --notepad aulin

if you want to use either "Sublime Text 3" (my personal current favorite text editor across platforms) or notepad (I leave guessing which -- switch is which editor as an exercise to the reader).

or

You can gamble on the EMDN data. This is publicly sourced data that may be subject to errors, poisoning and other problems. That's what emdn-tap.py is for. Basically, use at your own risk.

* Do I need to be running marketdump?

That's part of EMDN, and if you want to participate in it, I would recommend also running MarketDump to send the prices you discover back into the network.

 

[kfsone]

Except the excessive polling, it seems plenty fast to me.

Before you waste time on pre-optimization p) take a look at the call graph for your code, after I opened the polling valve a bit with the patch above.

Polling is still where most time is spend for obvious reasons (waiting for data). The tradedb.* processing logic bit might need some looking into in the future, but for now it seems plenty fast. Not something I'd waste a second on if it'd been my project.

Short version: Thanks for catching that, I've checked in a fix.

Background:

"poll" in this context means "wait until timeout for something to happen", except I was passing it "0" instead of "None", which pyzmq interpreted as meaning "no timeout, return immediately". What I wanted was the behavior other languages get when you pass a negative value to poll()

http://linux.die.net/man/2/poll

The timeout argument specifies the minimum number of milliseconds that poll() will block. (This interval will be rounded up to the system clock granularity, and kernel scheduling delays mean that the blocking interval may overrun by a small amount.) Specifying a negative value in timeout means an infinite timeout. Specifying a timeout of zero causes poll() to return immediately, even if no file descriptors are ready.

I'm still a little clumsy with 0 vs None in python, sorry

 

[kfsone]

QUESTION: TradeDangerous.prices file

Newer users of TD are understandably confused when it shows them out of date prices. I'm trying to figure out a way to make it clearer that TD is the calculator, the data itself has to be populated (by hand or something like EMDN).

Part of this is perhaps that I have been checking in my copy of TradeDangerous.prices, much to the annoyance of anyone using 'git' to keep their code up to date.

I think I'm going to make the app detect no .db + no .prices and give a "Greetings, Commander!" introduction and provide a boot-strap file that I won't change very often, and instruct them that they need to get it fed with data (e.g. by using emdn-tap.py).

Then I can remove the .prices file from the repository and stop checking in changes.

Any thoughts?

 

[jojje]

What I wanted was the behavior other languages get when you pass a negative value to poll()

Won't that cause the process to block indefinitely, and ignore interrupt signals?

You mentioned in the code comment that you were considering using a poller to allow CTRL-C signals so the user could abort the program. Now since you did have a no-wait-polling implementation in a loop, I just figured you opted to solve the problem of "zombie blocking" by not handing control over to native code, and consequently the OS, but instead retain that control in "python land" through the loop, thus being able to respond to interrupts.

When I've looked at other python code, it seems that for Unix/Linux python is able to interrupt the zmq recv blocking call, but on Windows it is not. The workaround I've seen for Windows (to handle CTRL-C) is to put the recv call in a separate daemon thread, and have the main thread run an ordinary sleep loop.

All I'm saying is I'm concerned that the user won't be able to quit the script as expected with CTRL-C if you hand control over to a blocking sys call.

Let me know when you've committed your change, so I can check it out on Windows this evening (I'm currently on my macbook, so can't test it right now)

I'm still a little clumsy with 0 vs None in python, sorry

Completely understandable. Takes a while to get a grip on each language's way of representing nothing (0, NULL, null, nil, None, undefined ....). And even when your brain does know, you fingers seem to ignore it anyway, so I also tend to slip up a lot on the details when polyglotting too much

 

[jojje]

Newer users of TD are understandably confused when it shows them out of date prices. I'm trying to figure out a way to make it clearer that TD is the calculator, the data itself has to be populated (by hand or something like EMDN).
...
Any thoughts?

I think it's a splendid idea.
Always good to have a quick guide to get going initially. Better yet is when it's possible to avoid guides altogether, by making the software realize the most common use case automatically (the "apple and frontier way", making decisions regarding reasonable defaults).

E.g. if the software doesn't find a default sqlite-db file, then create it and start populating it with data from emdn, which IIRC emdn-tap already does. Haven't looked into the actual trade.py script yet, but it could leverage a similar check, and just spawn off emdn-tap.py in a separate thread/process if it detects the default sqlite-db is missing. Anyways, just a though for making the boot-strapping easy on new users.

 

[rikkles]

When I've looked at other python code, it seems that for Unix/Linux python is able to interrupt the zmq recv blocking call, but on Windows it is not. The workaround I've seen for Windows (to handle CTRL-C) is to put the recv call in a separate daemon thread, and have the main thread run an ordinary sleep loop.

All I'm saying is I'm concerned that the user won't be able to quit the script as expected with CTRL-C if you hand control over to a blocking sys call.

Let me know when you've committed your change, so I can check it out on Windows this evening (I'm currently on my macbook, so can't test it right now)

I've just checked it out. Indeed on Windows CTRL-C is now completely ignored. Now it's not that big a deal, as you can just close the shell window and it'll kill all the child processes.

 

[rikkles]

Any thoughts?

What would be useful is a command to completely wipe out the prices and start from scratch. I suggest putting it in the emdb-tap.py as an argument, which would help debugging emdb issues. It would start with a completely blank slate, and keep on adding data from emdb as it comes in, so you'd know as a user what emdb has been doing. And then you'd be able to determine if emdb is useful for you (which it might well not be).

 

[kfsone]

Won't that cause the process to block indefinitely, and ignore interrupt signals?

You mentioned in the code comment that you were considering using a poller to allow CTRL-C signals so the user could abort the program. Now since you did have a no-wait-polling implementation in a loop, I just figured you opted to solve the problem of "zombie blocking" by not handing control over to native code, and consequently the OS, but instead retain that control in "python land" through the loop, thus being able to respond to interrupts.

Nope - in fact, kind of the opposite. PyZMQ calls sys.exit instead of re-raising the exception. Using a poller was supposed to fix it but I tried it (I created a poller, registered the socket and then simply set self.poll = self.__poller.poll) and it had the same bug. It should be fixed in a future version of PyZMQ.

It just means that ctrl-c exits without saving the current batch of records. So I've just made the auto-commit thing more accurate and you can ctrl-c at the loss of only the most recent uncommitted changes.

All I'm saying is I'm concerned that the user won't be able to quit the script as expected with CTRL-C if you hand control over to a blocking sys call.

I wouldn't want that either

Let me know when you've committed your change, so I can check it out on Windows this evening (I'm currently on my macbook, so can't test it right now)

I'm only running it under Windows thus far (moving house and all).

That said - I've checked it in now, along with a bit of a cleanup for v3.7

v3.7 Sep 15/2014
  Fixed excessive CPU usage in emdn-tap.py (ty, jojje)
  Added 'cleanup' command to help remove bad data from emdn
    EMDN isn't foolproof, and sometimes receives invalid or
    deliberately poisoned data. The best way to detect this
    is currently to look for prices that are somewhat older
    than the rest of the information received for the same
    station. This command checks for those records and
    removes them.
  emdn-tap now tries harder to honor commit intervals.
API Changes:
  makeSubParser now takes an epilog

 

[kfsone]

What would be useful is a command to completely wipe out the prices and start from scratch. I suggest putting it in the emdb-tap.py as an argument, which would help debugging emdb issues. It would start with a completely blank slate, and keep on adding data from emdb as it comes in, so you'd know as a user what emdb has been doing. And then you'd be able to determine if emdb is useful for you (which it might well not be).

Yeah, I can do that. I'm probably going to make it so that you operate emdn-tap as a subcommand of trade fairly soon.

In the meantime, anyone wanting to do the above, you can do it by hand with the following commands (presented in batchable form):

echo Resetting TDdb
del data\tradedangerous.db
echo >data\tradedangerous.prices
echo Starting EMDN tap
emdn-tap.py -vv --commit 5

or with git bash:

#! /usr/bin/env bash
echo "* Resetting TDdb"
rm data/t*.db
echo >data/t*.prices
echo "* Commence to tapping"
emdn-tap.py -vv --commit 5

 

[kfsone]

I've just checked it out. Indeed on Windows CTRL-C is now completely ignored. Now it's not that big a deal, as you can just close the shell window and it'll kill all the child processes.

Hrm. I'm running it through the bash that comes with git (git bash).

Ah - ok - I've tried it under the command prompt.

You can either press Ctrl-Break to interrupt it (ick) or grab version 3.7 and it'll pick up the ctrl-c at the next auto commit tick.

I'll re-read jojje's latest post in the morning

 

[preynolds2]

* Updating the prices:

There are two approaches you can take. You can either manually track prices as you move between stations, by using the command line:

trade.py update Aulin

Where does this have to be done? Inside the station? Inside the station when you open the commodity market?

and specify a way to edit the data:

trade.py update --edit "c:\program files\thingy\thingy.exe" aulin

I do not understand this option.. Is this grabbing data from another source?? thingy?

if you want to specify an editor,

trade.py update --sublime aulin
trade.py update --notepad aulin

What does specifying an editor do?

if you want to use either "Sublime Text 3" (my personal current favorite text editor across platforms) or notepad (I leave guessing which -- switch is which editor as an exercise to the reader).

or

You can gamble on the EMDN data. This is publicly sourced data that may be subject to errors, poisoning and other problems. That's what emdn-tap.py is for. Basically, use at your own risk.

what does the command emdn-tap.py do?? Grabs data from emdn? In which case.. do I need to be running some sort of marketdump?

* Do I need to be running marketdump?

That's part of EMDN, and if you want to participate in it, I would recommend also running MarketDump to send the prices you discover back into the network.

I think it would be useful to have a list of commands and what they do..
Im trying to piece things together but I keep seeing commands that I had no clue about previously. For example, CTRL-C, this closes the program? Previously I thought just closing the cmd window would close out the program.

thanks


Next:
Tried running emdn-tap.py

Received:
ImportError: This module requires the ZeroMQ library to be installed. The easiest way to obtain this is to type: pip install pyzmq

Next:
tried what it said..
pip install pyzmq

Received error that "pip" is not a recognized command

 

[rikkles]

In the meantime, anyone wanting to do the above, you can do it by hand with the following commands (presented in batchable form):

echo Resetting TDdb
del data\tradedangerous.db
echo >data\tradedangerous.prices
echo Starting EMDN tap
emdn-tap.py -vv --commit 5

FYI that won't work. Your code expects a file with at least a certain amount of data in it. Just a station with a single price entry will work, anything else will crash the program because the try loop is pretty unforgiving. That won't matter when we pass in a "start from scratch" parameter.

 

[rikkles]

tried what it said..
pip install pyzmq

Received error that "pip" is not a recognized command

Yes. That's an idiotic Python issue. The "pip" script is generally at

C:\Python34\Scripts\pip.exe

You'll need to fully specify the path to run pip, or go inside the Scripts directory and run pip from there.

 

[preynolds2]

emdn-tap.py -v --commit 30 --warn 2>warnings.txt

Whats is this doing??
I can run this command alone and it starts telling me that it is committing records.

ED is not even running.
Where is the data coming from and going to?

Also..
Im not sure how to use it..

How long do I have to let it run?
Let it run all the time? And open a new window to use trade.py ??

 

[Net-burst]

emdn-tap.py -v --commit 30 --warn 2>warnings.txt

Whats is this doing??
I can run this command alone and it starts telling me that it is committing records.

ED is not even running.
Where is the data coming from and going to?

Also..
Im not sure how to use it..

How long do I have to let it run?
Let it run all the time? And open a new window to use trade.py ??

I wanted to write big post with instructions, but re-read OP and understood that it is all already there. So, just read it more carefully. Same goes for README in repository.

About EMDN: you can read in more detail about it here. To make things short: EMDN is a remote service that collects price data from multiple pilots that run MarketDump and give it to everyone who need it.

So, you should just run "emdn-tap.py -v" and keep it running for duration of your game session. It will update prices in real time. But ONLY if someone scrapes prices of given station. If you also want to make sure that every station you pass have most recent scrape, then you need to run MarketDump yourself.

 

[kfsone]

Also..

is there something better to use other than running a "cmd" window?

Hard to see the output.

If you have an X52, and you're going through a complex route, you can use the "--check --x52" options.