In-Development TradeDangerous: power-user trade optimizer

kfsone

kfsone
DRY411S said:
So are you going down a C++ route now?
Click to expand...

Not at this point, but if I do flesh out that project it will only be the back-end of TD.

DRY411S said:
Will TD continue to run on Mac, Windoze and Linux?
Click to expand...

If that project became anything, it would simply be a replacement for the "TradeDB()" part of the current python project that you'd leave running in the background as a service backed by SQLite for persistent storage. Or I've been wondering about changing to MySQL/Postgres with spatial extensions for the back-end, but not seriously.

So TD and its cross-platform are safe :)
 
Last edited:

Avi0013

Avi0013
kfsone said:
Issues that get reported on the issue tracker (http://kfs.org/td/issues) tend to get fixed faster. That said, I fixed the numpy issue earlier today and I just pushed my fix for --age now.
Click to expand...

What about plain old questions about the code, that isn't a bug, proposal, task, or enhancement :D

kfsone said:
The key feature of numpy is vectorization, and that can be leveraged if we store data the right way - for instance, refactor the buy/sell lists the right way and you could vectorize the few math operations we need doing there. The question being how much it will cost to marshal them into a usefully vectorizable layout.
Click to expand...

Not being an expert in Python, and being a total ignoramus when it comes to OOP, I'm not completely clear about the code which is calculating the costs, but given a particular item, if you extract a vector of costs and sale prices by station, the maximal profit should be able to be calculated by creating the distance matrix and finding the largest value for which cost < price. I don't know if that is any faster, and you may have already thought of it for that matter :)

- - - Updated - - -

kfsone said:
and then there's all the blas/lapack stuff to install to get the high-end perf that would make it rock, it just gets nasty
Click to expand...
For those of us on Windows, there is always http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
 

Razem

R
This is more of a fyi;

Ran this on OS X
NUMPY=1 trade.py run --from "BES/Stephonson Terminal" --detail --progress --credits 25000000 --capacity 240 --ly-per 12.54 --hops 8 --jumps 4 -vv

and got this


/Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python: can't open file 'NUMPY=1': [Errno 2] No such file or directory

then further attempts to run trade.py would output nothing. Python 3.4.3 seemed to be still working, just TD wasn't doing anything, at least that I could determine. After puttering around I reset my TD Repository on Git and that resolved what ever my issue was.

hehe you inspire me to learn Python.
 

Avi0013

Avi0013
Razem said:
Try without the spaces in between the argument NUMPY=1
Click to expand...

Code: 
G:\Elite\TD>NUMPY=1 trade.py run -vvv --summary --avoid slaves,imperial --ly 18.14 --insurance 550k --cap 116 --credits
3768464 --from Wolf10/Town --hops 3 --jumps 3
'NUMPY' is not recognized as an internal or external command,
operable program or batch file.
 

kfsone

kfsone
Razem said:
This is more of a fyi;

Ran this on OS X

NUMPY=1 trade.py run --from "BES/Stephonson Terminal" --detail --progress --credits 25000000 --capacity 240 --ly-per 12.54 --hops 8 --jumps 4 -vv

and got this
/Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python: can't open file 'NUMPY=1': [Errno 2] No such file or directory

then further attempts to run trade.py would output nothing. Python 3.4.3 seemed to be still working, just TD wasn't doing anything, at least that I could determine. After puttering around I reset my TD Repository on Git and that resolved what ever my issue was.
Click to expand...

Ah - it only works from the command line:

Code: 
you@bash$ NUMPY=1 python trade.py --help

But don't sweat it - As yet, TD isn't functionally using numpy, that's why it's firewalled in this awkward way. It just turns on some hooks and adds a numpy copy of every star's position (extra work, basically).

As soon as there's something to test, it'll be more easily accessible :)

Razem said:
hehe you inspire me to learn Python.
Click to expand...

Three tricks to learning Python:

1. Python has a "help" command which works on just about everything, e.g. "help(print)" or "import tradedb; help(tradedb)". Use it relentlessly.
2. Install and use "ipython" ("ipython qtconsole" is a much cleaner interface) or use Python's batteries-included GUI, idle, and then make Python your - I almost always have a python prompt handy and I type stuff into it all the time. Couple with #1 and the massive number of modules available for python and you have awesome power.
3. Use #1 and #2 whenever you encounter something small and possibly useful in your daily computing life, generously rely on stackoverflow and google to assist you.

- - - Updated - - -

Avi0013 said:
Being on Windows, trying to pass NUMPY = 1 before the call to trade.py just gives me an error :(
Click to expand...

On Windows:

Code: 
C:\Dev\Trade\> set NUMPY=1
C:\Dev\Trade\> set NUMPY
NUMPY=1
C:\Dev\Trade\> trade.py ...
C:\Dev\Trade\> set NUMPY=

In Bash:

Code: 
user@box:~/trade$ export NUMPY=1
user@box:~/trade$ trade.py ...

or

user@box:~/trade$ NUMPY=1 python trade.py ...

or

user@box:~/trade$ NUMPY=1 trade.py ...
 

Minyeni

M
kfsone said:
But I've also been pondering whether I would redo TD In C#, especially now that .NET is open source. The perf would be better than Python, there's a common GUI, and I've written a few small things in it (https://bitbucket.org/kfsone/houseomatic).
Click to expand...
If you built the thing as a web service plus client, hosting the service in the cloud would be pretty easy. I have some unused developer credit for Microsoft Azure; standing up a replicated service in the cloud wouldn't be hard and would get pretty good performance. (And since you have to have network connectivity to play ED at all, asking someone to have a live internet connection to use TD isn't a big stretch.)
I'd love to see (and help with) a rewrite to C#. Heck, I'd even be okay with C++, since that's my current day-job language. I'm vastly more comfortable in C++ than I am in Python, and unlike Python, C++ actually shows program flow using braces instead of indentation, the way the good Lord intended it to be shown.
 

kfsone

kfsone
Minyeni said:
Heck, I'd even be okay with C++, since that's my current day-job language.
Click to expand...

An aside: Have you found anything good on 11/14/17? Most of what I've picked up sweats the small stuff - lambdas, rvalues, reference collapsing, but coverage of library changes in 11 and 14 seems to be vague handwavy and "well, don't you know about that?" e.g. traits, alias_t<T> vs traits<T>::type, etc.

Miyeni said:
I'm vastly more comfortable in C++ than I am in Python, and unlike Python, C++ actually shows program flow using braces instead of indentation, the way the good Lord intended it to be shown.
Click to expand...

I've done more than my share of Javascript, Lua, Perl, etc. But Python really does lend itself well to interactive iteration - literally building stuff in the REPL so that you've seen exactly what it does as you put it together. That's how it drew me in. I broke myself of the bracket dependency with this:

Code: 
if (foo.bar == x)
#{
    print("It hurts, it hurts...");
#}

Plus the ease with which a good Python install lets you do something like:

Code: 
>>> pip.main(['install', '--upgrade', 'requests'])
>>> import requests
>>> help(requests)
>>> r = requests.get("http://kfs.org/td/thread")
>>> help(r)
>>> r.headers, r.history, r.status_code

I just found working so interactively with emerging code was a real buzz compared to the usual need to fill out all the boiler plate, compile, fix, etc, before you could get close to seeing what it was doing. That's why Python has inspired things like Skulpt (scroll down and see 'trinket' and run the code). At some point, the "fun" of the early tangibility of a piece of code won out over my revulsion at white space control, but then ... I did write a white-space delimited MUD language in the 80s.
 

Minyeni

M
kfsone said:
An aside: Have you found anything good on 11/14/17? Most of what I've picked up sweats the small stuff - lambdas, rvalues, reference collapsing, but coverage of library changes in 11 and 14 seems to be vague handwavy and "well, don't you know about that?" e.g. traits, alias_t<T> vs traits<T>::type, etc.
Click to expand...
  • Biggest improvement in C++11 is move semantics. With proper thought applied in advance, and some judicious refactoring done after analysis, one can remove a whole lot of copying from the proceedings. The fact that the library containers handle move/rvalue/&& makes it all work.
  • The emplace container methods also eliminate a lot of copying.
  • The container iterator syntax [ for (const auto & foo : my_container) { ... } ] really makes iterators natural; code is a whole lot cleaner.
  • You already mentioned lambdas. Combined with <functional>, I find state machines and similar constructs to be easily represented as declaratively-initialized data structures; my state table actually looks like a table, and it's implemented in an std::map<state_t, std::function<blahblah> > with some trivial driving code.
  • The explicit-size datatypes (int32_t etc.) significantly improve code portability between Linux and Windows.
  • Big chunks of Boost appeared in the C++11 library, which means deployment is much easier.
  • Combine asio and timers and something like Microsoft's free cross-platform Casablanca library (async REST API), and stupid-fast client code for web services becomes pretty simple to write.
  • Unicode support in C++14 will be a blessing, once everyone's implemented it.
  • I'm not a big metaprogramming expert, but I'm all agog at what the brainiacs who understand that stuff can do.

All I can tell you is that the code I write under C++11 is more robust and bug-free than the stuff I wrote under C++98. I spend far less time screwing around with new/delete; the standard "smart pointers" really help with object lifetime management.

My biggest disappointment: unordered_map<K, V> just doesn't automagically handle an arbitrary type K which is a pair or tuple of simpler types for which std::hash is already defined.
Code: 
If K is std::pair<T1, T2> then hash(K) is as simple as xor(rotate_left(hash(T1), 16), hash(T2))
. It's just not that hard to have the library figure out the correct hashing function when it's trivially composable. (Ditto unordered_set, of course.)
 

kfsone

kfsone
Minyeni said:
  • Biggest improvement in C++11 is move semantics. With proper thought applied in advance, and some judicious refactoring done after analysis, one can remove a whole lot of copying from the proceedings. The fact that the library containers handle move/rvalue/&& makes it all work.
  • The emplace container methods also eliminate a lot of copying.
  • The container iterator syntax [ for (const auto & foo : my_container) { ... } ] really makes iterators natural; code is a whole lot cleaner.
  • You already mentioned lambdas. Combined with <functional>, I find state machines and similar constructs to be easily represented as declaratively-initialized data structures; my state table actually looks like a table, and it's implemented in an std::map<state_t, std::function<blahblah> > with some trivial driving code.
  • The explicit-size datatypes (int32_t etc.) significantly improve code portability between Linux and Windows.
  • Big chunks of Boost appeared in the C++11 library, which means deployment is much easier.
  • Combine asio and timers and something like Microsoft's free cross-platform Casablanca library (async REST API), and stupid-fast client code for web services becomes pretty simple to write.
  • Unicode support in C++14 will be a blessing, once everyone's implemented it.
  • I'm not a big metaprogramming expert, but I'm all agog at what the brainiacs who understand that stuff can do.

All I can tell you is that the code I write under C++11 is more robust and bug-free than the stuff I wrote under C++98. I spend far less time screwing around with new/delete; the standard "smart pointers" really help with object lifetime management.

My biggest disappointment: unordered_map<K, V> just doesn't automagically handle an arbitrary type K which is a pair or tuple of simpler types for which std::hash is already defined.
Code: 
If K is std::pair<T1, T2> then hash(K) is as simple as xor(rotate_left(hash(T1), 16), hash(T2))
. It's just not that hard to have the library figure out the correct hashing function when it's trivially composable. (Ditto unordered_set, of course.)
Click to expand...

I started using <cstdint> as soon as GCC had it and what move semantics deliver in win is eroded by what they cost in template shenanigans and runtime code inspection -- part of that is just feature maturity in the compilers, but you have to learn about universal references (these are not the rvalues&& you are looking for), the flaws in forwarding ("{...}"), so forth.

The sharp edges on shared pointers are going to trade memory leaks for hard bugs.

And so many of the new library features are badly named, dizzyingly verbose or not generally well understood/documented.

My sense is that they didn't sufficiently iterate some of the features. Consider the following piece of code:

Code: 
#include <iostream>
#include <vector>
#include <map>

struct Thing {
  Thing(const char* const name_) : m_name(name_) { std::cout << "created " << m_name << '\n'; }
  Thing(const Thing& rhs_) : m_name(rhs_.m_name) { std::cout << "copied " << m_name << '\n'; }
  Thing(Thing&& rhs_) : m_name(rhs_.m_name) { std::cout << "moved " << m_name << '\n'; rhs_.m_name = nullptr; }
  Thing& operator = (const Thing& rhs_) { m_name = rhs_.m_name; std::cout << "=& " << m_name; return *this; }
  Thing& operator = (Thing&& rhs_) { m_name = rhs_.m_name; std::cout << "=&& " << m_name; return *this; }
  ~Thing() { std::cout << "dtor(" << (m_name ? m_name : "null") << ")\n"; }

  const char* m_name;
};
int main() {
  std::map<int64_t, std::vector<Thing>> myMap { { 1, { "Hello", "World" } }, { 2, { "Eat", "Pizza" } } };
  std::cout << "for\n";
  for (auto kv : myMap) {
    std::cout << "for " << kv.first << " -> " << kv.second[0].m_name << "\n";
  }
  std::cout << "endfor\n";
}

You spotted the missing ref-qualifier on the auto, but did you spot the other bug? If not, take a look at the output: http://ideone.com/EzbNCp

Code: 
created Hello
created World
copied Hello
copied World
copied Hello
copied World
created Eat
created Pizza
copied Eat
copied Pizza
copied Eat
copied Pizza
copied Hello
copied World
copied Eat
copied Pizza
dtor(Eat)
dtor(Pizza)
dtor(Hello)
dtor(World)
dtor(Eat)
dtor(Pizza)
dtor(Pizza)
dtor(Eat)
dtor(Hello)
dtor(World)
dtor(World)
dtor(Hello)
for

Can you figure out why we made all these copies?
 
Last edited:

kfsone

kfsone
Code: 
v7.0.3 May 09 2015
. (kfsone) "run" command:
    - Added "--show-jumps" (aka -J)
    - Jumps are no-longer shown by default,
    - Request #233 Jumps now include distance
    - If start and end station of a hop are in the same system,
      display "Supercruise to ..." instead of a jump
    - When a hop involves multiple jumps using --show-jumps, it will
      tell you the direct and total distances,
. (kfsone) Revamped the intro of the README.md (http://kfs.org/td/source)
. (gazelle) Corrected unicode system names (Argetlamh etc)
 

Serpentstar

S
I believe something has gotten broken again from Maddavo. I have not been able to load systems again. So I wiped my existing folder, pulled the latest and am getting the following:

e:\elite\tradedangerous>trade.py import --plug maddavo --opt stations --opt=systems
NOTE: Importing Systems
NOTE: Traceback (most recent call last):
File "E:\elite\tradedangerous\trade.py", line 102, in <module>
main(sys.argv)
File "E:\elite\tradedangerous\trade.py", line 76, in main
results = cmdenv.run(tdb)
File "E:\elite\tradedangerous\commands\commandenv.py", line 80, in run
return self._cmd.run(results, self, tdb)
File "E:\elite\tradedangerous\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "E:\elite\tradedangerous\plugins\maddavo_plug.py", line 709, in run
self.import_systems()
File "E:\elite\tradedangerous\plugins\maddavo_plug.py", line 302, in import_systems
commit=False,
File "E:\elite\tradedangerous\tradedb.py", line 812, in addLocalSystem
ID, name, x, y, z
File "E:\elite\tradedangerous\tradeenv.py", line 61, in __NOTE_ENABLED
print("NOTE:", outText.format(*args, **kwargs), file=file)
File "E:\Python34\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xcd' in position 26: character maps to <undefined>

It also doesn't matter if I use this format either:
e:\elite\tradedangerous>trade.py import --plug maddavo --opt stations,systems

Has something changed as I have not been able to pull systems for several days now?

I also know this may not exactly be a TD thing, but I wanted to verify and see if anyone else was seeing this.
 
Last edited:

kfsone

kfsone
I saw this briefly yesterday, was looking at it and then it went away. The problem is happening when td tries to print a utf8 name in a non-utf dos window. You can scrape past it by running with -q until I figure a good way to work around it. V
 

Minyeni

M
Believe me, it's far from perfect, but it makes the code I used to write with C++98 easier to write correctly. I try to stay out of dark and dusty corners of the language; there lies madness. I am painfully acquainted with the limitations of forwarders. I think I know where the extra set of construction operations takes place (building the std::pair<int64_t, vector<Thing>> that will get copied into the map). What I'm unclear on is precisely why the move operations aren't used in going from the (temporary) initializer to the fully initialized myMap object.

I won't use a feature unless both GCC and Clang support it. (And once Visual Studio 2015 ships, I'll restrict myself to what all three support.) That keeps me out of the places where dragons lurk.
 

kfsone

kfsone
v7.1.0 is up: I changed the minor number because although the changes were small, they are internally significant (TD works extra hard to keep everything utf-8 now).

Basically: Windows console uses a local characterset by default, TD and Python use utf-8 by default. utf-8 uses "ascii" for the first 127 characters and then it uses trickery to support the other 80,000+ characters of unicode. The result is that, usually, 79,745+ of the unicode characters can't be printed in a windows terminal. So when TD is trying to tell you that the ED systems with unicode characters have been removed, Python throws its hands up and says "can't translate that".

You could work around it either by using the "-q" argument to TD so that it removed the problem characters quietly, or you could issue the dos-command "chcp.com 65001".

But now I've told TD that it should go business as usual but added special-cases to some of the code that says "if python blows up when you try to print this, try replacing funky characters with '?' and try again".
 
You must log in or register to reply here.
Back
Top Bottom