In-Development TradeDangerous: power-user trade optimizer

Ah! I see that now. I keep your page open in FireFox and I hadn't refreshed it.

So the ADD, EDIT and DELETE functions effectively render the corrections.csv method of updating redundant for the everyday user, if I understand correctly ?

Yeah, needed a user-friendly way to edit the Stations DB. The corrections.csv is mainly to support TD (future support) - not really for manual editing. When a station is deleted in the shared DB, via whatever method, the corrections.csv is updated. So TD can download and use this file to also correct local station DB derps. In this way shared station additions/updates come via station.csv and shared station deletions come via corrections.csv .
 
Oh. I sent the whole station.csv with the above lines in, then deleted those lines before the next buildcache.
i just presented those lines in the post that way for convenience.
Seems that the timestamp field is needed after all.

But I'm getting there.
I'll still use the station.csv update where relevant, especially for my local data, but I'll also go for the station ADD, EDIT and DELETE buttons at Maddavo's, if that makes sense.

All this "making sense" fair turns a man's head it does ...

I'd really recommend using Mad's page now over trying to manually submit the station file any more - especially since we're probably going to add some more fields in the near future and then it just gets to be a RPITA trying to do this by hand.

- - - - - Additional Content Posted / Auto Merge - - - - -

Just checked in a change to fix the "'set' does not support indexing" that some users were seeing.
 
Yeah, needed a user-friendly way to edit the Stations DB. The corrections.csv is mainly to support TD (future support) - not really for manual editing. When a station is deleted in the shared DB, via whatever method, the corrections.csv is updated. So TD can download and use this file to also correct local station DB derps. In this way shared station additions/updates come via station.csv and shared station deletions come via corrections.csv .

TD itself uses "corrections.py" from when there was significantly more manual curation. What we're trying to do is avoid including deletions in station.csv, although that might not actually be such a terrible deal.

Ok. I've changed the maddavo plugin so that it for a station deletion, you can just send

'System','Station',-1,,,,,

(as many commas as the .csv file requires, basically blank fields)

and you can also delete a system with

'System',,,,'DELETED',

However it's only the plugin that supports these, so people wouldn't be able to use these as drop-in .csv files.

That said, that's the long term plan - to move away from TD coming with .csv files and instead users always needing to import some data to their local cache and keeping a local, stable database.

In-case you're wondering why I haven't done this already ... the short version is that I have to figure out how to distribute schema data in order to support that.
 
Linux gui?

Hi there,

Running TD on Linux, some crontab lines to git pull new updates for the code and station, 2d and 3h data from maddavo's site. I use Steam overlay web browser and my Linux machine has Shellinabox installed, so ssh terminal on web page is heavy use :)

I really like use of command line and do various things about bash, poweshell, windows command prompt, etc. But to prevent reinventing the wheel, has anyone do any web-frontend for Linux, like using Django framework or substitute?
 
The station was not removed from the station.csv because the <trade.py station> command does not export the station file when removing a station. This does seem to be different behaviour from the --add option that does update the station.csv .

When you edit the station.csv the last-modified timestamp on the file is changed. TD detects that so any command causes a strict buildcache. To do a buildcache ignoring errors use:
trade.py buildcache -f -i

The system DITIBI is in the latest system file - if you cannot get it using the plugin (because of the buildcache issue) you can always get it manually on the webpage.

Thanks. I did manually download a new systemscsv to get around this. Lesson learned. :)

- - - - - Additional Content Posted / Auto Merge - - - - -

Ah - ok - that's one of two things;
(a) you don't have write access to the repository, only I do, you'd have to "fork" the project and submit your changes back to your branch and then click the "make pull request" button,
(b) it could be that you have a little more setup to do with windows git bash foo - something to do with ~/.ssh/config/ where you need to add an entry:
Code:
Host bitbucket.org
        IdentityFile ~/.ssh/bitbucket_rsa
        User git
I usually find the answer on stackoverflow every time I set up a new box :)
(c) Learning git while learning the git command line is rubbing salt in the wound AND your eyes, I can't recommend using Atlassian's (*free!*) SourceTree app strongly enough.

Thanks for the tips.

(a) I certainly hadn't forked, I had just cloned. I've now set up a private fork and will see how it goes.
(b) I'd followed the tutorial at github and have completed that step.
(c) I'm trying GitGUI for now. Seems OK.
 
Better to report things like this on the issue tracker; this was a bug in the station command that mean't it never thought there was any data to export when removing a station. It's now fixed. Good catch!

Haha. After my last bug report on the tracker, when I was so certain I'd found a bug, only to realise that:


  1. it wasn't, it was bad data
  2. I was the cause of the bad data
  3. I had carelessly failed to spot that there was bad data being downloaded

I thought I had better post here first and follow up with a bug report later. :)
 
Hi there,

Running TD on Linux, some crontab lines to git pull new updates for the code and station, 2d and 3h data from maddavo's site. I use Steam overlay web browser and my Linux machine has Shellinabox installed, so ssh terminal on web page is heavy use :)

I really like use of command line and do various things about bash, poweshell, windows command prompt, etc. But to prevent reinventing the wheel, has anyone do any web-frontend for Linux, like using Django framework or substitute?

A while back, someone made a flask-based front end. If you go through the forks you should find it. That's why I partition each command module into two main components: "run(...)" does the work, "render(...)" turns the results into text, but calling render() is optional; so if you were making your own front end you could basically call your own render functions.

- - - - - Additional Content Posted / Auto Merge - - - - -

Haha. After my last bug report on the tracker, when I was so certain I'd found a bug, only to realise that:

  1. it wasn't, it was bad data

It wasn't a trivial case, and you'd clearly done more than cursory diligence before reporting it, and you raised a perfectly legit problem :)


-Oliver

- - - - - Additional Content Posted / Auto Merge - - - - -

Code:
[b]
v6.12.4 Mar 02 2015
. (kfsone) Added 175 Systems
. (kfsone) Fixed #193 "run" was ignoring --ls-max
. (kfsone) assorted minor fixes
+ DRy411S : ShipVendors
[/b]
 
Code:
[B]
v6.12.4 Mar 02 2015
. (kfsone) Added 175 Systems
. (kfsone) Fixed #193 "run" was ignoring --ls-max
. (kfsone) assorted minor fixes
+ DRy411S : ShipVendors
[/B]

Well thank you. :)

I guess what I don't understand now is how to make my fork inherit any updates that you make to the master, and then how to sync those with my clone of the fork on my PC

(I also note that maddavo's site now supports uploads of the shipvendor.csv file)

Edit: Never mind, I figured it out. Coolio, now I can keep my local code automatically in sync. :)
 
Last edited:
kfsone said:
If you can provide your entire command line and the output you got, we can assist you further.

e:\trade>trade.py run -vvv --lsp 1.5 --ls-max 2000 --credits 10000000 --cap 400 --pad l --ly-per 14.92 --jum 2 --hops 4
SHAMBHALA/Baynes City -> TAMAR/Gould Port (score: 2280901.120000)
Start CR: 10,000,000
Hops : 4
Jumps : 7
Gain CR : 2,314,800
Gain/Hop: 578,700
Final CR: 12,314,800

Load from SHAMBHALA/Baynes City (510ls/star, No/bm, Lrg/pad):
400 x Imperial Slaves 15,595cr vs 16,827cr, 3 days vs 31 hrs, total: 6,238,000cr
Jump SHAMBHALA -> HR 494
Unload at HR 494/Jordan Orbital (3.23Kls/star, Unk/bm, Lrg/pad) => Gain 492,800cr (1,232cr/ton) => 10,492,800cr
Load from HR 494/Jordan Orbital (3.23Kls/star, Unk/bm, Lrg/pad):
400 x Palladium 12,585cr vs 14,276cr, 31 hrs vs 28 hrs, total: 5,034,000cr
Jump HR 494 -> CD-40 395 -> SIDDHA
Unload at SIDDHA/Meuron Station (25ls/star, Unk/bm, Lrg/pad) => Gain 676,400cr (1,691cr/ton) => 11,169,200cr
Load from SIDDHA/Meuron Station (25ls/star, Unk/bm, Lrg/pad):
400 x Imperial Slaves 15,711cr vs 16,827cr, 28 hrs vs 31 hrs, total: 6,284,400cr
Jump SIDDHA -> CD-40 395 -> HR 494
Unload at HR 494/Jordan Orbital (3.23Kls/star, Unk/bm, Lrg/pad) => Gain 446,400cr (1,116cr/ton) => 11,615,600cr
Load from HR 494/Jordan Orbital (3.23Kls/star, Unk/bm, Lrg/pad):
400 x Palladium 12,585cr vs 14,333cr, 31 hrs vs 3 days, total: 5,034,000cr
Jump HR 494 -> CD-40 395 -> TAMAR
Unload at TAMAR/Gould Port (1.22Kls/star, No/bm, Lrg/pad) => Gain 699,200cr (1,748cr/ton) => 12,314,800cr
----------------------------------------------------------------------------
Finish at TAMAR/Gould Port (1.22Kls/star, No/bm, Lrg/pad) gaining 2,314,800cr => est 12,314,800cr total

Ok, I have it set to a max of 2000 ls, but I am getting results for 3.23 LS. This is not isolated. I get this results with any search I make... Unless I am doing it wrong. And THAT is distinctly possible.
 
e:\trade>trade.py run -vvv --lsp 1.5 --ls-max 2000 --credits 10000000 --cap 400 --pad l --ly-per 14.92 --jum 2 --hops 4
SHAMBHALA/Baynes City -> TAMAR/Gould Port (score: 2280901.120000)
...
Ok, I have it set to a max of 2000 ls, but I am getting results for 3.23 LS. This is not isolated. I get this results with any search I make... Unless I am doing it wrong. And THAT is distinctly possible.


Are you running 16.2.4 the readme is at the end of my reply 2-3 above yours.
 
Bug report: "trade.py run" doesn't add the insurance reserve back to the credit total at the end of the run when reporting the expected final total. Example invocation:
Code:
C:\Users\Jason\TradeDangerous>trade.py run -vvv --lsp 1.5 --ls-max 2000 --credits 2000000 --cap 84 --pad m --ly-per 19.68 --jum 2 --hops 4 --ins 250000 --fr pand --start 4
MACUA/Brillant Relay -> ROSS 718/Lenoir Hub (score: 251183.011800)
Start CR:  1,750,000
Hops    :          4
Jumps   :          8
Gain CR :    251,076
Gain/Hop:     62,769
Final CR:  2,001,076
  Load from MACUA/Brillant Relay (1,450ls/star, No/bm, Med/pad):
       84 x Uraninite      740cr vs      898cr, 52 days vs 27 days, total:     62,160cr
  Jump MACUA -> YUROK -> IKSVATI
  Unload at IKSVATI/Olsen Horizons (1,018ls/star, No/bm, Med/pad) => Gain 13,272cr (158cr/ton) => 1,763,272cr
  Load from IKSVATI/Olsen Horizons (1,018ls/star, No/bm, Med/pad):
       84 x Gold         9,276cr vs   10,304cr, 27 days vs 8 days, total:    779,184cr
  Jump IKSVATI -> TEDE -> ROSS 718
  Unload at ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad) => Gain 86,352cr (1,028cr/ton) => 1,849,624cr
  Load from ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad):
       84 x Liquor         550cr vs      853cr, 8 days vs 15 days, total:     46,200cr
  Jump ROSS 718 -> ALRAI SECTOR NN-T -7 -> CAUANI
  Unload at CAUANI/Poisson Hub (189ls/star, Yes/bm, Med/pad) => Gain 25,452cr (303cr/ton) => 1,875,076cr
  Load from CAUANI/Poisson Hub (189ls/star, Yes/bm, Med/pad):
       84 x Gold         8,804cr vs   10,304cr, 15 days vs 8 days, total:    739,536cr
  Jump CAUANI -> ALRAI SECTOR NN-T -7 -> ROSS 718
  Unload at ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad) => Gain 126,000cr (1,500cr/ton) => 2,001,076cr
  ----------------------------------------------------------------------------
  Finish at ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad) gaining 251,076cr => est 2,001,076cr total
It's clear that the way you implement the --insurance option is to subtract the value of the insurance reserve from the available credits before you start the analyses. You can see this in the first step, which shows "Starting CR" as 1750000 (my initial credits of 200000 minus my insurance reserve of 250000). The final computed result shows a total gain of 251,076 (just a hair more than my insurance reserve), which should show a final balance of 2,251,076 but instead shows that I barely made 1000 credits.
 
I'm having a problem when I attempt to run the "Trading tour of a multi-station system" listed on your cookbook page.

The page displays:
Code:
We can use the "--unique" option to run and --jumps 0 to tell it not to make any inter-system jumps.

Lets say you're in Sol, where there are 9 stations, which means 8 hops. We can either specify a starting station:

$ trade.py run --jumps 0 --ly 0.1 --capacity 50 --credits 40000 --from "mars high" --hops 8 --unique

or better still we can let TD do its thing and find the most profitable route, including which station we start from:

$ trade.py run --ju 0 --ly 0.1 --cap 50 --cr 40000 --from sol --hops 8 --uni -vv

However, when I try running that (modifying only the credits I have on hand and the capacity of my ship) I get an error:

Code:
trade.py run --jumps 0 --ly 0.1 --capacity 20 --credits 244476 --from "mars high" --hops 8 --unique -vvv
Traceback: (most recent call last):
  File "C:\trade\trade.py", line 102 in <module>
    main(sys.argv)
  File "C:\trade\trade.py" line 76, in main
    results - cmdenv.run(tdb)
  File "C:\trade\commands\commandenv.py", line 80, in run
    return self._cmd.run(results, self, tdb)
  File "C:\trade\commands\run_cmd.py", line 909, in run
    validateRunArguments(tdb, cmdenv)
  File "C:\trade\commands\run_cmd.py", line 693, in validateRunArguments
    checkDestinations(tdb, cmdenv)
  File "C:\trade\commands\run_cmd.py", line 632, in checkDestinations
 for station in stationsSrc
UnboundLocalError: local variable 'stationSrc' referenced before assignment

I've made sure I wiped the directory, downloaded and re-installed the lasted version from the download repository link at https://bitbucket.org/kfsone/tradedangerous/downloads and then re-imported the data using the full import:

Code:
trade.py import --plug maddavo --opt stations --opt systems

I even tried closing, re-opening the command prompt, then re-importing and re-running the command as listed above, with the same results.

(Running Win7x64, if that makes a difference.)

Hope this helps, my command line practice is limited to the minimum I needed in the DOS days and my programming experience is not even that advanced, hopefully this is helpful, and not some derp I've missed.

Other than this speedbump, I love this tool so very much and really doubt I'd be enjoying E:D nearly as much without it!

[EDIT]: Looks like there is something borked with my install, trying to run a more standard run gave me similar errors. Further investigation is clearly required on my part.

[Son of Edit]: Okay, there's definitely something funny going on at my end, I wiped the C:\trade folder again, launched SourceTree, removed the settings I had in that and re-selected the master file, confirmed it had downloaded the selected files, imported maddavo's plugin with both options again, and still get the same error.

The only thing I haven't tried at this point is to reboot, though I'm not certain why that would make a difference.

But then, on this level I am barely more than a button-hamster, so I don't really expect to understand.
 
Last edited:
Are you running 16.2.4 the readme is at the end of my reply 2-3 above yours.

I am not sure. I performed a pull last night as I seen there was a version difference. I will check again in around 9 hours. I am also having difficulties entering in a starting location. It always complains. I will look at the values again when I check the version later tonight.
 
I'm having a problem when I attempt to run the "Trading tour of a multi-station system" listed on your cookbook page.

...

The only thing I haven't tried at this point is to reboot, though I'm not certain why that would make a difference.

But then, on this level I am barely more than a button-hamster, so I don't really expect to understand.

Fixed in 16.3.0, but this means you'll get a different error: the current trade data (see trade.py market -vv "mars high") claims that the only thing mars high is selling is bio waste.

Code:
[b]$ ./trade.py run --jumps 0 --ly 0.1 --capacity 20 --credits 244476 --from "mars high" --hops 8 --unique -vvv [/b]
./trade.py: Error: No profitable buyers found for the goods at SOL/Mars High.

You may want to try:
  ./trade.py local "SOL" --ly 0.0 -vv --stations --trading
or:
  ./trade.py market "SOL/Mars High" --sell -vv

Possible causes:
- No profitable runs or routes meet your criteria,
- Missing Systems or Stations along the route (see "local -vv"),
- Missing price data (see "market -vv" or "update -h"),

If you are not sure where to get data from, consider using a crowd-sourcing
project such as "maddavo's" (http://www.davek.com.au/td/).

For more help, see the TradeDangerous Wiki:
    http://kfs.org/td/wiki

[b]$ ./trade.py market "SOL/Mars High" -S -vv[/b]
    Item     Selling     Avg Supply Age/Days
--------------------------------------------
+WASTE
    Biowaste      20      18 8,293M     0.66

(this is a list of everything the station is known to be selling at the time)
- - - - - Additional Content Posted / Auto Merge - - - - -

Take a moment to digest this one :)

Short version:
. You can use "--opt=csvs" to import systems, stations and shipvendors from maddvo,
. "local" can be used to find a nearby black market/shipyard,
. "market" is much more friendly,
. "run" should give better feedback when asked to do confusing things,

Code:
[SIZE=3]
v6.13.0 Mar 03 2015
. (kfsone) Added "modified" column to ShipVendor csv,
. (kfsone) "maddavo" import plugin:
  - Added "--opt=shipvendors" which imports his ShipVendor.csv,
     - entries with a 'modified' of "DELETED" will be ...
  - Added "--opt=csvonly" to stop after importing any csv files (no prices),
  - Added "--opt=csvs" to import all the csvs
    (equivalent to typing --opt=systems --opt=stations --opt=shipvendor)
. (kfsone) "market" sub-command:
  - Default behavior is equivalent to "--buy --sell",
  - You now only need to specify --buy (-B) or --sell (-S) to list ONLY
    those columns, so "trade.py market SOL -vv" now shows both sets of data.
. (kfsone) "local" sub-command:
  - Added "--stations" option: only list systems with stations,
  - Added "--trading" option: only list stations that are flagged as having
    a market or have trade data available
  - Added "--blackmarket" option: only list stations with a black market,
  - Added "--shipyard" option: only list stations with a ship yard,
. (kfsone) "run" sub-command:
  - check from stations for whether they have anything that can be bought,
  - better feedback on some edge-cases where a route cannot be found,
  - fixed some problems with --via,
  - better feedback when using --jumps=0 or --ly=0,
[/SIZE]
 
Last edited:
Bug report: "trade.py run" doesn't add the insurance reserve back to the credit total at the end of the run when reporting the expected final total. Example invocation:
Code:
C:\Users\Jason\TradeDangerous>trade.py run -vvv --lsp 1.5 --ls-max 2000 --credits 2000000 --cap 84 --pad m --ly-per 19.68 --jum 2 --hops 4 --ins 250000 --fr pand --start 4
MACUA/Brillant Relay -> ROSS 718/Lenoir Hub (score: 251183.011800)
Start CR:  1,750,000
Hops    :          4
Jumps   :          8
Gain CR :    251,076
Gain/Hop:     62,769
Final CR:  2,001,076
  Load from MACUA/Brillant Relay (1,450ls/star, No/bm, Med/pad):
       84 x Uraninite      740cr vs      898cr, 52 days vs 27 days, total:     62,160cr
  Jump MACUA -> YUROK -> IKSVATI
  Unload at IKSVATI/Olsen Horizons (1,018ls/star, No/bm, Med/pad) => Gain 13,272cr (158cr/ton) => 1,763,272cr
  Load from IKSVATI/Olsen Horizons (1,018ls/star, No/bm, Med/pad):
       84 x Gold         9,276cr vs   10,304cr, 27 days vs 8 days, total:    779,184cr
  Jump IKSVATI -> TEDE -> ROSS 718
  Unload at ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad) => Gain 86,352cr (1,028cr/ton) => 1,849,624cr
  Load from ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad):
       84 x Liquor         550cr vs      853cr, 8 days vs 15 days, total:     46,200cr
  Jump ROSS 718 -> ALRAI SECTOR NN-T -7 -> CAUANI
  Unload at CAUANI/Poisson Hub (189ls/star, Yes/bm, Med/pad) => Gain 25,452cr (303cr/ton) => 1,875,076cr
  Load from CAUANI/Poisson Hub (189ls/star, Yes/bm, Med/pad):
       84 x Gold         8,804cr vs   10,304cr, 15 days vs 8 days, total:    739,536cr
  Jump CAUANI -> ALRAI SECTOR NN-T -7 -> ROSS 718
  Unload at ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad) => Gain 126,000cr (1,500cr/ton) => 2,001,076cr
  ----------------------------------------------------------------------------
  Finish at ROSS 718/Lenoir Hub (118ls/star, No/bm, Med/pad) gaining 251,076cr => est 2,001,076cr total
It's clear that the way you implement the --insurance option is to subtract the value of the insurance reserve from the available credits before you start the analyses. You can see this in the first step, which shows "Starting CR" as 1750000 (my initial credits of 200000 minus my insurance reserve of 250000). The final computed result shows a total gain of 251,076 (just a hair more than my insurance reserve), which should show a final balance of 2,251,076 but instead shows that I barely made 1000 credits.

Is this fixed in 6.13.0?
 
First of all: awesome tool, guys! Hats off :)

Are there any plans to separate all the SQL stuff from the rest of the code? Right now it's very tightly coupled with SQLite, which makes it daunting to try to turn it into a multi-user, hosted tool.
 
First of all: awesome tool, guys! Hats off :)

Are there any plans to separate all the SQL stuff from the rest of the code? Right now it's very tightly coupled with SQLite, which makes it daunting to try to turn it into a multi-user, hosted tool.

No plans per se but decoupling the back-end like would be a natural extension of the modularity I've tried for.
 
Outstanding!

Latest version installed (--opt=csvs is already making my life easier!) and I'll go back to planning my runs this way.

Odd that Mars High currently only shows Biowaste for trading. I know I uploaded the OCR'd market lists when I was there yesterday. And that little glitch is fixed now also, plenty of commodities available again.

Ahh, well, time to plot a course from Wyrd back to Sol now, and then do a Grand Tour of the home system. :D
 
I was just trying to import stations from Maddavo's and got an error because someone uploaded bad ls-from-star values. I've just checked in a change that will make it do this instead:

Code:
trade.py import --plug=maddavo
Connecting to server: http://www.davek.com.au/td/prices-3h.asp
import.prices: 396,874/396,874 bytes |  216.4KB/s | 100.00% 
NOTE: 29 stations updated:
Q1 ERIDANI/Windt Terminal, GRABIKA/Hermite Horizons, ERAVATE/Russell Ring, HIP 118311/Lubbock Market, LAMIA/Noon Port, FUTHARK/Barker's Horizon, TARTARUS/Foerster Vision, HR 7291/Yourkevitch Hub, XIHE/Zhen Dock, CEMIESS/Mackenzie Relay, ...
NOTE: Import complete: 3,215 items (3,215 buyers, 1,160 sellers) for 57 stations in 47 systems
NOTE: Import completed.
 
Just tried a reload of data..

trade.py import --plug maddavo --opt stations --opt systems

got

NOTE: WUONANIE/Picacio Station (#12772) updated in local db: mkt('?'=>'Y')
NOTE: XI-2 CAPRICORNI/Bouch Terminal (#13458) added to local db: ls=0, mkt=Y, bm=?, yard=?, pad=?, mod=2015-03-06 09:38:00
Traceback (most recent call last):
File "E:\trade\trade.py", line 102, in <module>
main(sys.argv)
File "E:\trade\trade.py", line 76, in main
results = cmdenv.run(tdb)
File "E:\trade\commands\commandenv.py", line 80, in run
return self._cmd.run(results, self, tdb)
File "E:\trade\commands\import_cmd.py", line 101, in run
if not plugin.run():
File "E:\trade\plugins\maddavo_plug.py", line 461, in run
self.import_stations()
File "E:\trade\plugins\maddavo_plug.py", line 288, in import_stations
lsFromStar = int(values[2])
ValueError: invalid literal for int() with base 10: '11.5'


So yup, something is bugged....

Nope, is now fixed....
 
Last edited:
Back
Top Bottom