In-Development TradeDangerous: power-user trade optimizer

<Network
Port="0"
upnpenabled="0"
LogFile="netLog"
DatestampLog="1"
VerboseLogging="1"
>
</Network>

The Verboselogging allows your system to cache your system location. This in turn allows you to use EliteOCR so you can easily extract invalid and make corrections (including single click uploads to for inclusion into TD... Since ALL of the trade tools are crowd-sourced, if you are willing to use the tools, you should be willing to contribute...

Try this one:
trade.py run -vvv --ls-max 2000 --credits 10000000 --cap 1 --ly-per 10 --jum 3 --hops 10 --mgpt 2500

That will typically give you a round trip somewhere in the middle. I use these values because it is easier to see actual profit per ton.
 
Before i checkout an update that overwrites my station.csv, i run this batch script: (located in my scripts folder)
Code:
@echo off
cd ..
echo Dateiupload beginnt..
misc\madupload.py data/Station.csv
pause
So all changes regarding the stations are uploaded to maddavo, that's probably what you already do.

Afterwards:

I click on "Fetch" inside SourceTree and normally it gets all changes from git, only sometimes, it doesn't show up there, but don't know why.
With the now updated list of remote branches i select the latest and click on checkout.
If it stops because i have a changed stations.csv then i delete the changed file. (i already uploaded to maddavo, so it's fine)
checkout once more.
do a
Code:
@echo off
..\trade.py import --plug=maddavo --opt=csvs
..\trade.py import --plug=maddavo --opt=use3h
pause
And everything should be fine.

I hope that helps somehow.


This^^^ I had exactly the same problems as you describe. The issues are caused because some of the TD 'source' is actually data (the data/*.csv files). And you are updating that data outside of the Git system, either by issung trade.py station commands, or downloading new .csv files from maddavo (directly or using the trade.py import command)

Thanks to both of you for your replies. I can see that basically we all have the same problem and the same basic solution, save the 'data' we've cultivated ourselves and then just manually replace TD with the updated versions.

bdragon, love the batch file. I have copied it and will start using it instead of manually making changes through the Edit Stations link on maddavo's website.

dry411s, brillant idea with the email subscriptions to the TD repo, can't believe I didnt think of that myself. I think that is basically how I'll do it for now, wait for an update and then download and replace/edit only the files that have changed manually, instead of messing with git and sourcetree.
 
Ok so now that I have been introduced to some more of TD's cool scripts (madupload.py) I've written a batch file to easily upload to Maddavo's Market Share appropriate data and I thought I would share and get feedback.

Code:
@echo off
echo Importing...
grep @ C:\EliteDangerous\EliteOCR_Exports\import.prices
echo(
echo Updating Maddavo^'s Market Share
c:\EliteDangerous\tradedangerous\misc\madupload.py c:\EliteDangerous\EliteOCR_Exports\import.prices
echo(
echo Updating TradeDangerous
echo C:\EliteDangerous\tradedangerous\trade.py import C:\EliteDangerous\EliteOCR_Exports\import.prices -vv

The script uses absolute paths simply in-case I decide to move it later. You would obviously replace with your own paths to the appropriate files:

C:\EliteDangerous\EliteOCR_Exports = Where EliteOCR saves it's export data (Can be set in preferences for EliteOCR)
C:\EliteDangerous\tradedangerous = Where TD lives

And it uses grep which is not available in a standard Windows system and will need to be installed along with lots of other Linux deliciousness from one of the many UNIX tools for Windows installations out there. grep is only used to show what stations it is actually importing (so I can OCR several stations from a single system in one import) so you don't need it for the script to work.

So you would use EliteOCR to screenscrape the price data
Then click the Export to TradeDangerous button in EliteOCR
Then run the script

Volia data is uploaded to maddavo's and imported into TD.


Really basic scripting but it's more for the Windows-non-programmer users of TD than the advanced programmer users.

Hopefully it will help someone.
 
Before i checkout an update that overwrites my station.csv, i run this batch script: (located in my scripts folder)
Code:
@echo off
cd ..
echo Dateiupload beginnt..
misc\madupload.py data/Station.csv
pause
So all changes regarding the stations are uploaded to maddavo, that's probably what you already do.

Afterwards:

<snip>

If you are also using trade.py shipvendors you should also use madupload.py to upload your shipvendors.csv to maddavo before blatting it with the latest TD version.

Essentially, what you are doing for stations.csv, but also needs doing for shipvendors.csv
 
Yeah not really looking at ships yet on account of how I'm so poor couldn't afford a new ship if I wanted to, But I should be updating the crowd-sourced data re ship vendors so will start doing that now and will make the appropriate changes to batch files.

Thanks for that.
 
The point of the story.....

Can anyone give a set of simple steps/commands to use to prepare for and download/merge an update, without having to teach me git?

This is one of the reasons I'm going to stop providing data in the TD source repository -- I'll provide my own baseline data to download someplace but not in the repository...

It's important to realize there are three things: There's the origin repository which I'll refer to as "bitbucket"; there's your local repository in the ".git" directory and there's your local checked out copy of trade dangerous which I'll call "trade". It's sometimes a little difficult to orient yourself because the ".git" directory lives inside the "trade" directory, but that's by the by.

A git "repository" is a collection of changes, its like a saved undo buffer: add these lines, remove that line. A "working copy" is what happens if you play those changes.

Because of git magic, it's easy to forget that when you "clone"d TD locally, you actually created something new and independent - the TD repository on your local machine is its own unique snowflake. Since you populated it with changes from the TD repo on bitbucket, it will generate the same TD that everyone has.

When you do "git pull" it's going to freshen up your repository and working copy with new changes that were added to the origin on bitbucket. So if you have made changes to local files, it will stop.

There are three approaches you can take.

1. Blow your local changes.

do a "git status" to see which files you've modified and then restore them using the "git checkout" command (or 'undo changes' in source tree' by clicking on the files).

Then you can just "git pull".

Code:
git checkout data
git pull
trade.py buildcache -fi

2. Sing-a-long-a-merge

The ".git" folder on your local machine is actually a *clone* of the TD repo, meaning it's independent. It is said to "track" the TD repository so you can incrementally bring in new changes.

So you can just go ahead and "commit" your changes into your local repository.

Code:
git commit -m "gonna pull" data

(-m specifies the commit message)

You can now incorporate the changes from the bitbucket repository either by "merging" them or by "rebasing". Merging is the safer option but generates an extra commit that can make for a noisy commit history if you care about that, rebase can just take a little longer to wrap your head around. At the bottom of the "pull" dialog box in SourceTree there is a "rebase instead of merge" checkbox.

Rebase is the "safe" version when your commits are private/local only. "merge" is for when you are combining shared changes. Since I'm the only person who can write to the BitBucket TD repository, I'm the only person that can "push" to it, it's safe to say you can always use rebase.

One more catch: Order matters. If you're going to use this mechanism, you probably want to get into the practice of doing the following:

Code:
trade.py export
git commit data -m "Local data changes before pull"
git pull --rebase
trade.py buildcache -f -i

You may want to the command line

Code:
git gui

or look back into using source tree because you're eventually going to run into a Merge Conflict. The first few times, these will seem like a special form of hell.

3. Fork it and don't give up.

The other option would potentially allow you to become a contributor. If you've gone to the lengths of learning all of the above, you might look into going a step further.

Basically, when you "fork", BitBucket (or github, or whoever you prefer) creates a clone of TD for you, allowing you to make and commit changes in your own version of TD that you can share with others - you can clone it on your machine, make changes and push them.

The purpose of doing this is that these services provide a feature called a "pull request". If you want to add a feature or fix a bug in td, or submit data changes until 7.0, this is the preferred mechanism.

The downside is: You'll have to get your hands dirtier with git.

Pro Tip: I wouldn't do it.
(I actually have forked things and submitted changes, but I don't want to do it again)
 
No, its not a starting system. Its the center system from which the search looks for the best bidirectional route, which usually does not start in the system you are at. I know you can tell TD with --start-jumps X to have a bigger search radius, BUT I can't tell TD to look for a bidirectional route. I am not sure if you understand the difference. Probably my english is too bad to explain in more depth.

Chem has submitted a command to do this that has been sitting for an awfully long time as a pull request, and I've looked at it a number of times trying to wrap my head around it.

My biggest problem is understanding why people want this. There's nothing in TD that avoids round trips. If you happen to be in an area with them. I found one just now first try:

Code:
./trade.py run --ly 20.1 --cap 120 --cr 5000000 --jumps 3 --hops 7 --ls-p 10 --progress -vvv --age=21 --mgpt=2200 --gpt=1500 --from Achenar -s 5
* Hop   1: .......589 origins
* Hop   2: .......228 origins
* Hop   3: ........48 origins
* Hop   4: ........19 origins
* Hop   5: .........1 origins
* Hop   6: ........18 origins
* Hop   7: .........1 origins
WONOMI/Yuzhe Survey -> HUANG GUN/Pittendreigh City (score: 1314343.680000)
Start CR:  5,000,000
Hops    :          7
Jumps   :         20
Gain CR :  1,319,160
Gain/Hop:    188,451
Final CR:  6,319,160

  Load from WONOMI/Yuzhe Survey (102ls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Palladium         12,517cr vs   14,270cr, 16 days vs 6 days, total:  1,502,040cr
  Jump WONOMI -> OSHANLA -> KREMOVI -> KELIUONA
  Unload at KELIUONA/Wenzel City (44ls, Pad:L, Shp:Y, Out:Y, Ref:Y) => Gain 210,360cr (1,753cr/ton) => 5,210,360cr
  Load from KELIUONA/Wenzel City (44ls, Pad:L, Shp:Y, Out:Y, Ref:Y):
      120 x Imperial Slaves   15,717cr vs   17,219cr, 6 days vs 17 days, total:  1,886,040cr
  Jump KELIUONA -> HIP 14752 -> MUKH
  Unload at MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y) => Gain 180,240cr (1,502cr/ton) => 5,390,600cr
  Load from MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Gold               8,779cr vs   10,291cr, 17 days vs 14 days, total:  1,053,480cr
  Jump MUKH -> HIP 15438 -> AVALO -> TOTENIS
  Unload at TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L) => Gain 181,440cr (1,512cr/ton) => 5,572,040cr
  Load from TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L):
      120 x Imperial Slaves   15,652cr vs   17,219cr, 14 days vs 17 days, total:  1,878,240cr
  Jump TOTENIS -> AVALO -> HIP 15438 -> MUKH
  Unload at MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y) => Gain 188,040cr (1,567cr/ton) => 5,760,080cr
  Load from MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Gold               8,779cr vs   10,291cr, 17 days vs 14 days, total:  1,053,480cr
  Jump MUKH -> HIP 15438 -> AVALO -> TOTENIS
  Unload at TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L) => Gain 181,440cr (1,512cr/ton) => 5,941,520cr
  Load from TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L):
      120 x Imperial Slaves   15,652cr vs   17,219cr, 14 days vs 17 days, total:  1,878,240cr
  Jump TOTENIS -> AVALO -> HIP 15438 -> MUKH
  Unload at MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y) => Gain 188,040cr (1,567cr/ton) => 6,129,560cr
  Load from MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Gold               8,779cr vs   10,359cr, 17 days vs 12 days, total:  1,053,480cr
  Jump MUKH -> HIP 11819 -> REDDIS -> HUANG GUN
  Unload at HUANG GUN/Pittendreigh City (790ls, BMk:N, Pad:L, Shp:Y, Out:Y, Ref:Y) => Gain 189,600cr (1,580cr/ton) => 6,319,160cr
  ----------------------------------------------------------------------------
  Finish at HUANG GUN/Pittendreigh City gaining 1,319,160cr => est 6,319,160cr total

The thing is, FD generally try to balance their economy to prevent them, it takes player intervention to cause them. Using gpt here kept TD from hops that were less profitable but might have taken us to richer overall finds, which is why we ended up with a loop.

There has to be a disclaimer here: this is shop talk for me, I'm not saying "nyeah nyeah you guys are wrong". I'm saying, "I don't get it". I'm not telling you all of the above to lay down the law, I'm giving you my frame of reference so that perhaps someone can help me understand what it is we're looking for here.
 
Last edited:
Thanks for the git tutorial kfsone, you've taught me just enough to know I don't want to be messing with git and there is no way I'm forking anything.

But now I have a new question:

I tried my shiny new importOCR.bat script
Code:
@echo off
echo Importing...
grep @ C:\EliteDangerous\EliteOCR_Exports\import.prices
echo(
echo Updating Maddavo^'s Market Share
c:\EliteDangerous\tradedangerous\misc\madupload.py c:\EliteDangerous\EliteOCR_Exports\import.prices
echo(
echo Updating TradeDangerous
echo C:\EliteDangerous\tradedangerous\trade.py import C:\EliteDangerous\EliteOCR_Exports\import.prices

and was disappointed to see that madupload.py wont accept my .prices file.
Code:
Upload failed: FAILURE - invalid file.
I'm assuming its because it comes from EliteOCR and not TD directly which is fine except on line 44-45 of madupload.py it says I should be uploading from my OCR Directly before importing into TD, not after.


So which is it before TD or after and if it should be before why wont it let me?

BTW I can upload the same file manually through the Maddavo website no problems.

Thanks yet again.
 
Last edited:
I had problems like this. The invalid file error message is misleading. It's more a case of not finding the file to upload. I believe that the madupload.py is written from a *nix filepath convention perspective, and expects '/' between folder names, not '\'.
 
Ok, I'm gonna ask a stupid question here, please don't scold me :p

Why can't I input 'Demand' numbers and levels, even in my local .prices file? (Have been using the experimental GUI so far, and 'Demand' is always greyed out) Am I doing something wrong or is it not allowed by Frontier or is there simply no use for that data ?

Thanks in advance for enlightening me :)

==>Edit: After a moment of thought, I'm thinking no use for it, all you really need to care about is sell price right ?
==>Edit2: Just read the rest of the manual and got my answer. Time to scold myself now: 'Read manuals much, you silly **** !' :p
 
Last edited:
I had problems like this. The invalid file error message is misleading. It's more a case of not finding the file to upload. I believe that the madupload.py is written from a *nix filepath convention perspective, and expects '/' between folder names, not '\'.

Yep that got it thanks, it all works fine now.

I would never have gotten that on my own, thanks again.
 
Ok, I'm gonna ask a stupid question here, please don't scold me :p

Why can't I input 'Demand' numbers and levels, even in my local .prices file? (Have been using the experimental GUI so far, and 'Demand' is always greyed out) Am I doing something wrong or is it not allowed by Frontier or is there simply no use for that data ?

Thanks in advance for enlightening me :)

==>Edit: After a moment of thought, I'm thinking no use for it, all you really need to care about is sell price right ?
==>Edit2: Just read the rest of the manual and got my answer. Time to scold myself now: 'Read manuals much, you silly **** !' :p

I still haven't come across a case where demand was useful - if someone makes a useful case for it or I run into and realize I've been a pratt, it'll happen :) There's a flag to allow you to populate the field, "--help" is usually your friend.

hint: DOS and bash let you use the verbiage | more (that's the pipe symbol) to put the output through a program called more which displays a page at a time with a 'more' prompt.

e.g.

Code:
trade.py update --help | more

bash users may want to use "| less".

DOS users who really want to print stuff out can use "| print":

Code:
trade.py run --help | print

It's a few years since I printed anything from a unix box so, uhm, I don't remember how. I think I physically threw the machine at the printer, and a fax may have been involved too.
 
Yep that got it thanks, it all works fine now.

I would never have gotten that on my own, thanks again.

That error is actually from Maddavo (I'm checking in a version that makes that clearer in future, and will let you see the entire message if you need to).

It seems to be that mad is rejecting filenames with a "\" in them, which makes sense.

Fixed in 16.4.3 (on my phone, it may take me a bit to figure out how to check it in)
 
Thanks for the example. I am trying to better understand the run options. What is -s 5 do in the following line?
./trade.py run --ly 20.1 --cap 120 --cr 5000000 --jumps 3 --hops 7 --ls-p 10 --progress -vvv --age=21 --mgpt=2200 --gpt=1500 --from Achenar -s 5


Chem has submitted a command to do this that has been sitting for an awfully long time as a pull request, and I've looked at it a number of times trying to wrap my head around it.

My biggest problem is understanding why people want this. There's nothing in TD that avoids round trips. If you happen to be in an area with them. I found one just now first try:

Code:
./trade.py run --ly 20.1 --cap 120 --cr 5000000 --jumps 3 --hops 7 --ls-p 10 --progress -vvv --age=21 --mgpt=2200 --gpt=1500 --from Achenar -s 5
* Hop   1: .......589 origins
* Hop   2: .......228 origins
* Hop   3: ........48 origins
* Hop   4: ........19 origins
* Hop   5: .........1 origins
* Hop   6: ........18 origins
* Hop   7: .........1 origins
WONOMI/Yuzhe Survey -> HUANG GUN/Pittendreigh City (score: 1314343.680000)
Start CR:  5,000,000
Hops    :          7
Jumps   :         20
Gain CR :  1,319,160
Gain/Hop:    188,451
Final CR:  6,319,160

  Load from WONOMI/Yuzhe Survey (102ls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Palladium         12,517cr vs   14,270cr, 16 days vs 6 days, total:  1,502,040cr
  Jump WONOMI -> OSHANLA -> KREMOVI -> KELIUONA
  Unload at KELIUONA/Wenzel City (44ls, Pad:L, Shp:Y, Out:Y, Ref:Y) => Gain 210,360cr (1,753cr/ton) => 5,210,360cr
  Load from KELIUONA/Wenzel City (44ls, Pad:L, Shp:Y, Out:Y, Ref:Y):
      120 x Imperial Slaves   15,717cr vs   17,219cr, 6 days vs 17 days, total:  1,886,040cr
  Jump KELIUONA -> HIP 14752 -> MUKH
  Unload at MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y) => Gain 180,240cr (1,502cr/ton) => 5,390,600cr
  Load from MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Gold               8,779cr vs   10,291cr, 17 days vs 14 days, total:  1,053,480cr
  Jump MUKH -> HIP 15438 -> AVALO -> TOTENIS
  Unload at TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L) => Gain 181,440cr (1,512cr/ton) => 5,572,040cr
  Load from TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L):
      120 x Imperial Slaves   15,652cr vs   17,219cr, 14 days vs 17 days, total:  1,878,240cr
  Jump TOTENIS -> AVALO -> HIP 15438 -> MUKH
  Unload at MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y) => Gain 188,040cr (1,567cr/ton) => 5,760,080cr
  Load from MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Gold               8,779cr vs   10,291cr, 17 days vs 14 days, total:  1,053,480cr
  Jump MUKH -> HIP 15438 -> AVALO -> TOTENIS
  Unload at TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L) => Gain 181,440cr (1,512cr/ton) => 5,941,520cr
  Load from TOTENIS/Edwards Terminal (157ls, BMk:N, Pad:L):
      120 x Imperial Slaves   15,652cr vs   17,219cr, 14 days vs 17 days, total:  1,878,240cr
  Jump TOTENIS -> AVALO -> HIP 15438 -> MUKH
  Unload at MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y) => Gain 188,040cr (1,567cr/ton) => 6,129,560cr
  Load from MUKH/Kawasato Terminal (1.33Kls, BMk:N, Pad:M, Shp:N, Out:N, Ref:Y):
      120 x Gold               8,779cr vs   10,359cr, 17 days vs 12 days, total:  1,053,480cr
  Jump MUKH -> HIP 11819 -> REDDIS -> HUANG GUN
  Unload at HUANG GUN/Pittendreigh City (790ls, BMk:N, Pad:L, Shp:Y, Out:Y, Ref:Y) => Gain 189,600cr (1,580cr/ton) => 6,319,160cr
  ----------------------------------------------------------------------------
  Finish at HUANG GUN/Pittendreigh City gaining 1,319,160cr => est 6,319,160cr total

The thing is, FD generally try to balance their economy to prevent them, it takes player intervention to cause them. Using gpt here kept TD from hops that were less profitable but might have taken us to richer overall finds, which is why we ended up with a loop.

There has to be a disclaimer here: this is shop talk for me, I'm not saying "nyeah nyeah you guys are wrong". I'm saying, "I don't get it". I'm not telling you all of the above to lay down the law, I'm giving you my frame of reference so that perhaps someone can help me understand what it is we're looking for here.
 
Thanks for the example. I am trying to better understand the run options. What is -s 5 do in the following line?
./trade.py run --ly 20.1 --cap 120 --cr 5000000 --jumps 3 --hops 7 --ls-p 10 --progress -vvv --age=21 --mgpt=2200 --gpt=1500 --from Achenar -s 5

"-s" is short for "--start-jumps"; it lets you use "--from" as a starting point. It includes all the stations from "--from" and within N jumps of "--from". The counter to it is "-e" for "--end-jumps" which works with "--to".

Code:
[i]find trades from abraham lincoln specifically[/i]
run --from abrahamlinc

[i]find trades from somewhere in sol[/i]
run --from sol

[i]find from sol, or a system one jump away[/i]
run --from abrahamlinc --start-jumps 1
run --from abrahamlinc -s1
run --from sol -s1

[i]find from sol or a system I can reach upto 3 jumps away when empty[/i]
run --from sol -s3 --ly 20.5 --empty-ly 25.2

[i]find a 4-hop trade that stops at a station every 3 jumps and ends no more than 2 jumps from lave.[/i]
run --hops 4 --jumps 3 --from sol --to lave --end-jumps 2
 
I can see value in asking for routes where supply is high (for buying) and/or demand is high (for selling).

But what's the value in insisting on a minimum quantity? It's very, very rare that you find a station where there isn't more of a commodity that you can carry in your cargo hold.
 
The reason? Traveling 300 ly to find out they only have 200. I would much rather skip any place with less than, say, 20,000 quantity on hand. This would allow me to use the route longer with lower credit reduction.
 
I know that too well. It happens a lot when you are flying to outposts in a python. Sometimes OCRed supplies are at 300 with High Supply, but arriving there and you often see just 160 left. Someone other was faster. And in a python there are approx. 280 tons cargo space, so that really matters in big ships. + prices fluctuate much more with such low quantities
 
Back
Top Bottom