In-Development TradeDangerous: power-user trade optimizer

MarkAusten

MarkAusten
CMDR Ibizan said:
Mark, first results on TD Helper v2 beta test. I ran through your instructions and downloaded TDH v2, the latest Trade Dangerous from Eyeonus and the latest EDDBlink. Started up the new TD Helper and hit the Update DB button. Practically straight away the display showed "Database is locked. Waiting for access". That line was then repeated every second or so. I left it running for about 10 minutes but looking at my Trade\data folder, the file TradeDangerous.db existed but remained stubbornly at 0 bytes. My brother was doing the same thing and he got the same results.

Obviously at this stage, there was no database to work with but it didn't look like one was being created either. It just kept showing the Database is locked message.

We eventually abandoned that command and ran the eddblink -O clean command to create the database.

Back to TD Helper once the database existed and before clicking anything in the window. the words Building database appeared. Only a few seconds later, everything was ready to go. We both filled in the relevant fields that we wanted and did a 4 hop 2 jump run from our current location. In my case, 10.191 seconds later, the results appeared. Fantastic!

So, in summary, with an existing database it all seems to work very well. Something is not right with the initial database creation at the moment. Perhaps, if we had left it running for the 90 minutes that one of your examples took, it may have completed but was yours displaying anything while it was doing that? All we had was the repeating Database is locked message.

However, with the old version, I always used to get up to 60 seconds of "Not Responding" after making any changes to any of the fields or running a command. I still get them on the new version but only momentary ones - probably 0.25 seconds then off it goes doing what it is supposed to so there is quite a performance improvement there!

I'll let you know if anything else crops up!

Thanks again for doing this.
Click to expand...

The performance improvement is entirely down to Eyeonus and his crew, TDHelper just displays the results from their improved algorithms. I have also figured out why the locked database from your description and will amend my set up instructions to deal with it. Essentially, TD Helper tries to load up various things from the start, locking the database as it does so, and the eddblink plugin cannot get access. The answer is to run eddblink -O clean in a terminal window before starting up TD Helper as you found out.

Glad it's working for you, GUI improvements to come as and when I have time.
 

MarkAusten

MarkAusten
eyeonus said:
No:
Code: 
        if tdenv.maxAge:
            maxDays = datetime.timedelta(days=tdenv.maxAge)
            cutoff = datetime.datetime.now() - maxDays
            wheres.append("(modified >= ?)")
            binds.append(str(cutoff.replace(microsecond=0)))

With an age of 2, it ignores anything that is more than precisely 48 hours old.

On the other hand, the depiction of a station's age is "rounded" quite a bit:
Code: 
def describeAge(ageInSeconds):
    """
    Turns an age (in seconds) into a text representation.
    """
    hours = int(ageInSeconds / 3600)
    if hours < 1:
        return "<1 hr"
    if hours == 1:
        return "1 hr"
    if hours < 48:
        return str(hours) + " hrs"
    days = int(hours / 24)
    if days < 90:
        return str(days) + " days"

    months = int(days / 31)
    return str(months) + " mths"
Anything less than 48<=x<=72 hours will be shown as "2 days".

Also, I looked at the data for the first station, apparently it's been updated since your post, because the modified time is saying 26 June when I looked at it in the DB, so I have no idea how old it actually was when you were doing this.
Click to expand...

I can't tell you how old the data was either, however, the output was quite specific. The price age was 2 days but setting the age parameter to 3 did not find it whereas setting the parameter to 4 or more did find the prices.

I have just run an update (22:10 BST) and then the command again with 3 days for the age and the results show no profitable route. Change the age to 4 and the profitable route with 2 day old price data is found.

The command I am using is:

Code: 
-u "trade.py" run --fr="TURIR/Brooks Terminal" --cap=720 --ins=3172226 --cr=430000000 --ly=24.73 --empty=37.61 --pad=L --margin=0.1 --hops=2 --jum=3 --age=3 --avoid="slaves,narcotics" --loop --summary --progress -vv

Very odd!
 

eyeonus

E
I just ran the same command (with age=2):
Code: 
>python "trade.py" run --fr="TURIR/Brooks Terminal" --cap=720 --ins=3172226 --cr=430000000 --ly=24.73 --empty=37.61 --pad=L --margin=0.1 --hops=2 --jum=3 --age=2 --avoid="slaves,narcotics" --loop --summary --progress -vv
* Hop   1: .........1 origins
* Hop   2: .......379 origins .. 7,200-3,125,520cr gain, 10-4,341cr/ton
Turir/Brooks Terminal -> Turir/Brooks Terminal (score: 1850271.772203)

  Load from Turir/Brooks Terminal (42ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      720 x Medicines/Basic Medicines      525cr vs    4,527cr, 1 hr vs 2 hrs
    Expect to gain 2,881,440cr (4,002cr/ton)

  Load from Sherones/Gustav Sporer Station (324ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      720 x Chemicals/Mineral Oil          189cr vs    1,315cr, 2 hrs vs 1 hr
    Expect to gain 810,720cr (1,126cr/ton)
  ----------------------------------------------------------------------------
Finish at Turir/Brooks Terminal (42ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 3,692,160cr (2,564cr/ton) => est 433,692,160cr total
 

eyeonus

E
MarkAusten said:
The performance improvement is entirely down to Eyeonus and his crew....
Click to expand...

I have a crew?

In any case, I've been meaning to ask: Could you make it so TDHelper can get the current credits in the same way it gets the recent systems? I know ED:MC is able to get the credits, although I don't know how, so I know it's possible.

It'd just be nice to not have to update that manually every time.
 

Tromador

Tromador
eyeonus said:
I have a crew?

In any case, I've been meaning to ask: Could you make it so TDHelper can get the current credits in the same way it gets the recent systems? I know ED:MC is able to get the credits, although I don't know how, so I know it's possible.

It'd just be nice to not have to update that manually every time.
Click to expand...

Standard journal event, I believe.

Edit:

It's in the game load event, then you need to programmatically track it from various other events (such as market trades and such).
However - I have been advised by a buddy who works with this that some events which can change the balance are not logged, so the longer you play without a load event, the further out of whack the balance would get.
The workaround would be to log out/in to create another load event, with the accurate balance.

So - ED helper would have to monitor the journal for a load event and all other credit events. It's not hard as such - but could be time consuming.

http://hosting.zaonce.net/community/journal/v11/Journal_Manual_v11.pdf
 
Last edited:

Avi0013

Avi0013
MarkAusten said:
I can't tell you how old the data was either, however, the output was quite specific. The price age was 2 days but setting the age parameter to 3 did not find it whereas setting the parameter to 4 or more did find the prices.

I have just run an update (22:10 BST) and then the command again with 3 days for the age and the results show no profitable route. Change the age to 4 and the profitable route with 2 day old price data is found.

The command I am using is:

Code: 
-u "trade.py" run --fr="TURIR/Brooks Terminal" --cap=720 --ins=3172226 --cr=430000000 --ly=24.73 --empty=37.61 --pad=L --margin=0.1 --hops=2 --jum=3 --age=3 --avoid="slaves,narcotics" --loop --summary --progress -vv

Very odd!
Click to expand...

Updated to latest version ( bc8b8a6 ) and ran import listings. For me it works (finds @3, not @2):

Code: 
F:\Elite\TD>trade run --fr="TURIR/Brooks Terminal" --cap=720 --ins=3172226 --cr=430000000 --ly=24.73 --empty=37.61 --pad=L --margin=0.1 --hops=2 --jum=3 --age=3 --avoid="slaves,narcotics" --loop --summary --progress -vv
* Hop   1: .........1 origins
* Hop   2: .......498 origins .. 149,040-3,222,000cr gain, 207-4,475cr/ton
Turir/Brooks Terminal -> Turir/Brooks Terminal (score: 1657584.998936)

  Load from Turir/Brooks Terminal (42ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      720 x Textiles/Military Grade Fabrics      625cr vs    3,557cr, 2 days vs 2 days
    Expect to gain 2,111,040cr (2,932cr/ton)

  Load from Facece/Fortress York (913ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      720 x Metals/Gold                        9,304cr vs   10,968cr, 2 days vs 2 days
    Expect to gain 1,198,080cr (1,664cr/ton)
  ----------------------------------------------------------------------------
Finish at Turir/Brooks Terminal (42ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 3,309,120cr (2,298cr/ton) => est 433,309,120cr total


F:\Elite\TD>trade run --fr="TURIR/Brooks Terminal" --cap=720 --ins=3172226 --cr=430000000 --ly=24.73 --empty=37.61 --pad=L --margin=0.1 --hops=2 --jum=3 --age=2 --avoid="slaves,narcotics" --loop --summary --progress -vv
F:\Elite\TD\trade.py: Error: No buying prices at Turir/Brooks Terminal.

Possible causes:
- No profitable runs or routes meet your criteria,
- Missing Systems or Stations along the route (see "local -vv"),
- Missing price data (see "market -vv" or "update -h"),

If you are not sure where to get data from, consider using a crowd-sourcing
project such as EDDBlink (https://github.com/eyeonus/EDDBlink).

For more help, see the TradeDangerous Wiki:
    https://github.com/eyeonus/Trade-Dangerous/wiki
 

Tromador

Tromador
Thinking about future maintenance with updates to ED which may affect us, such as new commodities and such.

Do we need to manually add any of this information in? I see that last time (when beyond was released) Bernd did a lot of manual maintenance of files, including ships.csv and items.csv
If it is the case that we need to deal with this, then where do we get all the data from? Some could be lifted from game, but how do we get a hold of (for example) the fdev ID for the Krait II when that comes in?

Hopefully it's a non-issue. Otherwise best to address it now, before the next update is released.
 

eyeonus

E
Tromador said:
Thinking about future maintenance with updates to ED which may affect us, such as new commodities and such.

Do we need to manually add any of this information in? I see that last time (when beyond was released) Bernd did a lot of manual maintenance of files, including ships.csv and items.csv
If it is the case that we need to deal with this, then where do we get all the data from? Some could be lifted from game, but how do we get a hold of (for example) the fdev ID for the Krait II when that comes in?

Hopefully it's a non-issue. Otherwise best to address it now, before the next update is released.
Click to expand...

Any new anything that shows up on EDDB will automatically be added to the DB with the next EDDB update. So, basically, we're passing the buck. EDDB handles adding the new whatever, we just grab EDDB's data and build the DB with it.
 

MarkAusten

MarkAusten
eyeonus said:
I have a crew?

In any case, I've been meaning to ask: Could you make it so TDHelper can get the current credits in the same way it gets the recent systems? I know ED:MC is able to get the credits, although I don't know how, so I know it's possible.

It'd just be nice to not have to update that manually every time.
Click to expand...

There is some ED Companion API access code in TD Helper so it shouldn't be too difficult to achieve. I'll have a look.
 
Last edited:

Tromador

Tromador
eyeonus said:
Any new anything that shows up on EDDB will automatically be added to the DB with the next EDDB update. So, basically, we're passing the buck. EDDB handles adding the new whatever, we just grab EDDB's data and build the DB with it.
Click to expand...

*nods* Just the rares then possibly. Looking at the repository, it appears that you've done away with some of the default .csv files which used to live there. (e.g. item.csv https://github.com/eyeonus/Trade-Dangerous/commit/bb45911ddc03fae9a11fc9954778355c81c31fd1).

Is this a good idea? In the case where a user wants to run TD with only their own data, rather than an EDDB source, then those files would need to be present.
 

eyeonus

E
Tromador said:
*nods* Just the rares then possibly. Looking at the repository, it appears that you've done away with some of the default .csv files which used to live there. (e.g. item.csv https://github.com/eyeonus/Trade-Dangerous/commit/bb45911ddc03fae9a11fc9954778355c81c31fd1).

Is this a good idea? In the case where a user wants to run TD with only their own data, rather than an EDDB source, then those files would need to be present.
Click to expand...

I did that specifically because EDDBlink plugin generates them. In the case where someone doesn't want to use crowd-sourced data, '-O item,shipvend,upvend" would generate all the tables except the listings.

('item' cause the plugin to generate the Item, Category, Station, and System tables, 'shipvend' causes it to generate the Ship, Station, System, and ShipVendor tables, and 'upvend' causes it to generate the Upgrade, Station, System, and UpgradeVendor tables. Combining them does not cause overlapping tables to be generated multiple times, it just turns on the 'generate this table' option at start up, and then any tables that have that option turned on get generated during the run.)

Since it is possible there are some people who would prefer not to crowd-source, I suppose that should be put in the documentation.
 
Last edited:

MarkAusten

MarkAusten
eyeonus said:
I have a crew?

In any case, I've been meaning to ask: Could you make it so TDHelper can get the current credits in the same way it gets the recent systems? I know ED:MC is able to get the credits, although I don't know how, so I know it's possible.

It'd just be nice to not have to update that manually every time.
Click to expand...

It turns out that there is a fairly neat Companion API application already available by the name of EDCE. You can find it here: https://github.com/EDCD/EDDI. It's in python and you need to initialise it before using it but having done that it is easy to call from TDHelper. Since I don't like reinventing the wheel I'm going to use this utility instead of writing c# code that does the same thing.

I have a version working on my development copy of TD Helper. I've added a new button 'Cmdr Profile' which calls EDCE and then retrieves the credit balance from the resulting json file. It works quite well so I'll now integrate it nicely with TD Helper and let it out into the wild for testing. Probably sometime next week since doing all this has meant that my household To Do list has not been worked on and my wife is getting antsy.

Happy wife and all that.

At least I know that the system works without too much pain.
 

Tromador

Tromador
As I know you all want to fly a 2500LY trade route, I just got the prices uploaded at GM CEPHEI/Elephant's Trunk Mine

No no - don't all thank me at once. :D

(I had no idea it was there...)
 
Last edited:

Avi0013

Avi0013
This isn't right, is it?

Code: 
F:\Elite\TD>trade run -vvv --summary --progress --avoid "imperial,slaves" --credits 120m --cap 192 --ins 10m --empty 15.92 --ly 12.75 --pad ML --from "GM CEPHEI/Elephant's Trunk Mine" --hops 1 --jumps 1000 --lsp=0
* Hop   1: .........1 origins
F:\Elite\TD\trade.py: Error: Couldn't find any trading links within 1000 x 12.75ly jumps of GM Cephei.

Possible causes:
- No profitable runs or routes meet your criteria,
- Missing Systems or Stations along the route (see "local -vv"),
- Missing price data (see "market -vv" or "update -h"),

If you are not sure where to get data from, consider using a crowd-sourcing
project such as EDDBlink (https://github.com/eyeonus/EDDBlink).
 

eyeonus

E
MarkAusten said:
It turns out that there is a fairly neat Companion API application already available by the name of EDCE. You can find it here: https://github.com/EDCD/EDDI. It's in python and you need to initialise it before using it but having done that it is easy to call from TDHelper. Since I don't like reinventing the wheel I'm going to use this utility instead of writing c# code that does the same thing.

I have a version working on my development copy of TD Helper. I've added a new button 'Cmdr Profile' which calls EDCE and then retrieves the credit balance from the resulting json file. It works quite well so I'll now integrate it nicely with TD Helper and let it out into the wild for testing. Probably sometime next week since doing all this has meant that my household To Do list has not been worked on and my wife is getting antsy.

Happy wife and all that.

At least I know that the system works without too much pain.
Click to expand...

Just out of curiosity, are there any other TDHelper fields that could be auto-filled the same way, like insurance, cargo size, etc.?

Avi0013 said:
This isn't right, is it?

Code: 
F:\Elite\TD>trade run -vvv --summary --progress --avoid "imperial,slaves" --credits 120m --cap 192 --ins 10m --empty 15.92 --ly 12.75 --pad ML --from "GM CEPHEI/Elephant's Trunk Mine" --hops 1 --jumps 1000 --lsp=0
* Hop   1: .........1 origins
F:\Elite\TD\trade.py: Error: Couldn't find any trading links within 1000 x 12.75ly jumps of GM Cephei.

Possible causes:
- No profitable runs or routes meet your criteria,
- Missing Systems or Stations along the route (see "local -vv"),
- Missing price data (see "market -vv" or "update -h"),

If you are not sure where to get data from, consider using a crowd-sourcing
project such as EDDBlink (https://github.com/eyeonus/EDDBlink).
Click to expand...


It could be. That system is pretty hinterlandish.
 

MarkAusten

MarkAusten
eyeonus said:
Just out of curiosity, are there any other TDHelper fields that could be auto-filled the same way, like insurance, cargo size, etc.?
Click to expand...

I've not found anything remotely resembling insurance so that is out but cargo might be a possibility. The load out of the current ship is included in the profile so it would just be a matter of finding the cargo rack installed and then adding up the capacity.
 
You must log in or register to reply here.
Back
Top Bottom