In-Development TradeDangerous: power-user trade optimizer

Bam · Jan 9, 2016

I have been using the horizons branch recently but I cant stop it showing planetary ports when using the --no-planet switch
I am using TDHelper with --no-planet added the the run parameters, which gives me this command.

Code:

Command line: -u "D:\Trade\trade.py" run --fr="HIP 106213" --cap=448 --cr=10000000 --ly=19.34 --empty=25 --pad=L --supply=10000 --hops=2 --jum=1 --age=7 --loop --no-planet -vv

and this outputs:

Code:

HIP 106213/Tanaka Terminal -> HIP 106213/Tanaka Terminal (score: 721770.618240)
  Load from HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      448 x Imperial Slaves   13,591cr vs   15,756cr, 4 hrs
  Unload at LTT 8750/Luu Vision (71ls, BMk:N, Pad:L, Plt:Y, Shp:N, Out:Y, Ref:Y) => Gain 969,920cr (2,165cr/ton) => 10,969,920cr
  Load from LTT 8750/Luu Vision (71ls, BMk:N, Pad:L, Plt:Y, Shp:N, Out:Y, Ref:Y):
      448 x Beryllium          7,849cr vs    8,906cr, 4 hrs
  Unload at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 473,536cr (1,057cr/ton) => 11,443,456cr
  ----------------------------------------------------------------------------
Finish at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 1,443,456cr (1,611cr/ton) => est 11,443,456cr total

Luu Vision is a planetary outpost. I have also tried running the command from cmd prompt in case it was TDHelper, same results. I have checked my Stations.csv file and the planetary field is there.

Any ideas?

Andargor · Jan 9, 2016

Bam said:
I have been using the horizons branch recently but I cant stop it showing planetary ports when using the --no-planet switch
I am using TDHelper with --no-planet added the the run parameters, which gives me this command.

Code:

Command line: -u "D:\Trade\trade.py" run --fr="HIP 106213" --cap=448 --cr=10000000 --ly=19.34 --empty=25 --pad=L --supply=10000 --hops=2 --jum=1 --age=7 --loop --no-planet -vv

and this outputs:

Code:

HIP 106213/Tanaka Terminal -> HIP 106213/Tanaka Terminal (score: 721770.618240) Load from HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y): 448 x Imperial Slaves 13,591cr vs 15,756cr, 4 hrs Unload at LTT 8750/Luu Vision (71ls, BMk:N, Pad:L, Plt:Y, Shp:N, Out:Y, Ref:Y) => Gain 969,920cr (2,165cr/ton) => 10,969,920cr Load from LTT 8750/Luu Vision (71ls, BMk:N, Pad:L, Plt:Y, Shp:N, Out:Y, Ref:Y): 448 x Beryllium 7,849cr vs 8,906cr, 4 hrs Unload at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 473,536cr (1,057cr/ton) => 11,443,456cr ---------------------------------------------------------------------------- Finish at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 1,443,456cr (1,611cr/ton) => est 11,443,456cr total

Luu Vision is a planetary outpost. I have also tried running the command from cmd prompt in case it was TDHelper, same results. I have checked my Stations.csv file and the planetary field is there.

Any ideas?

Works for me, except the final destination might be a planet.

maddavo · Jan 10, 2016

Bam said:
I have been using the horizons branch recently but I cant stop it showing planetary ports when using the --no-planet switch
I am using TDHelper with --no-planet added the the run parameters, which gives me this command.

Code:

Command line: -u "D:\Trade\trade.py" run --fr="HIP 106213" --cap=448 --cr=10000000 --ly=19.34 --empty=25 --pad=L --supply=10000 --hops=2 --jum=1 --age=7 --loop --no-planet -vv

and this outputs:

Code:

HIP 106213/Tanaka Terminal -> HIP 106213/Tanaka Terminal (score: 721770.618240) Load from HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y): 448 x Imperial Slaves 13,591cr vs 15,756cr, 4 hrs Unload at LTT 8750/Luu Vision (71ls, BMk:N, Pad:L, Plt:Y, Shp:N, Out:Y, Ref:Y) => Gain 969,920cr (2,165cr/ton) => 10,969,920cr Load from LTT 8750/Luu Vision (71ls, BMk:N, Pad:L, Plt:Y, Shp:N, Out:Y, Ref:Y): 448 x Beryllium 7,849cr vs 8,906cr, 4 hrs Unload at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 473,536cr (1,057cr/ton) => 11,443,456cr ---------------------------------------------------------------------------- Finish at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 1,443,456cr (1,611cr/ton) => est 11,443,456cr total

Luu Vision is a planetary outpost. I have also tried running the command from cmd prompt in case it was TDHelper, same results. I have checked my Stations.csv file and the planetary field is there.

Any ideas?

Mmm - odd. I get the same. Thought it might have been a loop problem but I also get the same with:

Code:

trade run --fr="HIP 106213" --to"HIP 106213" --cap=448 --cr=10000000 --ly=19.34 --empty=25  --pad=L --supply=10000 --hops=2 --jum=1 --age=7 --no-planet -vv

I suggest opening an issue on bgol's bitbucket - HERE

maddavo · Jan 10, 2016

Bam said:
Luu Vision is a planetary outpost. I have also tried running the command from cmd prompt in case it was TDHelper, same results. I have checked my Stations.csv file and the planetary field is there.

Any ideas?

Looking further into this I think it may have something to do with the limits on the dataset. You're limiting to 1 jump and 7 days old. By expanding the search criteria I see that runs are generated with exclusively non-planetary stations. I think the bug is that rather than spit out "Sorry no routes found" it gives you a route with a planetary station.

As a workaround, I suggest expanding your search criteria.

Cheers,
Maddavo

OhGman · Jan 10, 2016

I downloaded the latest code but even with --i or --ignore-unknown option it still displays all the unknown stuff.

NE1 · Jan 10, 2016

DRY411S said:
Yep, the branch is not part of the original master. I don't understand how to make it work with Atlassian, but I do know how to download it and overwrite what I've got.

I haven't actually done it yet though.

Only way i can do it ATM: is when you goto https://bitbucket.org/bgol/tradedangerous/branch/horizon - on the Rightside of the page select Checkout and then select "Checkout in SourceTree" auto loads atlassian and adds the branch.

What i still want to do is add it as a branch in the "Branches dropdown" of the original. Not even sure if this can be done,
may have to stick to above method only.

DRY411S · Jan 10, 2016

If I upload prices to maddavo, do they get forwarded to EDDN?

Bam · Jan 10, 2016

Thanks for the replies chaps.
I cant really complain its still the best trade tool out there

- - - - - Additional Content Posted / Auto Merge - - - - -

Falcon_D said:
Hello,

How do I get TD and TD helper to stop returning planetary stations and 2.0 commodities?
It's a bit cumbersome to keep entering planetary stations and 2.0 commodities into the avoid list.

Thank you and best regards.

You can click on the little cog icon in TDHelper to modify additional settings and add --no-planet to the run parameters box. This adds --no-planet to the end of any command you send to TD via TDhelper.

DRY411S · Jan 11, 2016

maddavo said:
TDHelper has not been updated yet to handle the bgol/horizons branch version of TradeDangerous which has the --no-planet switch in the buy/sell/run/rares/local/nav commands.

I consider bgol's horizon branch to be the latest version of TD as Oliver (@kfsone) seems to have dropped interest in ED/TD.

Oliver has recently posted on his TradeDangerous group on Facebook. I've encouraged him to come and discuss the future of his wonderful work here too.

NE1 · Jan 11, 2016

maddavo's site seems to be down atm

C:\Program Files Games\Frontier\TradeDangerous\Archives\Atlassian\repo\TDHorizons>trade.py import --plug=maddavo -i
NOTE: Requesting http://www.davek.com.au/td/prices-2d.asp
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 349, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Python34\lib\http\client.py", line 1088, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1126, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1084, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 922, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 857, in send
self.connect()
File "C:\Python34\lib\site-packages\requests\packages\urllib3\connection.py", line 155, in connect
conn = self._new_conn()
File "C:\Python34\lib\site-packages\requests\packages\urllib3\connection.py", line 134, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "C:\Python34\lib\site-packages\requests\packages\urllib3\util\connection.py", line 64, in create_connection
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
File "C:\Python34\lib\socket.py", line 533, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11004] getaddrinfo failed

OhGman · Jan 12, 2016

This is an abbreviated dump of what I get with the following comand line. The unrecognized item error returns thousands.

trade.py import --ignore-unknown -P maddavo -O corrections,stations,exportcsv

NOTE: Importing Corrections
NOTE: Importing Stations
NOTE: 49 ARIETIS/Gaiman Dock (#540) updated in local db: mkt('Y'=>'N')
NOTE: AULIN/Kuo City (#3263) updated in local db: mkt('Y'=>'N')
NOTE: BELALANS/Shoemaker City (#4737) updated in local db: mkt('Y'=>'N')
NOTE: CHAPSUGAIBO/Borrego Port (#7031) updated in local db: mkt('Y'=>'N')

NOTE: Unrecognized system for station DEIVE/O'Leary Ring
NOTE: Unrecognized system for station HIP 51652/Leavitt Mine
NOTE: Unrecognized system for station HIP 51652/Leavitt Mine
NOTE: Unrecognized system for station HIP 51652/Leavitt Mine

NOTE: KAIAKUL/Bliss Gateway (#20652) updated in local db: mkt('Y'=>'N')
NOTE: KAUSHPOOS/Norgay Port (#21304) updated in local db: mkt('Y'=>'N')
NOTE: KHARINDA/Chargaff Hub (#21498) updated in local db: mkt('Y'=>'N')
NOTE: MARIDAL/Dedman Gateway (#27183) updated in local db: mkt('Y'=>'N')

NOTE: Unrecognized system for station NARAKA (ALTERNATE)/Hawking Depot
NOTE: Unrecognized system for station NARAKA (ALTERNATE)/Hawking Depot

NOTE: NIJUNA/Kagawa Port (#29644) updated in local db: mkt('Y'=>'N')
NOTE: RAPA BAO/Thomson Orbital (#32702) updated in local db: mkt('Y'=>'N')
NOTE: RHEA/Carter Port (#32869) updated in local db: mkt('Y'=>'N')
NOTE: D:\kfsoneTD\data\Category.csv re-exported.
NOTE: D:\kfsoneTD\data\Item.csv re-exported.
NOTE: D:\kfsoneTD\data\System.csv re-exported.
NOTE: D:\kfsoneTD\data\Station.csv re-exported.
NOTE: D:\kfsoneTD\data\Ship.csv re-exported.
NOTE: D:\kfsoneTD\data\ShipVendor.csv re-exported.
NOTE: D:\kfsoneTD\data\RareItem.csv re-exported.
NOTE: Requesting http://www.davek.com.au/td/prices-3h.asp
[====================] 503.6KB [ 29.5MB/s] 109% ...
NOTE: Downloaded 5.5MB of gziped data 2.6MB/s
NOTE: 10 stations updated:
SHAPSUGABUS/Abe Dock, LHS 2123/Swanson Orbital, LTT 8750/Luu Vision, SKEGGIKO O/Kuo Terminal, YAKABUGAI/Aristotle Gateway, KAUSALYA/Foda Ring, SHARAMO/Pinto Vista, SAN QIN GU/Sinclair Colony, IMAN CABER/Dobrovolski Station, USDIAE/Forward Vision

WARNING: import.prices:26 ERROR Unrecognized item name: "CERAMIC COMPOSITES"
WARNING: import.prices:27 ERROR Unrecognized item name: "META-ALLOYS"
WARNING: import.prices:33 ERROR Unrecognized item name: "BOOTLEG LIQUOR"
--------------------------
^
3000 +
v
--------------------------
WARNING: import.prices:45529 ERROR Unrecognized item name: "BUILDING FABRICATORS"
WARNING: import.prices:45531 ERROR Unrecognized item name: "GEOLOGICAL EQUIPMENT"
WARNING: import.prices:45535 ERROR Unrecognized item name: "SKIMMER COMPONENTS"
WARNING: import.prices:45536 ERROR Unrecognized item name: "THERMAL COOLING UNITS"

NOTE: Import complete: 643 updated items over 599 stations in 507 systems
NOTE: Import completed.

DRY411S · Jan 12, 2016

OhGman said:
This is an abbreviated dump of what I get with the following comand line. The unrecognized item error returns thousands.

trade.py import --ignore-unknown -P maddavo -O corrections,stations,exportcsv

NOTE: Importing Corrections
NOTE: Importing Stations
NOTE: 49 ARIETIS/Gaiman Dock (#540) updated in local db: mkt('Y'=>'N')
NOTE: AULIN/Kuo City (#3263) updated in local db: mkt('Y'=>'N')
NOTE: BELALANS/Shoemaker City (#4737) updated in local db: mkt('Y'=>'N')
NOTE: CHAPSUGAIBO/Borrego Port (#7031) updated in local db: mkt('Y'=>'N')

NOTE: Unrecognized system for station DEIVE/O'Leary Ring
NOTE: Unrecognized system for station HIP 51652/Leavitt Mine
NOTE: Unrecognized system for station HIP 51652/Leavitt Mine
NOTE: Unrecognized system for station HIP 51652/Leavitt Mine

<snip>

WARNING: import.prices:26 ERROR Unrecognized item name: "CERAMIC COMPOSITES"
WARNING: import.prices:27 ERROR Unrecognized item name: "META-ALLOYS"
WARNING: import.prices:33 ERROR Unrecognized item name: "BOOTLEG LIQUOR"
<snip>

NOTE: Import complete: 643 updated items over 599 stations in 507 systems
NOTE: Import completed.

There are 2 problems here.

You are not importing the new systems, only new stations. Try trade.py import --plug=maddavo --opt=csvonly,systems,exportcsv to get JUST the systems file. Then use your command.
You don't have an up to date list of commodity items. You need to download Item.csv from maddavo and overwrite the Item.csv in your td/data folder.

OhGman · Jan 12, 2016

DRY411S said:
There are 2 problems here.

You are not importing the new systems, only new stations. Try trade.py import --plug=maddavo --opt=csvonly,systems,exportcsv to get JUST the systems file. Then use your command.

You don't have an up to date list of commodity items. You need to download Item.csv from maddavo and overwrite the Item.csv in your td/data folder.

Excellent - Thanks for your help on this. What would be the recommended command line to execute prior to running ED?

DRY411S · Jan 12, 2016

maddavo said:
I have recently been asked about "old prices" that persist at a station for items that are no longer bought or sold at a station. The following is an explanation of the scenario and some advice on how to resolve it.

Firstly, the situation doesn't have to do with crowdsourced data - it can happen with just local data that is updated using the import command. Using the import command, a user adds data for a station - this might be from EliteOCR, ED:Market Connector or Maddavo Market Share data. Then at some later time data is updated for a station via the import command. But a commodity that was listed the first time is now missing. This means that prices get updated for all the commodities except the missing one. The price for the missing item remains in the local data with the original timestamp. This situation can lead to some undesirable trade routes being generated where you are told to deliver a load of something that you can no longer sell for a profit at the station or worse it may be illegal. Then you have to work out what to do with your cargo and generate a whole new trade run. But then the trade run generation may be tainted again causing you to try to buy/sell non-existent/unwanted items.

The TD import command used to delete this old price - essentially overwriting data for a station. But since a few versions ago the import command merges data and there is no option to clobber old data.

So, the price database needs to be 'cleaned' of old prices. On Maddavo Market Share, there is a routine that runs every 3 hours that deletes any price that is 2days older than the latest station price. This has the effect that prices for a station are all timestamped within a 2-day window. Ages ago this timeframe was 14days, then it was reduced to 7days and now it is 2days. This is because we initially were updating data completely manually and people didn't update all the prices for a station so a 14-day window was good. But now the vast majority of price updates are automated so even a 2-day window is probably stretching it. But I digress...

The price 'cleaning' on MMS just makes sure that anyone using that data isn't downloading old prices. BUT it doesn't do anything to the old prices in your local database - you still need to 'clean' your local data. There is no command that does this automatically. You can manually UPDATE prices for a station - I use this command (trade update "SYSTEM/Station" --sub -T) to open Sublime and look at the timestamp column to see if there are any prices that are old compared to the other timestamps. This is extremely tedious and has to be done station-by-station.

Sick of this, I created the following SQL query to 'clean' the local database. It will delete all prices that are 2 days older than the latest price at their respective stations. I use DBBrowser for SQLite to open the TradeDangerous.db file and run the query. The query is:

Code:

DELETE FROM StationItem WHERE StationItem.station_id || '/' || StationItem.item_id IN (SELECT StationItem.station_id || '/' || StationItem.item_id AS Thing FROM StationItem INNER JOIN (SELECT MAX(modified) AS Stationtimestamp, station_id FROM StationItem GROUP BY station_id) AS MyQ ON MyQ.station_id = StationItem.station_id WHERE datetime(modified)<datetime(Stationtimestamp,'-2 day'))

The "-2 day" at the end might be substituted with some other timeframe like "-6 hour".

I am hopeless at Python. Please can someone make a Python command that will do that OR better modify the import command to have the option of running that after an import (or having an option to clobber previous data for a station would achieve the same thing - actually better).

Cheers,
Maddavo

gazelle said:
That's because the maddavo plugin sets "tdenv.mergeImport" to "True".

Normaly that would only happen when you specify "--merge-import" (or "-M") in the commandline, but the plugin always sets it to True.

If you don't want this, just delete the line "tdenv.mergeImport = True" (file: plugins/maddavo_plug.py, Line: 695).

maddavo said:
Aha! Superb.

We have a Horizons related issue here.

If I visit a station with my Horizons install and use EDMC, it sends a list of commodities that includes Horizons items.

If somebody else visits the station later who does not have Horizons and uses EDMC, it sends a shorter list of commodities.

Thus unless I use merge-import, (and my preference is not to use it) I will lose my Horizons updates for all stations that have been subsequently visited by non-Horizons players.

I feel as though we need a --Horizons flag in the code that tells TD whether it should include or ignore Horizons only stations and commodities, in addition/instead of the current bgol branch which filters based on planetary stations.

- - - - - Additional Content Posted / Auto Merge - - - - -

DRY411S said:
We have a Horizons related issue here.

If I visit a station with my Horizons install and use EDMC, it sends a list of commodities that includes Horizons items.

If somebody else visits the station later who does not have Horizons and uses EDMC, it sends a shorter list of commodities.

Thus unless I use merge-import, (and my preference is not to use it) I will lose my Horizons updates for all stations that have been subsequently visited by non-Horizons players.

I feel as though we need a --Horizons flag in the code that tells TD whether it should include or ignore Horizons only stations and commodities, in addition/instead of the current bgol branch which filters based on planetary stations.

Hmm, scrub that please. Wooly thinking by me. The non-Horizons player using EDMC does indeed get the Horizons commodities through the API, and keeps them up to date. It's just the ED code that doesn't display them.

Nevertheless we still need a way of telling TD to ignore 'Horizons commodities' in trades, rather than just the planetary bases.

maddavo · Jan 13, 2016

DRY411S said:
Hmm, scrub that please. Wooly thinking by me. The non-Horizons player using EDMC does indeed get the Horizons commodities through the API, and keeps them up to date. It's just the ED code that doesn't display them.

Nevertheless we still need a way of telling TD to ignore 'Horizons commodities' in trades, rather than just the planetary bases.

I've raised an issue on bitbucket.

Falcon_D · Jan 14, 2016

maddavo said:
TDHelper has not been updated yet to handle the bgol/horizons branch version of TradeDangerous which has the --no-planet switch in the buy/sell/run/rares/local/nav commands.

I consider bgol's horizon branch to be the latest version of TD as Oliver (@kfsone) seems to have dropped interest in ED/TD.

Thank you.

gazelle · Jan 15, 2016

Bam said:
I have been using the horizons branch recently but I cant stop it showing planetary ports when using the --no-planet switch

Fixed with latest version, --no-planet should work now:

Code:

$ trade.py run --fr="HIP 106213" --cap=448 --cr=10000000 --ly=19.34 --empty=25 --pad=L --supply=10000 --hops=2 --jum=1 --age=7 --loop --no-planet -vv
HIP 106213/Tanaka Terminal -> HIP 106213/Tanaka Terminal (score: 637795.957120)
  Load from HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      448 x Imperial Slaves   13.591cr vs   16.228cr, 2 days vs 6 days
  Unload at KURRAE/Camm Market (406ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 1.181.376cr (2.637cr/ton) => 11.181.376cr
  Load from KURRAE/Camm Market (406ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
      448 x Fish                 340cr vs      549cr, 6 days vs 2 days
  Unload at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 93.632cr (209cr/ton) => 11.275.008cr
  ----------------------------------------------------------------------------
Finish at HIP 106213/Tanaka Terminal (120ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 1.275.008cr (1.423cr/ton) => est 11.275.008cr total

maddavo · Jan 17, 2016

Falcon_D said:
Maddavo said:

TDHelper has not been updated yet to handle the bgol/horizons branch version of TradeDangerous which has the --no-planet switch in the buy/sell/run/rares/local/nav commands.

I consider bgol's horizon branch to be the latest version of TD as Oliver (@kfsone) seems to have dropped interest in ED/TD.

Click to expand...

Thank you.

ATTENTION TDHelper users and users of ED v1.5 (non-Horizons)
I just saw the "Run parameters" option in the Misc. Settings. Can't believe I missed that. So, if you put "--no-planet" in there then it will be added to the run command.

CMDR Boozeman · Jan 17, 2016

How about Maddavo's Ship.csv?

It is still pre 1.5/2.0. Is there something big changes in process? The ships are the same on both versions.

DRY411S · Jan 17, 2016

CMDR Boozeman said:
How about Maddavo's Ship.csv?

It is still pre 1.5/2.0. Is there something big changes in process? The ships are the same on both versions.

I believe that this is because some of the ship names have changed (in terms of what EDDN sends to Maddavo). I haven't had any shipvendor updates from Maddavo for a few weeks now.