In-Development TradeDangerous: power-user trade optimizer

MarkAusten · Jun 23, 2018

Tromador said:
It's there, but it's not in .md format, I need to convert it and not gotten around to it yet.

I have no idea about virtualenv - these were Oliver's original instructions and again to stress - I know next to nothing about MacOs, or I would rewrite these instructions myself. If you can write up a correct and current set of MacOS instructions I would be very glad to include them.

Ah, I understand. Sorry for the misunderstanding.

I'll find some time to write the instructions out for Mac OS and send them to you once done.

eyeonus · Jun 23, 2018

Tromador said:
Yup. Sounds good - with the following correction.

It was I who stated the default setting of 0.5 (not 0.05) as per the documentation. Thus you need to either match the existing document, or if that is not a sensible setting with your new curve, then we need to update the doc.

Edit: Might not be a bad idea to give advice in the documentation as to some reasonable upper and lower limits for this setting.

Code:

--ls-penalty N.NN --lsp N.NN DEFAULT: 0.5 Reduces the score of routes by this percentage for every 1000ls you have to travel to stations, which helps prioritize routes with a shorter supercruise time. e.g. --ls-penalty 2.5 --lsp=0 (disables this feature)

Ah. Oops. Fixed.

As to reasonable limits, I agree, but I have no idea what they would be.

Tromador · Jun 23, 2018

MarkAusten said:
I'll find some time to write the instructions out for Mac OS and send them to you once done.

That will be very helpful, thanks.

eyeonus said:
As to reasonable limits, I agree, but I have no idea what they would be.

Lol - it's your equation. Well, I guess it doesn't matter too much. If a user suddenly finds all stations past 100ls are being ignored, then they probably can figure out that they overdid it.

eyeonus · Jun 23, 2018

Here's the same run command 'trade.py run --fr="Liu Bajara/Silva Terminal" --cap=36 --ins=156182 --cr=4450263 --ly=12.65 --empty=14.3 --hops=6 --age=30 --loop -vvv'

with the default penalty of 0.5:

Code:

Liu Bajara/Silva Terminal -> Liu Bajara/Silva Terminal (score: 68456.528457)
Start CR:  4,294,081
Hops    :          6
Jumps   :         10
Gain CR :    411,624
Gain/Hop:     68,604
Final CR:  4,705,705
  Load from Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Textiles/Military Grade Fabrics           942cr vs    1,634cr, 4 hrs vs 3 days, total:     33,912cr
  Unload at Dalannovale/Siodmak Prospect (1.77Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y) => Gain 24,912cr (692cr/ton) => 4,475,175cr
  Load from Dalannovale/Siodmak Prospect (1.77Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y):
       36 x Metals/Gold                             8,725cr vs   11,104cr, 3 days vs 25 days, total:    314,100cr
  Unload at CD-27 5409/Shipton Terminal (52Kls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y) => Gain 85,644cr (2,379cr/ton) => 4,560,819cr
  Load from CD-27 5409/Shipton Terminal (52Kls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y):
       36 x Metals/Indium                           4,078cr vs    6,681cr, 25 days vs 15 hrs, total:    146,808cr
  Unload at BD-20 2665/Curie Port (567ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 93,708cr (2,603cr/ton) => 4,654,527cr
  Load from BD-20 2665/Curie Port (567ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Textiles/Military Grade Fabrics           780cr vs    1,980cr, 15 hrs vs 22 days, total:     28,080cr
  Unload at Nyikamana/Daley Prospect (84Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y) => Gain 43,200cr (1,200cr/ton) => 4,697,727cr
  Load from Nyikamana/Daley Prospect (84Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y):
       36 x Metals/Gold                             8,741cr vs   11,104cr, 22 days vs 25 days, total:    314,676cr
  Unload at CD-27 5409/Shipton Terminal (52Kls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y) => Gain 85,068cr (2,363cr/ton) => 4,782,795cr
  Load from CD-27 5409/Shipton Terminal (52Kls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y):
       36 x Industrial Materials/Superconductors    5,352cr vs    7,549cr, 25 days vs 4 hrs, total:    192,672cr
  Unload at Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 79,092cr (2,197cr/ton) => 4,861,887cr
  ----------------------------------------------------------------------------
Finish at Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 411,624cr (1,905cr/ton) => est 4,861,887cr total

and with the penalty set to 5:

Code:

Liu Bajara/Silva Terminal -> Liu Bajara/Silva Terminal (score: 67896.948402)
Start CR:  4,294,081
Hops    :          6
Jumps   :         12
Gain CR :    399,620
Gain/Hop:   66,603.3
Final CR:  4,693,701
  Load from Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Chemicals/Explosives                          340cr vs      930cr, 4 hrs vs 5 days, total:     12,240cr
  Unload at Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y) => Gain 21,240cr (590cr/ton) => 4,471,503cr
  Load from Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y):
       21 x Metals/Palladium                           11,813cr vs   13,980cr, 5 days vs 5 days, total:    248,073cr
       15 x Metals/Gold                                 8,984cr vs   10,796cr, 5 days vs 5 days, total:    134,760cr
  Unload at Kivah/Sadi Carnot Ring (43ls, BMk:Y, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 72,687cr (2,019.08cr/ton) => 4,544,190cr
  Load from Kivah/Sadi Carnot Ring (43ls, BMk:Y, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Medicines/Basic Medicines                     314cr vs    4,527cr, 5 days vs 22 hrs, total:     11,304cr
  Unload at Sosong/Chu Orbital (415ls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y) => Gain 151,668cr (4,213cr/ton) => 4,695,858cr
  Load from Sosong/Chu Orbital (415ls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y):
       19 x Metals/Gold                                 9,375cr vs   10,953cr, 22 hrs vs 4 days, total:    178,125cr
       17 x Industrial Materials/Insulating Membrane   10,085cr vs   11,664cr, 22 hrs vs 4 days, total:    171,445cr
  Unload at UBV 8670/Hurley Dock (195ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 56,825cr (1,578.47cr/ton) => 4,752,683cr
  Load from UBV 8670/Hurley Dock (195ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Foods/Food Cartridges                         124cr vs      753cr, 4 days vs 5 days, total:      4,464cr
  Unload at Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y) => Gain 22,644cr (629cr/ton) => 4,775,327cr
  Load from Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y):
       21 x Metals/Palladium                           11,813cr vs   14,004cr, 5 days vs 4 hrs, total:    248,073cr
       15 x Metals/Gold                                 8,984cr vs   10,887cr, 5 days vs 4 hrs, total:    134,760cr
  Unload at Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 74,556cr (2,071cr/ton) => 4,849,883cr
  ----------------------------------------------------------------------------
Finish at Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 399,620cr (1,850cr/ton) => est 4,849,883cr total

and at 100:

Code:

Liu Bajara/Silva Terminal -> Liu Bajara/Silva Terminal (score: 92475.634707)
Start CR:  4,294,081
Hops    :          6
Jumps   :         12
Gain CR :    399,620
Gain/Hop:   66,603.3
Final CR:  4,693,701
  Load from Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Chemicals/Explosives                          340cr vs      930cr, 4 hrs vs 5 days, total:     12,240cr
  Unload at Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y) => Gain 21,240cr (590cr/ton) => 4,471,503cr
  Load from Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y):
       21 x Metals/Palladium                           11,813cr vs   13,980cr, 5 days vs 5 days, total:    248,073cr
       15 x Metals/Gold                                 8,984cr vs   10,796cr, 5 days vs 5 days, total:    134,760cr
  Unload at Kivah/Sadi Carnot Ring (43ls, BMk:Y, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 72,687cr (2,019.08cr/ton) => 4,544,190cr
  Load from Kivah/Sadi Carnot Ring (43ls, BMk:Y, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Medicines/Basic Medicines                     314cr vs    4,527cr, 5 days vs 22 hrs, total:     11,304cr
  Unload at Sosong/Chu Orbital (415ls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y) => Gain 151,668cr (4,213cr/ton) => 4,695,858cr
  Load from Sosong/Chu Orbital (415ls, BMk:Y, Pad:M, Plt:N, Shp:N, Out:N, Ref:Y):
       19 x Metals/Gold                                 9,375cr vs   10,953cr, 22 hrs vs 4 days, total:    178,125cr
       17 x Industrial Materials/Insulating Membrane   10,085cr vs   11,664cr, 22 hrs vs 4 days, total:    171,445cr
  Unload at UBV 8670/Hurley Dock (195ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 56,825cr (1,578.47cr/ton) => 4,752,683cr
  Load from UBV 8670/Hurley Dock (195ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y):
       36 x Foods/Food Cartridges                         124cr vs      753cr, 4 days vs 5 days, total:      4,464cr
  Unload at Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y) => Gain 22,644cr (629cr/ton) => 4,775,327cr
  Load from Ardh/Nusslein-Volhard Platform (1.15Kls, BMk:N, Pad:M, Plt:N, Shp:N, Out:Y, Ref:Y):
       21 x Metals/Palladium                           11,813cr vs   14,004cr, 5 days vs 4 hrs, total:    248,073cr
       15 x Metals/Gold                                 8,984cr vs   10,887cr, 5 days vs 4 hrs, total:    134,760cr
  Unload at Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) => Gain 74,556cr (2,071cr/ton) => 4,849,883cr
  ----------------------------------------------------------------------------
Finish at Liu Bajara/Silva Terminal (462ls, BMk:N, Pad:L, Plt:N, Shp:Y, Out:Y, Ref:Y) gaining 399,620cr (1,850cr/ton) => est 4,849,883cr total

So, looks like the default isn't enough to definitely get rid of the routes involving >4Kls systems- it has me going to the same one twice in this route- but at 5 it is. In this particular case, 100 didn't do much. Same route just higher score. I ran the route with some other penalty values as well. For this case, a penalty <=3.96 gives the higher profit route with the very distant station, whereas >=3.97 gives the slightly less profitable route with no long distance stations.

Obviously this is only one case, but if we do enough test cases, I'm sure we can find a good pair of limits.

eyeonus · Jun 23, 2018

Tromador said:
Lol - it's your equation. Well, I guess it doesn't matter too much. If a user suddenly finds all stations past 100ls are being ignored, then they probably can figure out that they overdid it.

Well, yeah, but I'm not a mathematician, and figuring out good numbers would require either a lot of examples, or some very complicated math, or both.

GuruNot · Jun 23, 2018

Evening All, looking for some help.

Pulled down the latest commit from : https://github.com/eyeonus/Trade-Dangerous , I have python 3.6.5 installed. Got EDDBLink from https://github.com/eyeonus/EDDBlink , run : trade.py import -P eddblink -O clean

and get the following:

NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "D:\Games\TradeDangerous\data\TradeDangerous.prices" file - no price data.
NOTE: Processing Upgrades: Start time = 2018-06-23 21:11:20.768303
NOTE: Finished processing Upgrades. End time = 2018-06-23 21:11:20.811279
NOTE: Downloading file: 'index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded 0.7MB of gziped data 13.4MB/s
NOTE: Processing Ships: Start time = 2018-06-23 21:11:21.779001
NOTE: Finished processing Ships. End time = 2018-06-23 21:11:21.816974
NOTE: Processing Systems: Start time = 2018-06-23 21:15:11.343150
NOTE: Finished processing Systems. End time = 2018-06-23 21:15:15.748630
NOTE: Processing Stations, this may take a bit: Start time = 2018-06-23 21:15:15.940581
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
NOTE: Finished processing Stations. End time = 2018-06-23 21:33:13.155357
NOTE: Processing Categories and Items: Start time = 2018-06-23 21:33:24.941585
NOTE: Finished processing Categories and Items. End time = 2018-06-23 21:33:24.962598
NOTE: D:\Games\TradeDangerous\data\Category.csv exported.
NOTE: D:\Games\TradeDangerous\data\Item.csv exported.
NOTE: D:\Games\TradeDangerous\data\Ship.csv exported.
NOTE: D:\Games\TradeDangerous\data\ShipVendor.csv exported.
NOTE: D:\Games\TradeDangerous\data\Station.csv exported.
NOTE: D:\Games\TradeDangerous\data\System.csv exported.
NOTE: D:\Games\TradeDangerous\data\Upgrade.csv exported.
NOTE: D:\Games\TradeDangerous\data\UpgradeVendor.csv exported.
NOTE: Downloading file: 'listings.csv'.
NOTE: Requesting http://elite.ripz.org/files/listings.csv
Traceback (most recent call last):
File "D:\Games\TradeDangerous\trade.py", line 104, in <module>
main(sys.argv)
File "D:\Games\TradeDangerous\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "D:\Games\TradeDangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "D:\Games\TradeDangerous\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "D:\Games\TradeDangerous\plugins\eddblink_plug.py", line 812, in run
if self.downloadFile(LISTINGS, self.listingsPath) or self.getOption("force"):
File "D:\Games\TradeDangerous\plugins\eddblink_plug.py", line 162, in downloadFile
self.dataPath / path,
File "D:\Games\TradeDangerous\transfers.py", line 146, in download
raise Exception("Remote server replied with invalid content-length.")
Exception: Remote server replied with invalid content-length.

What am I missing ??

eyeonus · Jun 23, 2018

GuruNot said:
Evening All, looking for some help.

Pulled down the latest commit from : https://github.com/eyeonus/Trade-Dangerous , I have python 3.6.5 installed. Got EDDBLink from https://github.com/eyeonus/EDDBlink , run : trade.py import -P eddblink -O clean

and get the following:

NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "D:\Games\TradeDangerous\data\TradeDangerous.prices" file - no price data.
NOTE: Processing Upgrades: Start time = 2018-06-23 21:11:20.768303
NOTE: Finished processing Upgrades. End time = 2018-06-23 21:11:20.811279
NOTE: Downloading file: 'index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded 0.7MB of gziped data 13.4MB/s
NOTE: Processing Ships: Start time = 2018-06-23 21:11:21.779001
NOTE: Finished processing Ships. End time = 2018-06-23 21:11:21.816974
NOTE: Processing Systems: Start time = 2018-06-23 21:15:11.343150
NOTE: Finished processing Systems. End time = 2018-06-23 21:15:15.748630
NOTE: Processing Stations, this may take a bit: Start time = 2018-06-23 21:15:15.940581
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
NOTE: Finished processing Stations. End time = 2018-06-23 21:33:13.155357
NOTE: Processing Categories and Items: Start time = 2018-06-23 21:33:24.941585
NOTE: Finished processing Categories and Items. End time = 2018-06-23 21:33:24.962598
NOTE: D:\Games\TradeDangerous\data\Category.csv exported.
NOTE: D:\Games\TradeDangerous\data\Item.csv exported.
NOTE: D:\Games\TradeDangerous\data\Ship.csv exported.
NOTE: D:\Games\TradeDangerous\data\ShipVendor.csv exported.
NOTE: D:\Games\TradeDangerous\data\Station.csv exported.
NOTE: D:\Games\TradeDangerous\data\System.csv exported.
NOTE: D:\Games\TradeDangerous\data\Upgrade.csv exported.
NOTE: D:\Games\TradeDangerous\data\UpgradeVendor.csv exported.
NOTE: Downloading file: 'listings.csv'.
NOTE: Requesting http://elite.ripz.org/files/listings.csv
Traceback (most recent call last):
File "D:\Games\TradeDangerous\trade.py", line 104, in <module>
main(sys.argv)
File "D:\Games\TradeDangerous\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "D:\Games\TradeDangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "D:\Games\TradeDangerous\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "D:\Games\TradeDangerous\plugins\eddblink_plug.py", line 812, in run
if self.downloadFile(LISTINGS, self.listingsPath) or self.getOption("force"):
File "D:\Games\TradeDangerous\plugins\eddblink_plug.py", line 162, in downloadFile
self.dataPath / path,
File "D:\Games\TradeDangerous\transfers.py", line 146, in download
raise Exception("Remote server replied with invalid content-length.")
Exception: Remote server replied with invalid content-length.

What am I missing ??

Apparently something is wrong with the file it tried to download. (TD asked for the size of the file and got no answer.) Try again. If it doesn't work, go to http://elite.ripz.org/ and download the files manually, placing them in '<TD install>/data/eddb'.

GuruNot · Jun 23, 2018

eyeonus said:
Apparently something is wrong with the file it tried to download. (TD asked for the size of the file and got no answer.) Try again. If it doesn't work, go to http://elite.ripz.org/ and download the files manually, placing them in '<TD install>/data/eddb'.

That worked, cheers.

gazelle · Jun 24, 2018

eyeonus said:
Ah. Oops. Fixed

Nope, the default for --lsp is 0.006 (0.6 / 100) and set here: https://github.com/eyeonus/Trade-Dangerous/blob/master/commands/run_cmd.py#L140

eyeonus · Jun 24, 2018

gazelle said:
Nope, the default for --lsp is 0.006 (0.6 / 100) and set here: https://github.com/eyeonus/Trade-Dangerous/blob/master/commands/run_cmd.py#L140

Which also doesn't match the documentation. Fixed.

Avi0013 · Jun 24, 2018

Tromador said:
Are we sure about that new ls-per curve? It just tried to have me run hydrogen (fair, the station really only had that or biowaste) to a place the better part of 200K ls from the star.

Additionally, no such place as Daruwutja/Acropolis - not sure where that comes from, though this must be a separate issue.
I guess if I had remembered to include an age term, neither of these would have popped up, but I did forget and so here they are.

Problem 1 - Please don't send me to Amikul/Benz with hydrogen. The documentation states that there is a default ls penalty of 0.5, is that actually the case?
Problem 2 - Stale data about a place which doesn't exist (I presume it must have at one time).

Antiproblem 1 - It really is running a LOT quicker with the new "fill the hold" algorithm. Good job guys.

trade.py run --from paesan/abel --to cubeo --hops 10 -v --pla=YN? --credits=35000000 --capacity=150 --ly-per=27.98 --empty-ly=32.03 --insurance=15500000 --unique --progress --pad-size=L

Code:

* Hop 1: .........1 origins * Hop 2: .......109 origins * Hop 3: .....1,542 origins NOTE: Pruned 1 origins too far from any end stations * Hop 4: .....5,056 origins NOTE: Pruned 767 origins too far from any end stations * Hop 5: ....10,361 origins NOTE: Pruned 4428 origins too far from any end stations * Hop 6: ....13,562 origins NOTE: Pruned 9924 origins too far from any end stations * Hop 7: ....12,174 origins NOTE: Pruned 12099 origins too far from any end stations * Hop 8: .....7,262 origins NOTE: Pruned 9599 origins too far from any end stations * Hop 9: .....3,165 origins NOTE: Pruned 5943 origins too far from any end stations * Hop 10: .......652 origins Paesan/Abel Landing -> Cubeo/Medupe City (score: 4252233.733499) Load from Paesan/Abel Landing: 150 x Chemicals/Hydrogen Fuel (@84cr), Dock at Ambikul/Benz Settlement Load from Ambikul/Benz Settlement: 150 x Medicines/Basic Medicines (@359cr), Dock at Daruwutja/Acropolis Load from Daruwutja/Acropolis: 145 x Weapons/Reactive Armour (@227cr), 5 x Weapons/Personal Weapons (@416cr), Dock at 64 Piscium/Miller's Inheritance Load from 64 Piscium/Miller's Inheritance: 150 x Metals/Thallium (@2126cr), Dock at LFT 174/Berezovoy Gateway Load from LFT 174/Berezovoy Gateway: 150 x Medicines/Basic Medicines (@343cr), Dock at Mari/Guidoni Hub Load from Mari/Guidoni Hub: 150 x Foods/Tea (@1375cr), Dock at Gl 606.1 B/Harvey Ring Load from Gl 606.1 B/Harvey Ring: 149 x Slavery/Imperial Slaves (@14230cr), 1 x Machinery/Building Fabricators (@1272cr), Dock at Kipgeninka/Jordan Enterprise Load from Kipgeninka/Jordan Enterprise: 150 x Metals/Tantalum (@2245cr), Dock at Kamocan/Gasparis Vision Load from Kamocan/Gasparis Vision: 150 x Slavery/Imperial Slaves (@14196cr), Dock at Djakah/Al-Khujandi Hub Load from Djakah/Al-Khujandi Hub: 91 x Metals/Gold (@9881cr), 59 x Metals/Palladium (@12814cr), Dock at Cubeo/Medupe City Finish Cubeo/Medupe City + 4,249,183cr (2,832cr/ton)=> 39,249,183cr

The new penalty curve does not have the problems of the old one, but the how much of it to apply may need to be recalibrated. In the old curve, it went negative very quickly, and so 0.006 may have been enough. Now, with the new sigmoid application, further stations may need more application of the penalty to register. We may have to just trial and error it. Also, compare with passing a 0 which should return approximate best money.

Below is a graph comparing the multipliers for distances from 0 to 200Kls. Note that the current default really no longer penalizes distances, as it caps out just around 1. It asymptotically approaches (1 - lsp) so the default 0.006 means that even the worst case wont have a score multiplier less than 0.994, so it the money is 0.6% better, strap on your gas cans. We may have to start passing a higher penalty, or perhaps lobby @eyeonus for a stiffer default like 0.2 or something. Hope that helps!

Avi0013 · Jun 24, 2018

eyeonus said:
Excuse me, who was it that said the default penalty is 0.05?

Code:

# Penalty is expressed as percentage, reduce it to a multiplier if tdenv.lsPenalty: lsPenalty = tdenv.lsPenalty / 100 else: lsPenalty = 0

Let me just go ahead and fix that.

The default PERCENTAGE in run_cmd.py was 0.6. That got divided by 100 as you point out for a multiplier of 0.006. You just changed it from 0.006 to 0.005. Not going to matter much. If you want a 50% penalty, the default in run_cmd.py needs to be 50, not 0.5

eyeonus said:
The boost part of the penalty equation, yes.

But apparently the default penalty is not 0.05, but 0, as you can see in the code snippet above, which means unless an lspenalty is passed on the command line, ls distance isn't factored in at all. So, I reverted the boost and fixed the default setting. If it still turns out that high distance stations are showing up a lot, I'll revisit this.

I'm also pretty doubtful that a low distance hop would affect the likelihood of seeing a high distance hop, so if I do revisit it, I'll probably try sharpening the drop curve before I do any adjusting to the boost curve.

I do not believe that is true, since there is a default penalty pased in run_cmd.py. you have to pass 0 to clear it. Try it!

Avi0013 · Jun 24, 2018

Tromador said:
It was I who stated the default setting of 0.5 (not 0.05) as per the documentation. Thus you need to either match the existing document, or if that is not a sensible setting with your new curve, then we need to update the doc.

That's correct. Y'all are conflating PERCENTAGES with NOMINAL MULTIPLIERS. The latter is 1/100 of the former, so 0.5% = 0.005.

Avi0013 · Jun 24, 2018

eyeonus said:
Well, yeah, but I'm not a mathematician, and figuring out good numbers would require either a lot of examples, or some very complicated math, or both.

As you can see from the image below, terminal multiplier (which is always 1 - lsp) is approximately reached soon after 4kls.

Also note that the current formula gives a boost to closer stations inversely proprotional to the penalty. So passing a 99.4 as the lsp will significantly boost closer stations much more than passing 0.5 would (purple vs. red).

Is that what we want?

Tromador · Jun 24, 2018

eyeonus said:
Apparently something is wrong with the file it tried to download. (TD asked for the size of the file and got no answer.) Try again. If it doesn't work, go to http://elite.ripz.org/ and download the files manually, placing them in '<TD install>/data/eddb'.

If "something is wrong with the file" then downloading it with a browser won't make a difference, it's the same file.

...and it's fine (and always has been) for me.

Has anyone else had any issues?

MarkAusten · Jun 24, 2018

Tromador said:
If "something is wrong with the file" then downloading it with a browser won't make a difference, it's the same file.

...and it's fine (and always has been) for me.

Has anyone else had any issues?

I occasionally get issues with downloading the file. This is the Internet after all, so problems will occasionally happen. I just wait a while and try again. Usually downloads correctly second or third time if I'm having problems.

Tromador · Jun 24, 2018

Ok cool.

If this happens again, can who sees it please let me know and at what time, so I can check the logfile. I have looked for something based on the output Gurunot posted, but can't see anything - then again, there might be a timezone difference and I'm looking in the wrong place.

Also, why does it need 3 accesses to download a file, Eyeonus. I understood it was twice - once to check/compare the datestamp and once to download the file if necessary. Here is a recent logfile from my access (IP address redacted), the first few files aren't needed, but get accessed twice. The only file getting downloaded is listings-live.csv, which is access 3 times. Why two accesses, even if we don't need the file?

This isn't a problem, more to satisfy my curiosity and understanding.

Oh... whilst I am at it and without wishing to reopen this can of worms... now I look at the log - if eddblink_plug.py doesn't need requests, then why is "python-requests/2.18.4" downloading the file - is this called from a method inside the main TD code?

Code:

1.2.3.4 - - [24/Jun/2018:22:29:00 +0100] "GET /files/modules.json HTTP/1.1" 200 299285 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:00 +0100] "GET /files/modules.json HTTP/1.1" 200 299285 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:00 +0100] "GET /files/systems_populated.jsonl HTTP/1.1" 200 21520925 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/systems_populated.jsonl HTTP/1.1" 200 21520925 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/stations.jsonl HTTP/1.1" 200 119007525 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/stations.jsonl HTTP/1.1" 200 119007525 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/commodities.json HTTP/1.1" 200 101708 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/commodities.json HTTP/1.1" 200 101708 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings.csv HTTP/1.1" 200 156558666 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings.csv HTTP/1.1" 200 156558666 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings-live.csv HTTP/1.1" 200 2293455 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings-live.csv HTTP/1.1" 200 2293455 "-" "Python-urllib/3.6"
1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings-live.csv HTTP/1.1" 200 590141 "-" "python-requests/2.18.4"

eyeonus · Jun 25, 2018

Tromador said:
Ok cool.

If this happens again, can who sees it please let me know and at what time, so I can check the logfile. I have looked for something based on the output Gurunot posted, but can't see anything - then again, there might be a timezone difference and I'm looking in the wrong place.

Also, why does it need 3 accesses to download a file, Eyeonus. I understood it was twice - once to check/compare the datestamp and once to download the file if necessary. Here is a recent logfile from my access (IP address redacted), the first few files aren't needed, but get accessed twice. The only file getting downloaded is listings-live.csv, which is access 3 times. Why two accesses, even if we don't need the file?

This isn't a problem, more to satisfy my curiosity and understanding.

Oh... whilst I am at it and without wishing to reopen this can of worms... now I look at the log - if eddblink_plug.py doesn't need requests, then why is "python-requests/2.18.4" downloading the file - is this called from a method inside the main TD code?

Code:

1.2.3.4 - - [24/Jun/2018:22:29:00 +0100] "GET /files/modules.json HTTP/1.1" 200 299285 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:00 +0100] "GET /files/modules.json HTTP/1.1" 200 299285 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:00 +0100] "GET /files/systems_populated.jsonl HTTP/1.1" 200 21520925 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/systems_populated.jsonl HTTP/1.1" 200 21520925 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/stations.jsonl HTTP/1.1" 200 119007525 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/stations.jsonl HTTP/1.1" 200 119007525 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/commodities.json HTTP/1.1" 200 101708 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/commodities.json HTTP/1.1" 200 101708 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings.csv HTTP/1.1" 200 156558666 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings.csv HTTP/1.1" 200 156558666 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings-live.csv HTTP/1.1" 200 2293455 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings-live.csv HTTP/1.1" 200 2293455 "-" "Python-urllib/3.6" 1.2.3.4 - - [24/Jun/2018:22:29:01 +0100] "GET /files/listings-live.csv HTTP/1.1" 200 590141 "-" "python-requests/2.18.4"

One to check the file is available, one to check the timestamp, one to download the file.

Check file is available:

Code:

            try:
                request.urlopen(BASE_URL + urlTail)
            except:
                # If Tromador's mirror fails for whatever reason,
                # fallback to download direct from EDDB.io
                self.options["fallback"] = True

Get timestamp of file:

Code:

dDL = request.urlopen(url).getheader("Last-Modified").split(' ')

Download file using TD's transfers module:

Code:

transfers.download( self.tdenv, url, self.dataPath / path, )

I think I can knock off one of the checks, give me a bit.

eyeonus · Jun 25, 2018

Tromador said:
If "something is wrong with the file" then downloading it with a browser won't make a difference, it's the same file.

...and it's fine (and always has been) for me.

Has anyone else had any issues?

TD refuses to download a file when it gets "None" for content-length. Probably because then it can't tell how long it will take to download the file and it would break the progress bar. The file itself was fine, the header containing the file-size was borked.

eyeonus · Jun 25, 2018

I've knocked out one of the requests to server, and spent way too much time figuring out that the default urllib.request module doesn't have any easy way to show a progress indicator.

Any case, pushed to both solo plugin and eye-TD