In-Development TradeDangerous: power-user trade optimizer

DRY411S · Mar 18, 2015

FYI, EliteOCR update has been released that now no longer captures commoditites that have both 0 supply and 0 demand. These type of commodities show up in EliteOCR when you do the screen grab and you happen to have some in your cargo holds.

https://forums.frontier.co.uk/showthread.php?t=68771

- - - - - Additional Content Posted / Auto Merge - - - - -

Serpentstar said:
The reason? Traveling 300 ly to find out they only have 200. I would much rather skip any place with less than, say, 20,000 quantity on hand. This would allow me to use the route longer with lower credit reduction.

bdragon9222 said:
I know that too well. It happens a lot when you are flying to outposts in a python. Sometimes OCRed supplies are at 300 with High Supply, but arriving there and you often see just 160 left. Someone other was faster. And in a python there are approx. 280 tons cargo space, so that really matters in big ships. + prices fluctuate much more with such low quantities

OK. Out of interest, how often does it happen? And do you tend to be trading in very busy places?

bdragon9222 · Mar 18, 2015

@DRY411S
Alot of good prices are at outposts, so during a normal 1-2 hour run it happens at every third or forth station. I don't think it's very busy there. But good prices get you to such places. The main problem is, that sometimes you get into a loop and there is not enough supply to get 2 or 3 times the same commodities from the same station. (ok, could use --unique to avoid it) And refreshing prices after a direct buy doesn't work, because it takes some time to drop the price at your current station.

DRY411S · Mar 18, 2015

Refreshing prices does work if you add the updates into your local data in addition to sending it to EDDN.

bdragon9222 · Mar 18, 2015

DRY411S said:
Refreshing prices does work if you add the updates into your local data in addition to sending it to EDDN.

I know that, but refreshing prices does nothing if you are purchasing these items at the same time. Then there is NO price change.

DRY411S · Mar 18, 2015

Apologies, I thought we were discussing quantity changes, not price changes. If you have a route you have been repeating, and update the prices/quantities each time, eventually you'll be given a different route will you not, because Trade Dangerous does take into account quantities. It will not tell you to pick up 280t of X from Y, if it knows that there is less than 280t of X available.

Snake Man · Mar 19, 2015

I just now (UTC 0310hrs) downloaded TD with git, then Maddavo's system.csv and station.csv, then when I try to use TD to update prices I get this:

NOTE: Rebuilding cache file: this may take a moment.
*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id
CSV File: /home/<yup>/tradedangerous/data/Station.csv:96
SQL Query: INSERT INTO Station (system_id,name,ls_from_star,blackmarket,max_pad_size,market,shipyard,modified,outfitting,rearm,refuel,repair) VALUES((SELECT System.system_id FROM System WHERE System.name = ?),?,?,?,?,?,?,?,?,?,?,?)
Params: ['17 DRACONIS', 'Paradiso Outpost', '11330', 'Y', 'L', 'Y', 'Y', '2015-03-04 18:54:22', 'Y', 'Y', 'Y', 'Y']

Alaaarm!

maddavo · Mar 19, 2015

Snake Man said:
I just now (UTC 0310hrs) downloaded TD with git, then Maddavo's system.csv and station.csv, then when I try to use TD to update prices I get this:

NOTE: Rebuilding cache file: this may take a moment.
*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id
CSV File: /home/<yup>/tradedangerous/data/Station.csv:96
SQL Query: INSERT INTO Station (system_id,name,ls_from_star,blackmarket,max_pad_size,market,shipyard,modified,outfitting,rearm,refuel,repair) VALUES((SELECT System.system_id FROM System WHERE System.name = ?),?,?,?,?,?,?,?,?,?,?,?)
Params: ['17 DRACONIS', 'Paradiso Outpost', '11330', 'Y', 'L', 'Y', 'Y', '2015-03-04 18:54:22', 'Y', 'Y', 'Y', 'Y']

Alaaarm!

I see a station crept in for a system that isn't in the database (the system exists but the coords aren't known in EDSC yet). I'm not sure how this happened because stations shouldn't be able to be created for systems that aren't in the database! Mmm

Anyway, I've deleted the station and regenerated the Station.csv . It should work now. You may need to do a trade buildcache -f -i before doing an import.

- - - - - Additional Content Posted / Auto Merge - - - - -

I have received a msg from @LeoDR with screenshots of a market with new (non-rare) items - "AI Relics" and "Antiquities" in a new category "Salvage". I have updated the Item.csv and Category.csv files in anticipation of some prices coming through. If anyone submits prices for these new items and they don't go into the system then pls let me know.

DRY411S · Mar 19, 2015

maddavo said:
I see a station crept in for a system that isn't in the database (the system exists but the coords aren't known in EDSC yet). I'm not sure how this happened because stations shouldn't be able to be created for systems that aren't in the database! Mmm

Anyway, I've deleted the station and regenerated the Station.csv . It should work now. You may need to do a trade buildcache -f -i before doing an import.

I think that this can happen if somebody finds a new system (with station(s)) and adds it their system.csv and station.csv locally. They can upload station.csv to you but not system.csv.

maddavo said:
I have received a msg from @LeoDR with screenshots of a market with new (non-rare) items - "AI Relics" and "Antiquities" in a new category "Salvage". I have updated the Item.csv and Category.csv files in anticipation of some prices coming through. If anyone submits prices for these new items and they don't go into the system then pls let me know.

I assumed that this is some sort of bug or a new feature. There's a few more in the EliteOCR topic I think. All the items being found as 'Salvage' are normally associated with missions, and you can scoop them before you actually get the missions to deliver them. Previously I think you had to sell them in a Black Market. Perhaps that's no longer true, I haven't encountered this situation yet, I cannot help but think this is one of those situations where the commodities don't appear unless you are carrying them. EliteOCR does not support them yet.

Edit: Found the link with more commodities. https://forums.frontier.co.uk/showthread.php?t=68771&page=97&p=1935044&viewfull=1#post1935044

Minyeni · Mar 19, 2015

Any chance of the shipvendor subcommand getting a --near SYSTEM option? The idea would be that I could answer the question "OK, where's the nearest shipyard to PAND that sells the Vulture?" Something like this:

Code:

c:\> trade shipvendor --near PAND Vulture

DRY411S · Mar 19, 2015

trade.py shipvendor is for editing the ships that are on sale at stations.

To find out where to buy a ship you use trade.py buy. It has the functionality you require.

For example:

Code:

>trade.py buy vulture
Station                                StnLs B/mkt Pad
------------------------------------------------------
ALIGNAK/Piserchia Port                 1.07K    No Lrg
EZ AQUARII/Magnus Gateway                498    No Lrg
KOKARY/Faraday Enterprise                 95    No Lrg
KOKARY/Gidzenko Terminal                  52   Yes Lrg
LFT 926/Meredith City                    362    No Lrg
LTT 16764/Virtanen Ring                    ?     ?   ?
MIDGARD/Rontgen Port                       ?   Yes Lrg
SHINRARTA DEZHRA/Jameson Memorial        346    No Lrg
YZ CETI/Clement Orbital                1.23K    No Lrg



>trade.py buy vulture --near etadraconis
Station                DistLy      StnLs B/mkt Pad
--------------------------------------------------
ALIGNAK/Piserchia Port  12.54      1.07K    No Lrg


>trade.py buy vulture --near etadraconis --ly 100
Station                   DistLy      StnLs B/mkt Pad
-----------------------------------------------------
ALIGNAK/Piserchia Port     12.54      1.07K    No Lrg
LTT 16764/Virtanen Ring    71.74          ?     ?   ?
MIDGARD/Rontgen Port       80.85          ?   Yes Lrg
EZ AQUARII/Magnus Gateway  95.55        498    No Lrg
YZ CETI/Clement Orbital    99.34      1.23K    No Lrg

Minyeni · Mar 20, 2015

DRY411S said:
trade.py shipvendor is for editing the ships that are on sale at stations.

Awesome - I clearly missed that memo. Thanks!

Tromador · Mar 21, 2015

Couple of new things popping up here via Maddavo since 1.2

Code:

import.prices:17469 WARNING Unrecognized item name: "Ai Relics"
import.prices:17470 WARNING Unrecognized item name: "Antiquities"
import.prices:30351 WARNING Unrecognized item name: "Ai Relics"
import.prices:30352 WARNING Unrecognized item name: "Antiquities"
import.prices:34608 WARNING Unrecognized item name: "Ai Relics"
import.prices:34609 WARNING Unrecognized item name: "Antiquities"

Can we add a new line to category.csv

Code:

'Salvage'

And corresponding lines into item.csv

Code:

'Salvage','Ai Relics',1
'Salvage','Antiquities',2

Coyote21 · Mar 21, 2015

New issue with TD

I'm not sure if this should go into bitbucket or here but here goes... after doing a very long tour of a new system and adding all the stations/black markets/services/shipvendors, on the very last station of 7, TD started to do this.

Code:

trade.py station -u mbutas/dozo --bm y
NOTE: Rebuilding cache file: this may take a moment.
*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id
CSV File: C:\EliteDangerous\tradedangerous\data\Station.csv:11262
SQL Query: INSERT INTO Station (system_id,name,ls_from_star,blackmarket,max_pad_size,market,shipyard,modified,outfitting,rearm,refuel,repair) VALUES((SELECT System.system_id FROM System WHERE System.name = ?),?,?,?,?,?,?,?,?,?,?,?)
Params: ['PANDEMONIUM', 'Hirayama Port', '0', 'N', 'L', 'Y', 'Y', '2015-02-25 01:31:48', 'Y', 'Y', 'Y', 'Y']

I tried manually deleting the offending line 11262 from Station.csv but that did'nt fix the problem it just started complaining about the line that took it's place.

Please tell me I haven't just lost the last hour of work?

EDIT: Just checked on Maddavo's market share and most of the data got uploaded and parsed correctly. Made changes by hand so nothing lost but I'd still like to know how/why this happened so I don't do it again.

gazelle · Mar 21, 2015

Code:

*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id

This means it didn't find the system.

Code:

Params: ['PANDEMONIUM', 'Hirayama Port', '0', 'N', 'L', 'Y', 'Y', '2015-02-25 01:31:48', 'Y', 'Y', 'Y', 'Y']

The first one is the systemname: 'PANDEMONIUM'

So, it looks like 'PANDEMONIUM' wasn't in your System.csv file (it was recently corrected from 'PANDAMONIUM').

Coyote21 · Mar 21, 2015

gazelle said:
Code:

*** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id

This means it didn't find the system.

Code:

Params: ['PANDEMONIUM', 'Hirayama Port', '0', 'N', 'L', 'Y', 'Y', '2015-02-25 01:31:48', 'Y', 'Y', 'Y', 'Y']

The first one is the systemname: 'PANDEMONIUM'

So, it looks like 'PANDEMONIUM' wasn't in your System.csv file (it was recently corrected from 'PANDAMONIUM').

That might have been the issue, I only tried to delete two lines from Stations.csv both where PANDAMONIUM but I stopped before removing all of them.

I d/l a new System.csv and Station.csv from Maddavo and now it's all fixed. The new csv files use PANDEMONIUM.

Thanks for replying

CMDR Boozeman · Mar 22, 2015

Hi,

The data from EDDN are sometimes like ... Why some people just not check the data they send?

Just removed two stations from star Cupis (No OCR-mistake, just 2 stations that are not there) Also Removing some OCR-mistakes here and there. Ah, and there is a Station on Volungu, where you can get Beryllium at 870cr/ton

Tromador · Mar 22, 2015

There's obviously something wrong in the data - but is there any way to find out what, so I can fix it in some way?

Code:

D:\Gadgets\TradeDangerous>trade.py run --ca 40 --ly 17.07 --cr 6500 --fr shin/jam --mgpt 2500 --ls-max 2500 --hops 10 --debug -vvv
# Loaded 163987 selling values
# Loaded 503456 selling values
# origPlace: Station: SHINRARTA DEZHRA/Jameson Memorial
# Using all available destinations
# Via: set()
# Hop 1...
Traceback (most recent call last):
  File "D:\Gadgets\TradeDangerous\trade.py", line 102, in <module>
    main(sys.argv)
  File "D:\Gadgets\TradeDangerous\trade.py", line 76, in main
    results = cmdenv.run(tdb)
  File "D:\Gadgets\TradeDangerous\commands\commandenv.py", line 80, in run
    return self._cmd.run(results, self, tdb)
  File "D:\Gadgets\TradeDangerous\commands\run_cmd.py", line 1038, in run
    newRoutes = calc.getBestHops(routes, restrictTo=restrictTo)
  File "D:\Gadgets\TradeDangerous\tradecalc.py", line 839, in getBestHops
    items = [i for i in trade if i.costCr <= credits]
  File "D:\Gadgets\TradeDangerous\tradecalc.py", line 839, in <listcomp>
    items = [i for i in trade if i.costCr <= credits]
AttributeError: 'list' object has no attribute 'costCr'

kfsone · Mar 23, 2015

Tromador said:

There's obviously something wrong in the data - but is there any way to find out what, so I can fix it in some way?

Code:

D:\Gadgets\TradeDangerous>trade.py run --ca 40 --ly 17.07 --cr 6500 --fr shin/jam --mgpt 2500 --ls-max 2500 --hops 10 --debug -vvv
# Loaded 163987 selling values
# Loaded 503456 selling values
# origPlace: Station: SHINRARTA DEZHRA/Jameson Memorial
# Using all available destinations
# Via: set()
# Hop 1...
Traceback (most recent call last):
  File "D:\Gadgets\TradeDangerous\trade.py", line 102, in <module>
    main(sys.argv)
  File "D:\Gadgets\TradeDangerous\trade.py", line 76, in main
    results = cmdenv.run(tdb)
  File "D:\Gadgets\TradeDangerous\commands\commandenv.py", line 80, in run
    return self._cmd.run(results, self, tdb)
  File "D:\Gadgets\TradeDangerous\commands\run_cmd.py", line 1038, in run
    newRoutes = calc.getBestHops(routes, restrictTo=restrictTo)
  File "D:\Gadgets\TradeDangerous\tradecalc.py", line 839, in getBestHops
    items = [i for i in trade if i.costCr <= credits]
  File "D:\Gadgets\TradeDangerous\tradecalc.py", line 839, in <listcomp>
    items = [i for i in trade if i.costCr <= credits]
AttributeError: 'list' object has no attribute 'costCr'

Fixed, grab the latest code.

- - - - - Additional Content Posted / Auto Merge - - - - -

Tromador said:

Couple of new things popping up here via Maddavo since 1.2

Code:

import.prices:17469 WARNING Unrecognized item name: "Ai Relics"
import.prices:17470 WARNING Unrecognized item name: "Antiquities"
import.prices:30351 WARNING Unrecognized item name: "Ai Relics"
import.prices:30352 WARNING Unrecognized item name: "Antiquities"
import.prices:34608 WARNING Unrecognized item name: "Ai Relics"
import.prices:34609 WARNING Unrecognized item name: "Antiquities"

Can we add a new line to category.csv

Code:

'Salvage'

And corresponding lines into item.csv

Code:

'Salvage','Ai Relics',1
'Salvage','Antiquities',2

Added. Get latest code, data/Item.csv, data/Category.csv and do "trade.py buildcache -fiq"

kfsone · Mar 23, 2015

v6.14.4 Mar 22 2015
. (gazelle) Bernd's "getsation.sh" script (see scripts/README.txt)
. (kfsone) Fixed #209 Exception when you couldn't afford something,
. (kfsone) Fixed #210 Exception when --from not reached,
. (kfsone) Added "Salvage" Category,
. (kfsone) Added "AI Relics" and "Antiquities" Items,
- You will want to "trade.py buildcache -f -i"

...

Coyote21 said:
New issue with TD

I'm not sure if this should go into bitbucket or here but here goes... after doing a very long tour of a new system and adding all the stations/black markets/services/shipvendors, on the very last station of 7, TD started to do this.

Code:

trade.py station -u mbutas/dozo --bm y NOTE: Rebuilding cache file: this may take a moment. *** INTERNAL ERROR: NOT NULL constraint failed: Station.system_id CSV File: C:\EliteDangerous\tradedangerous\data\Station.csv:11262 SQL Query: INSERT INTO Station (system_id,name,ls_from_star,blackmarket,max_pad_size,market,shipyard,modified,outfitting,rearm,refuel,repair) VALUES((SELECT System.system_id FROM System WHERE System.name = ?),?,?,?,?,?,?,?,?,?,?,?) Params: ['PANDEMONIUM', 'Hirayama Port', '0', 'N', 'L', 'Y', 'Y', '2015-02-25 01:31:48', 'Y', 'Y', 'Y', 'Y']

I tried manually deleting the offending line 11262 from Station.csv but that did'nt fix the problem it just started complaining about the line that took it's place.

Please tell me I haven't just lost the last hour of work?

EDIT: Just checked on Maddavo's market share and most of the data got uploaded and parsed correctly. Made changes by hand so nothing lost but I'd still like to know how/why this happened so I don't do it again.

What's happening here is that when it's trying to look up the Station.system_id I can't find the System. It's looking up "Pandemonium" and not finding it.

What happened is that TD renamed "PANDAMONIUM" to "PANDEMONIUM". You may need to do a "trade.py buildcache -fi" to fix this, but your full change set should comprise:

. 'PANDAMONIUM' in data/System.csv becomes 'PANDEMONIUM',
. 'PANDAMONIUM' in data/Station.csv becomes 'PANDEMONIUM',
. A system entry in corrections.py,
. Delete the PANDAMONIUM entries in data/TradeDangerous.prices ok they don't conflict with the PANDEMONIUM prices you've since downloaded.

- - - - - Additional Content Posted / Auto Merge - - - - -

Minyeni said:
Any chance of the shipvendor subcommand getting a --near SYSTEM option? The idea would be that I could answer the question "OK, where's the nearest shipyard to PAND that sells the Vulture?" Something like this:

Code:

c:\> trade shipvendor --near PAND Vulture

You can use the "buy" command to do that

Code:

$ trade.py buy --help
./trade.py: TradeDangerous help

usage: trade.py buy [--quantity QUANTITY] [--near NEAR] [--ly N.NN]
                    [--limit LIMIT] [--pad-size PADSIZES]
                    [--one-stop | --price-sort | --stock-sort] [--gt N]
                    [--lt N] [-h] [--debug] [--detail] [--quiet]
                    [--db DBFILENAME] [--cwd CWD] [--link-ly MAXSYSTEMLINKLY]
                    name [name ...]

Required Arguments:
  name                  Items or Ships to look for.

Optional Switches:

For reference, the current command list:

Code:

$ trade.py --help
./trade.py: TradeDangerous provides a set of trade database facilities for Elite:Dangerous.

Usage: ./trade.py <command>

Where <command> is one of:

  buildcache    Build TradeDangerous cache file from sources
  buy           Find places to buy a given item within range of a given
                 station.
  export        CSV exporter for TradeDangerous database.
  import        Imports price data from a '.prices' format file.  Previous
                 data for the stations included in the file is removed, but
                 other stations are not affected.
  local         Calculate local systems.
  market        Lists items bought/sold at a given station.
  nav           Calculate a route between two systems.
  olddata       Show oldest data in database.
  rares         Find rares near your current local.
  run           Calculate best trade run.
  sell          Find places to sell a given item within range of a given
                 station.
  shipvendor    List, add or update available ships to a station
  station       Add (or update) a station entry
  trade         Find places to buy a given item within range of a given
                 station.
  update        Update prices for a station.

CMDR Boozeman · Mar 23, 2015

Hi,

I quess that there's several commanders with some bontys for minor factions here and there, but no recollection what star system this must be claimed. Is there any possibility or even mind to implement the factions somehow example to the station information? Then you can do some another magic trick with trade.py

How the new data will impact the performance or db size?

The data is not available on OCR-tools, so the validity of the data is our hands as users of TD