In-Development TradeDangerous: power-user trade optimizer

[kfsone]

Nice! I was actually about to put in an issue for that so I'd get to it this week. I can worry about making it easier to add items to stations instead.

I have a stack overflow question about making the window scroll with focus with 250 status bounty but no takers. Stupid tkinter. wxPython didn't seem to have python 3 support, guess I'll look at what other options there are.

 

[kfsone]

Just committed 6.1.0:
- Gazelle's "export" subcommand,
- The "update" GUI now tries to make life much easier, and also scrolls properly,
- "update" GUI will now behave nicely when editing a new station (instead of crashing),
- Added "--ignore-unknown" (-i) to import (it was only on buildcache),
- See CHANGES.txt for more.

 

[kfsone]

Ok, there are definitely some duplicate station names in the current game universe. For example, there is a "Padalka Orbital"s in two different systems...

 

[maddavo]

Ok, there are definitely some duplicate station names in the current game universe. For example, there is a "Padalka Orbital"s in two different systems...

Boo, that's problematic.

- - - - - Additional Content Posted / Auto Merge - - - - -

I have made some updates to the market prices share site - it should work much quicker uploading and downloading now.

 

[kfsone]

There are also a few star names that don't "normalize" well:

Camazotz and Cama Zotz,
Hunbinja and Hun Binja,
Popongo and Popong O,
Tengu and Ten Gu

 

[kfsone]

Ok, I'm about 30% of the way thru adding support for station names being unique per system rather than entirely. The csv system now treats multiple "unq:" columns to form a compound unique index, the code that "normalizes" names is a little less aggressive which may mean you have to be more specific with some names. Note that if you need to type a space in a system/station name, you'll probably have to put quotes around it:

trade.py local "1 Hydrae"

When I'm done with this work, it *should* you should be able to leave the spaces out again.

Once it's done I'll also be adding a new "trades" command,

trade.py trades ibootis sol
trade.py trades beagle2 sol

The first will tell you the best trades between all stations in iBootis and all stations in Sol, the second will narrow in on trades from I BOOTIS/Beagle 2 and all stations in Sol.

DEVELOPERS:

I'm replacing "TradeDB.lookupSystem", "TradeDB.lookupStation" and "TradeDB.lookupStationRelaxed" with "lookupPlace". I've committed the first version and made improvements to AmbiguityError, now I just need to go through and retrofit existing calls to use this instead and we should have a nice, simple, singular way of specifying "where I'm at".

I'll couple this with making the run sub-command accept a system rather than a station as a start/end. So you'll be able to say:

trade.py run --cr 1000 --cap 4 --ly 10 --from aulin --to sol --hops 5

and it'll compare all the stations in aulin and sol.

 

[DRY411S]

Personally, I've always enclosed the names in quotes just to give me flexibility if I need to alter the command and be more specific about the system/station. An example would be systems that begin 'LTT<space>'. There's loads (an official IT Project Manager quantity) of them.

 

[kfsone]

Current version is about 60% converted, "via" and "avoid" still use the old mechanisms so you may have to be picky, but the other station/system options on the command line are now swapped over.

 

[Eggplant!]

The current master breaks the update command if you do not supply the -S flag. (If you just provide sell/buy pricing)

 

[kfsone]

The current master breaks the update command if you do not supply the -S flag. (If you just provide sell/buy pricing)

Good catch, thanks - I'm using the GUI (-G) most of the time now, but I've fixed it in 6.1.2.

Also, in 6.1.2 the "import" command can now take a URL to download a .prices file from, and there's a "--maddavo" option. I recommend using it with "-i" for the time being (ignore unknown items)

 

[kfsone]

6.1.2 the 'run' command now also accepts system names as destinations, e.g.

trade.py run ... --to lhs64

this will try all the stations in the destination system instead of giving you an ambiguity error like it used to.

 

[kfsone]

6.2.1 the GUI update tool now saves your updates as "updated.prices" so that you can easily upload the individual station updates.

 

[kfsone]

I have flown through 3 systems this morning that don't appear in maddavo's system.csv. The common factor beween them is that they are Anarchies and/or my rubbish scanner has not found any stations. I mapped their co-ordinates using this link. If you wish to paste them into your system.csv, open the spoiler.

Spoiler

'Cephei Sector NX-U B2-6',-91.96875,0.78125,-29.03125,'Gamma1','2014-11-26 15:00:00'
'Cephei Sector NX-U B2-7',-96.53125,1.3125,-29.09375,'Gamma1','2014-11-26 15:00:00'
'Cephei Sector KM-W C1-24',-99.28125,-1.15625,-23.28125,'Gamma1','2014-11-26 15:00:00'

Please can you tell me how to force trade.py to print out all the steps along the way with a 'nav' argument.

Whoops - I finally got around to adding these!

 

[DRY411S]

I've found maybe 5 or 6 more of these in my part of space. It's having an interesting effect on trade.py. I'm in a Hauler with 18 cargo capacity, which reduces the jump range to something like 6.65. With the systems missing, trade.py is restricted in the routes it can offer.

So really, to optimise the results from trade.py, it's almost as important to add the missing systems with no stations into the databases, as it is to add/update station data.

There's a certain amout of risk in this. You have to remain in the system whilst you use the Galaxy Map to 'triangulate' the system coordinates using maddavo's co-ordinate generator at http://www.davek.com.au/td/ed-systems-master/entry.html

 

[DRY411S]

I'm getting confused about what I need to do to use the newest version. I'm trying NOT to get an update of the .prices file from maddavo's site, and to use my own station.csv and system.csv. The objective is to use TradeDangerous as though I was the only person adding/updating to the known galaxy.

So I've downloaded the new version and before I run it, copy my TradeDangerous database and .prices file across
Quick test for trade.py buy food and it returns only the stations I have input. So far so good, it seems to accept my old database.

Spoiler

>trade.py buy foodStation                                  Cost      Stock
--------------------------------------------------------
AIABIKO/Gooch Terminal                     47  7,763,601
CHEMAKU/Kaku Orbital                       47  5,372,973
ROSS 210/Fernandes De Queiros Dock         47  4,827,318
CHEMAKU/Ewald City                         47  4,457,621
CHEMAKU/Ferguson City                      47  4,457,621
CHEMAKU/Lawson Station                     47  3,890,582
7 ALPHA LACERTAE/Virchow Orbital           47  1,043,409
DARUDNIKEPA/Xuanzang Ring                  47    603,754
7 ALPHA LACERTAE/Leclerc Hub               47    472,255
LTT 16764/Smith Orbital                    47    393,162
BAGALIS/Evans Platform                     47    269,222
BD+49 3937/Kovalevsky Orbital              47    225,204

Then I move my station.csv and system.csv files across and try the same command. I get errors like this one.

>trade.py buy foodTraceback (most recent call last):
  File "trade.py", line 76, in <module>
    main(sys.argv)
  File "trade.py", line 47, in main
    tdb = tradedb.TradeDB(cmdenv, buildLinks=False, includeTrades=False)
  File "tradedb.py", line 307, in __init__
    self.reloadCache()
  File "tradedb.py", line 388, in reloadCache
    importTables=self.importTables
  File "cache.py", line 825, in buildCache
    processImportFile(tdenv, tempDB, Path(importName), importTable)
  File "cache.py", line 707, in processImportFile
    for linein in csvin:
  File "P:\Python34\lib\codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 577: invalid start byte

I've noticed that the first line has a new unq: specification, which I've added, and the station.csv now ends 0.0 rather than just 0, which I've also fixed in my .csv, but this error refuses to go away.

Edit Update: My own debugging proved that the new code does not like my old system.csv file. Has the format changed? Anyway, I fixed it by cutting my new systems off the top of my systems.csv file, and pasting them into the new one!

 

[gazelle]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 577: invalid start byte

The CSV files should be encoded in utf-8. "0xa0" is not a valid utf-8 char (it's non-breaking space in iso-8859-1). Maybe you have an editor which can convert between different charset encoding or search for the "0xa0" and replace it with a normal space.

You also can use "trade.py export" to generate the csv files of the current database which than should be in the correct encoding. Use "--path somewhereelse" if you don't want to overwrite your current csv files, eg: "trade.py export --path misc" will generate all csv files in the subdirectory "misc".

I've noticed that the first line has a new unq: specification, which I've added, and the station.csv now ends 0.0 rather than just 0, which I've also fixed in my .csv, but this error refuses to go away.

"0" or "0.0" doesn't make a difference. It's just the export command always uses decimals for "float" datatype columns.

The "unq" prefix is just for checking unique columns before they are inserted in the database, so you get a better error message instead of an sqlite error.

 

[gazelle]

How I do the updating:

1. Edit the csv files
2. test if I had made any error by building a test.db "trade.py buildcache -f --db misc/test.db"
3. if there are errors correct them, goto 2
4. build the new database "trade.py buildcache -f"
5. update the csv files "trade.py export"

 

[DRY411S]

So working again, a couple of bugs to report.

>trade.py update Borman -GF --all*** INTERNAL ERROR: NOT NULL constraint failed: ShipVendor.station_id
CSV File: data\ShipVendor.csv:2
SQL Query: INSERT INTO ShipVendor (station_id,ship_id,cost) VALUES((SELECT Station.station_id FROM Station INNER JOIN System USING(system_id) WHERE System.name = ? AND Station.name = ?),(SELECT Ship.ship_id FROM Ship WHERE Ship.name = ?),?)
Params: ['LHS 3447', 'Dalton Gateway', 'Asp Explorer', '6661153']

I emptied ShipVendor.csv file to get rid of that problem.

The 2nd bug I want to report is that when using trade.py update station -GF --all on a station that is already in the database, the 'all' commodities are displayed at the top of each commodity section, rather than in the 'correct place'. This is manageable I guess, but if you add prices for a new commodity, it remains in the same 'wrong' position every time the editor is opened (for that station).

By the way, I'm now updating prices with the graphical editor by using Windows Speech recognition. Takes about 2 minutes to add a whole station, and I don't have to use the keyboard at all. Great Job!

 

[Robert C. Simonson]

There's a certain amount of risk in this. You have to remain in the system whilst you use the Galaxy Map to 'triangulate' the system coordinates using maddavo's co-ordinate generator at <link removed>

Did anyone else noticed that in C:\Users\<youruserfolder>\AppData\Local\Frontier_Developments\Products\FORC-FDEV-D-1002\Logs\ there are some netLogYYMMDDHHMM.xx.log files that, amongst other things, list the systems you visit as soon as you jump into them, including name and exact game coordinates?

Like:

{18:36:17} System:21(Aulin) Body:14 Pos: (-22.5641,683.339,729.753)

Edit: I checked the coordinates in Systems.csv and they're wildly different: 'Aulin',-19.6875,32.6875,4.75,'Premium Beta2'... :-(
It doesn't even look they're comparable.

 

[gazelle]

*** INTERNAL ERROR: NOT NULL constraint failed: ShipVendor.station_id
CSV File: data\ShipVendor.csv:2
SQL Query: INSERT INTO ShipVendor (station_id,ship_id,cost) VALUES((SELECT Station.station_id FROM Station INNER JOIN System USING(system_id) WHERE System.name = ? AND Station.name = ?),(SELECT Ship.ship_id FROM Ship WHERE Ship.name = ?),?)
Params: ['LHS 3447', 'Dalton Gateway', 'Asp Explorer', '6661153']

I emptied ShipVendor.csv file to get rid of that problem.

That's because you didn't use the supplied Station.csv file which now includes stations from "LHS 3447". It's a "user error" not a bug

- - - - - Additional Content Posted / Auto Merge - - - - -

{18:36:17} System:21(Aulin) Body:14 Pos: (-22.5641,683.339,729.753)

Edit: I checked the coordinates in Systems.csv and they're wildly different: 'Aulin',-19.6875,32.6875,4.75,'Premium Beta2'... :-(
It doesn't even look they're comparable.

The coordinates in the net log are NOT galaxy system coordinates but relative body coordinates inside the current system.