In-Development TradeDangerous: power-user trade optimizer

Legotho · Dec 20, 2014

In a similar vein, is there any mechanism for renaming (or deleting) incorrect spellings?
I've just realised that I've uploaded correct data for Herzfeld Landing in system GCRV 4654, but I'm still getting routes planned using the obsolete data of Herzefeld Landing (with an extra e!)

I've "fixed" the problem by editing the data for the old, incorrect station name, and setting the prices to 0, which will at least mean that it's so totally uninteresting nobody will ever be sent there again. But it'd be cleaner to delete it entirely.

AngusRuffnut · Dec 20, 2014

Legotho said:
In a similar vein, is there any mechanism for renaming (or deleting) incorrect spellings?
I've just realised that I've uploaded correct data for Herzfeld Landing in system GCRV 4654, but I'm still getting routes planned using the obsolete data of Herzefeld Landing (with an extra e!)

I've "fixed" the problem by editing the data for the old, incorrect station name, and setting the prices to 0, which will at least mean that it's so totally uninteresting nobody will ever be sent there again. But it'd be cleaner to delete it entirely.

just let maddavo know and he will fix it in the files.

maddavo · Dec 20, 2014

DRY411S said:
If more people are using EliteOCR to grab prices data, that also captures the Demand as well as the supply. Perhaps that's one of the reasons? I wonder how big the file would be without the demand data? There are not any tools I know that are using it.

The way the file is formatted, there would be spaces instead of numbers so the size would be the same. What makes it big is the number of stations and prices which just grows. It is up to 4Mb now and I see updates are coming in much quicker. I suspect with new players and everyone using EliteOCR it is easier to update prices so it's all snowballing. I'm considering purging older data from before 30th Nov. Surely that's not relevant now

- - - - - Additional Content Posted / Auto Merge - - - - -

Legotho said:
In a similar vein, is there any mechanism for renaming (or deleting) incorrect spellings?
I've just realised that I've uploaded correct data for Herzfeld Landing in system GCRV 4654, but I'm still getting routes planned using the obsolete data of Herzefeld Landing (with an extra e!)

I've "fixed" the problem by editing the data for the old, incorrect station name, and setting the prices to 0, which will at least mean that it's so totally uninteresting nobody will ever be sent there again. But it'd be cleaner to delete it entirely.

If there are any duplicates or bad data then pls let me know so I can filter/block it.

hitchhiker54 · Dec 20, 2014

Is there any plan to be able to filter trade by available landing pad size? I see that data is not currently in stations.csv, so I'd imagine a fairly arduous crowd sourcing effort would be needed :s

kfsone · Dec 21, 2014

Dave: Part of the problem is people submitting similar but different names - you have quite a few duplicates in there. For instance, check the stations you have in Tyr...

This is one possible problem with trusting user-submitted data: we make mistakes (c.f. I screwed up "Ron Hubbard" because Rob Hubbard wrote awesome tunes and he and I hung out on C|Net). There are other cases caused by Frontier's typos. Sometimes people correct them (search for 'HANGAR' in your .csv, and notice that each one has a corresponding 'HANGER' entry because that's actually what Frontier used)

The other problem is just the steady growth in station numbers. Supply/Demand doesn't affect us because the format puts spaces for those in the first place.

You *could* enable on-the-fly compression, and I could check for it and decompress the file.

But I think the best thing to do is going to be to switch to a json format.

We can get it down from 3Mb to under 1Mb that way, and probably get it down to a couple hundred kb for a full dump if you enable lightweight compression.

- - - - - Additional Content Posted / Auto Merge - - - - -

hitchhiker54 said:
Is there any plan to be able to filter trade by available landing pad size? I see that data is not currently in stations.csv, so I'd imagine a fairly arduous crowd sourcing effort would be needed :s

Would love to incorporate this and black market data. The solution is to have a ternary: (Y, N, ?) with ? as the default.

- - - - - Additional Content Posted / Auto Merge - - - - -

Legotho said:
In a similar vein, is there any mechanism for renaming (or deleting) incorrect spellings?
I've just realised that I've uploaded correct data for Herzfeld Landing in system GCRV 4654, but I'm still getting routes planned using the obsolete data of Herzefeld Landing (with an extra e!)

I've "fixed" the problem by editing the data for the old, incorrect station name, and setting the prices to 0, which will at least mean that it's so totally uninteresting nobody will ever be sent there again. But it'd be cleaner to delete it entirely.

Ha - thanks for that, I'd spotted it earlier this evening and added a corrections.py entry for it. I didn't realize it had only just been fixed tho so I think I marked the old one as DELETED.

- - - - - Additional Content Posted / Auto Merge - - - - -

Work in-progress: JSON prices and on-the-fly addition of systems/stations while processing the json data.

Here's an expanded example of a station price entry. 'b' and 's' are 'buying' and 'selling' and they compact down to a single integer in the case of a price with no level information. If level information is present, they expand into an array, e.g "b":[2407,20610,3] indicates the item is bought for 2407cr, there is 20610 units of demand and its level 3 (high).

But the best thing is the inclusion if detailed system data, which allows me to add the system/station if its not in the database.

What I still have to figure out is how to deal with people making typos and creating new stations; you have "Sol / Someplace HangAr" but someone else is submitting data under the name "Sol / Somplace HangEr".

Writing a detection for this rule would be easy except that there are places where stations genuinely have off-by-one names.
Anyway - still working on it

Code:

{
  "m": "2014-12-19 16:47:00",
  "src": "td/price-json",
  "stn": {
    "ls": 0,
    "name": "Abraham Lincoln"
  },
  "items": {
    "Reactive Armour": {
      "b": [
        2407,
        20610,
        3
      ]
    },
    "Tea": {
      "b": [
        1729,
        131770,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Non-Lethal Weapons": {
      "b": [
        2069,
        12129,
        2
      ]
    },
    "Platinum": {
      "b": [
        19418,
        104630,
        2
      ]
    },
    "Coffee": {
      "b": [
        1537,
        58099,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Basic Medicines": {
      "b": [
        404,
        49893,
        2
      ]
    },
    "Biowaste": {
      "b": 11,
      "s": [
        14,
        34779,
        3
      ]
    },
    "Fish": {
      "b": [
        533,
        438253,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Clothing": {
      "b": [
        443,
        544225,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Animal Meat": {
      "b": [
        1537,
        139436,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Power Generators": {
      "b": [
        644,
        68089,
        2
      ]
    },
    "Synthetic Meat": {
      "b": [
        370,
        148491,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Food Cartridges": {
      "b": [
        234,
        75804,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Grain": {
      "b": [
        304,
        1049364,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Palladium": {
      "b": [
        14007,
        133513,
        2
      ]
    },
    "Progenitor Cells": {
      "b": [
        7313,
        30373,
        2
      ]
    },
    "Hydrogen Fuel": {
      "b": 159,
      "m": "2014-12-19 16:46:00"
    },
    "Water Purifiers": {
      "b": [
        388,
        119966,
        2
      ]
    },
    "Fruit And Vegetables": {
      "b": [
        429,
        209633,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Gold": {
      "b": [
        9831,
        135829,
        2
      ]
    },
    "Silver": {
      "b": [
        5306,
        1396917,
        2
      ]
    },
    "Domestic Appliances": {
      "b": [
        691,
        213807,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Liquor": {
      "b": [
        808,
        46359,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Beer": {
      "b": [
        256,
        497350,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Consumer Technology": {
      "b": [
        7100,
        56084,
        2
      ],
      "m": "2014-12-19 16:46:00"
    },
    "Wine": {
      "b": [
        354,
        535859,
        2
      ]
    }
  },
  "cmd": "unknown",
  "sys": {
    "pos": [
      0,
      0,
      0
    ],
    "name": "Sol",
    "mul": 32
  }
}

DRY411S · Dec 21, 2014

How will this affect the format of the .prices file kfsone?

DRY411S · Dec 21, 2014

maddavo said:
It is up to 4Mb now and I see updates are coming in much quicker. I suspect with new players and everyone using EliteOCR it is easier to update prices so it's all snowballing.

I wonder how much data corruption this is going to cause? It's possible to put EliteOCR into a lazy mode and assume that it doesn't make any OCR mistakes. But it does seems seem to struggle with 3,5,6,8 and 9. On my PC anyway, and not all the time, just in particular number sequences.

maddavo said:
I'm considering purging older data from before 30th Nov. Surely that's not relevant now

All data is relevant. And TD lets us filter out aged data if we wish. It's relevant in the sense that it helps to find stations that buy/sell certain things. Sure the prices and quantities may not be correct....

kfsone · Dec 21, 2014

DRY411S said:
How will this affect the format of the .prices file kfsone?

Both formats will be supported.

maddavo · Dec 21, 2014

kfsone said:
Dave: Part of the problem is people submitting similar but different names - you have quite a few duplicates in there. For instance, check the stations you have in Tyr...

This is one possible problem with trusting user-submitted data: we make mistakes (c.f. I screwed up "Ron Hubbard" because Rob Hubbard wrote awesome tunes and he and I hung out on C|Net). There are other cases caused by Frontier's typos. Sometimes people correct them (search for 'HANGAR' in your .csv, and notice that each one has a corresponding 'HANGER' entry because that's actually what Frontier used)

The other problem is just the steady growth in station numbers. Supply/Demand doesn't affect us because the format puts spaces for those in the first place.

You *could* enable on-the-fly compression, and I could check for it and decompress the file.

But I think the best thing to do is going to be to switch to a json format.

We can get it down from 3Mb to under 1Mb that way, and probably get it down to a couple hundred kb for a full dump if you enable lightweight compression.

I've gone through the database and fixed up a few stations that I could see. If anyone finds any more then let me know. Yeah, I also think they're going to always crop up so we just have to manage them manually.

I've purged pre-Gamma data and going forward will do another purge in a few weeks to take us from release. During the week I'll also work on a 48hr file and a 6hr file. I'll also look at compression.

When we lock down the json format then I'll get going on producing the json version. But more pressing at the moment is updates to/from EDDN which is a bit of a learning curve for me.

psuamier · Dec 21, 2014

DRY411S said:
I wonder how much data corruption this is going to cause? It's possible to put EliteOCR into a lazy mode and assume that it doesn't make any OCR mistakes. But it does seems seem to struggle with 3,5,6,8 and 9. On my PC anyway, and not all the time, just in particular number sequences.

EliteOCR has big issues with bad contrast in conjunction with low resolution. Unfortunately they recommend to use normal screenshots. My guess is that a lot of people don't have (or don't use) 1080P displays (myself included). Unfortunately EOCR is not able to parse two large resolution files (MemoryError). It looks like > 1080P is required for a most-of-the-time issue free scanning. What could be tried is to change the default HUD color to have more contrast and retry with a low-resolution screenshot.

I believe manually typing right now is nearly quite as fast to scanning + proof-reading. The main advantage could be that if you have 99,99% scan results the number of wrong data is reduced. I often had wrong trade results due to my own mistakes in the past. So in the long run I think EOCR at least is more reliable to manual input.

When ED market code finally is mostly stable I hope that the number of changes to be done on trade data goes down, so EOCR will only be seldom needed.

kfsone · Dec 21, 2014

maddavo said:
I've gone through the database and fixed up a few stations that I could see. If anyone finds any more then let me know. Yeah, I also think they're going to always crop up so we just have to manage them manually.

I've purged pre-Gamma data and going forward will do another purge in a few weeks to take us from release. During the week I'll also work on a 48hr file and a 6hr file. I'll also look at compression.

When we lock down the json format then I'll get going on producing the json version. But more pressing at the moment is updates to/from EDDN which is a bit of a learning curve for me.

If you browse back thru the TD source history, you'll find the old EMDN code

Gorudo · Dec 21, 2014

Quick question here. When I'm using OCR do I have to set the sytem name or is that just for the BPC program?
If i'm adding a new station do I just manually enter it in my station.csv like normal, scan the prices, hit trade dangerous export, then move the import.prices file generated to TD directory, then "trade.py import import.prices"?
On a side note the OCR scanner is really improved. I ran it for the first time today and absolutely no errors.
I was once sent to a federation outpost to buy slaves instead of silver!

gazelle · Dec 21, 2014

Gorudo said:
Quick question here. When I'm using OCR do I have to set the sytem name or is that just for the BPC program?

You need to set the system name. The station name alone is not unique.

Gorudo said:
If i'm adding a new station do I just manually enter it in my station.csv like normal, scan the prices, hit trade dangerous export, then move the import.prices file generated to TD directory, then "trade.py import import.prices"?

Yes, that's it. You don't need to move the file, you can just give the complete path. Or you can set the "Export Directory" in the settings to your TD directory and the import.prices will be generated there.

kfsone · Dec 22, 2014

UPDATE: v6.2.4 Checked in

. LOTS of Station Data (Maddavo, Gazelle, Kfsone, many others, thank you),
. Added "Black Market" column to station table (Y = Confirmed Yes, N = Confirmed No, ? = Only Buddah Knows),
. "local" command (with -v to show stations) will now show black market status,
. Ship data streamlined so it's easier to add ships (see data/ShipVendor.csv),
. Performance improvements,
. Early version of JSON code in-case anyone wants to comment,
. Early version of "add-station" command (which now modifies cache AND .csv file), misc/add-station.py, USE WITH CAUTION,
. Gazelle: bash completion for trade.py,
. Gazelle: Overhauled the csv export tool

- - - - - Additional Content Posted / Auto Merge - - - - -

The big goal is going to be improving the "import" system to be incremental. Right now import is destructive, and that's just not going to cut it for any of us.

I'd also like to try and get the Update UI improved (and working properly on the Mac). But I could really use some help there - I'm a lousy UI programmer, and I've only ever written two UIs in Python. You've seen one, and the other was an internal tool at Blizzard for testing the mobile chat API.

psuamier · Dec 22, 2014

Navigation is broken:

Code:

kfsone-tradedangerous-ce7ca724008c>trade.py nav --ly-per 14.14 Orrere "39 Tauri"
System                   JumpLy
-------------------------------
ORRERE                     0.00
USZAA                      2.00
ARQUE                     17.47
VUCUB HUAN                 9.45
CRUCIS SECTOR FM-V B2-1   20.05
CRUCIS SECTOR FM-V B2-4   11.25
ER 8                      12.66
LHS 336                   17.28
LHS 2447                   6.21
MILDEPTU                  17.64
G 99-49                   14.90
ROSS 33                   16.04
ERNE                      13.11
39 TAURI                  16.04

Even with --ly-per "10" I still get jumps > 14 LY.

Clanos · Dec 22, 2014

kfsone said:
UPDATE: v6.2.4 Checked in

<snip>

C:\Trade>trade.py buildcache -f
NOTE: Rebuilding cache file: this may take a moment
C:\Trade\trade.py: Station.csv:1630 Station 'Julian Gateway' is marked as DELETED and should not be used.

I was wondering if someone on the older version has put prices in for Julian Gateway and the new version now sees prices for it and rejects it instead of skipping it/handling it.

I haven't looked at the prices, I guess I'll try chopping it up and investigate a little further.

Clanos · Dec 22, 2014

Clanos said:
C:\Trade>trade.py buildcache -f
NOTE: Rebuilding cache file: this may take a moment
C:\Trade\trade.py: Station.csv:1630 Station 'Julian Gateway' is marked as DELETED and should not be used.

I was wondering if someone on the older version has put prices in for Julian Gateway and the new version now sees prices for it and rejects it instead of skipping it/handling it.

I haven't looked at the prices, I guess I'll try chopping it up and investigate a little further.

No, that's not it at all. I thought there were double references to Julian Gateway for Zeta Aquilae in the station.csv, but there isn't. Oh well, over to you.

Naruby · Dec 22, 2014

psuamier said:
Navigation is broken:

Even with --ly-per "10" I still get jumps > 14 LY.

I'm having the same issue. My workaround is subs tracing 3 LY form my max jumps range.

Clanos · Dec 22, 2014

Clanos said:
No, that's not it at all. I thought there were double references to Julian Gateway for Zeta Aquilae in the station.csv, but there isn't. Oh well, over to you.

I got there in the end by using a fresh directory, rather than trying to copy over the previous version.

Clanos · Dec 22, 2014

Combat Stabalisers in Bhil Mina/Mohmand Gateway need to be purged. At least, the station wasn't buying them (and they had no entry for them at all - so I'm assuming it was an auto-OCR error by someone) when I visited it.

This makes me wonder how/if this is being handled pre-EDMN/EDDN integration. I've updated that station and gone through the latest prices file from the madavo site and the entry is still there, albeit a different datestamp to my upload. I can remove it by hand, but perhaps Madovo's upload script on his site could list prices it has for a station that you have not uploaded, so they may be flagged or removed at that point. Perhaps multiple confirmations from different people removes the entry entirely from the db or something.

I also noted that it was painful to integrate rare pricing, maybe this isn't worthwhile, as rare pricing is fairly well understood to be based on distance from source.