Yuk. Any idea when local DB will be king?
In-Development TradeDangerous: power-user trade optimizer
- Thread starter kfsone
- Start date
Yuk. Any idea when local DB will be king?
One weird thing I'm seeing is people sending what appears to be a single OCR screenshot surrounded by deletes. My hunch is these are people who are importing a single screenshot into TD and then exporting to you. TD does updates station-at-a-time so when the import a single screenshot, they essentially delete all the other prices at that station, and then they send you an explicit "delete all these items" when they export from TD to you.
If anyone happens to be doing this, any chance you could share the thinking behind it?
Also, he doesn't want Silver or Uranium but takes Gold.
Code:./trade.py run --fr=GCRV4654/Herzfeld --to=branglal/skripochka --cap=100 --cr=800000 --ly=12.5 --hops=1 -vv GCRV 4654/Herzfeld Landing -> BRANGLAL/Skripochka Gateway (score: 41113.152000) Load from GCRV 4654/Herzfeld Landing (2483ls/star, No/bm, Med/pad): 100 x Cobalt 524cr each, 52400cr total,data from 21 hrs vs 22 hrs Jump GCRV 4654 -> LHS 1933 -> BRANGLAL Unload at BRANGLAL/Skripochka Gateway (882ls/star, No/bm, Lrg/pad) => Gain 41100cr (411cr/ton) => 841100cr ---------------------------------------------------------------------------- Finish at BRANGLAL/Skripochka Gateway (882ls/star, No/bm, Lrg/pad) gaining 41100cr => est 841100cr total ./trade.py run --fr=GCRV4654/Herzfeld --to=branglal/skripochka --cap=100 --cr=900000 --ly=12.5 --hops=1 -vv GCRV 4654/Herzfeld Landing -> BRANGLAL/Skripochka Gateway (score: 152348.736000) Load from GCRV 4654/Herzfeld Landing (2483ls/star, No/bm, Med/pad): 100 x Gold 8792cr each, 879200cr total,data from 21 hrs vs 22 hrs Jump GCRV 4654 -> LHS 1933 -> BRANGLAL Unload at BRANGLAL/Skripochka Gateway (882ls/star, No/bm, Lrg/pad) => Gain 152300cr (1523cr/ton) => 1052300cr ---------------------------------------------------------------------------- Finish at BRANGLAL/Skripochka Gateway (882ls/star, No/bm, Lrg/pad) gaining 152300cr => est 1052300cr total
Ok - grab 8.3.4 - thanks for catching this!
-Oliver
I'm giving this a shot for the first time and I'm absolutely loving it so far - thanks for the time you've put into developing this!
I have a minor usability request though: When going through the station update UI on my Windows system (trade.py update <blah>) I can tab through the various lines to quickly enter data. When I hit the end of the list, if I hit tab again it'll jump back to the top. But if I use Shift+Tab from the top, it won't jump back to the bottom again.
I tend to slog through the data-entry a bit too fast and accidentally hit tab one too many times, and find myself having to reach for my mouse and scroll all the way down to the bottom of the list to find the last field any time I do this. Could you make it so Shift+Tab will jump to the very bottom of the list if invoked while at the top (so, inverse behavior to what tab does now when you're at the bottom)?
I have a minor usability request though: When going through the station update UI on my Windows system (trade.py update <blah>) I can tab through the various lines to quickly enter data. When I hit the end of the list, if I hit tab again it'll jump back to the top. But if I use Shift+Tab from the top, it won't jump back to the bottom again.
I tend to slog through the data-entry a bit too fast and accidentally hit tab one too many times, and find myself having to reach for my mouse and scroll all the way down to the bottom of the list to find the last field any time I do this. Could you make it so Shift+Tab will jump to the very bottom of the list if invoked while at the top (so, inverse behavior to what tab does now when you're at the bottom)?
Grabbed 8.3.4, it's fixed. Thank you.
Great feedback - the best way to make sure I don't forget (since I just finished my coding run for the day) will be to create a little ticket on the issue tracker:
http://kfs.org/td/issues/
(might motivate me to also fix the thing where it doesn't select the level value when you're done typing in a level so you wind up with a '?' stuck infront of what you retype)
http://kfs.org/td/issues/
(might motivate me to also fix the thing where it doesn't select the level value when you're done typing in a level so you wind up with a '?' stuck infront of what you retype)
Well, I'm not even using OCR, personally, but I recall when I first started using trading tools overall, I ran across a web based one that worked pretty well in its time (it's way outdated and broken now, but the UX it provided is influencing my own work on a UI for TD)... that allowed updating individual item prices, rather than full station, allowing someone to just throw in data for the few things they were looking at (the usual high markup items) while not spending all day adjusting/verifying the rest of the items that either hadn't changed any meaningful amount or, generally, just weren't far enough off of average to catch their attention. Someone using OCR might very well be doing exactly that, just grabbing, say, the gold/platinum/palladium subset, ignoring the rest, because that's the commodities that mattered to them at that moment (and, potentially, without clearly understanding that it kills off the data for the rest of the station). All conjecture, though. I also have a feeling that those who approach it that way don't actually follow threads like this one very closely, and often likely push out data with that beloved DCR Oerp.
When I see someone do something unexpected with my code, I tend to assume culpability. So far as I know, running the data through TD to export it to, say, maddavos, is a lot more work than shipping it straight out of the ocr tool.
My hunch is they are either using TD as some kind of filter or there is some piece of black magic that is leading folks to believe that ocr -> td import -> td export -> upload is the easiest path for them. Maybe one of Mad's outtages lead them to reverse their ordering or something.
Take a look at what TD is how, how it's grown, how I've interacted with the thread, and you'll see why I'm interested in why people are doing this
Incidentally, in the database prices are individually stamped; it's just the current import code and the cache builder that blat things.
I've just downloaded the current zip file from the repository (https://bitbucket.org/kfsone/tradedangerous/get/24ab97191bb1.zip), unzipped it, and imported from maddavo (using -vvvvv -w for extra detail!)
The import errors out with:
Looking through import.prices, I find:
The @ TJURINAS/Herndon Dock line is 293836, so the two Hydrogen Fuel lines are numbers 293838 and 293839 as per the error message.
The import errors out with:
Code:
...
# NEW STATION: TIRIS/CHANG-DIAZ ORBITAL
# NEW STATION: TJAKIRI/OFFUTT DOCK
# NEW STATION: TJAPAHO/WEISS LEGACY
# NEW STATION: TJIWUAR/FELICE STATION
# NEW STATION: TJIWUAR/HUDSON TERMINAL
# NEW STATION: TJIWUAR/MARQUES RING
# NEW STATION: TJUPALI/HARRISON CITY
# NEW STATION: TJURINAS/HERNDON DOCK
./trade.py: import.prices:293839 ERROR Second occurrance of item "Hydrogen Fuel", previous entry at line 293838.Looking through import.prices, I find:
Code:
@ TJURINAS/Herndon Dock
+ Chemicals
Hydrogen Fuel 107 112 ? 45317? 2015-02-02 09:29:23 # Maddavo_Maddavo's Market Share_1.0
Hydrogen Fuel 101 106 ? 54380? 2015-01-19 07:47:58
+ Consumer Items
Clothing 222 238 ? 17874? 2015-02-02 09:29:23 # Maddavo_Maddavo's Market Share_1.0
Clothing 203 218 ? 21448? 2015-01-19 07:47:58
Consumer Technology 7484 0 1903? - 2015-01-19 07:47:58
Consumer Technology 7570 0 1822? - 2015-02-02 09:29:23 # Maddavo_Maddavo's Market Share_1.0
Domestic Appliances 418 438 ? 10663? 2015-02-02 09:29:23 # Maddavo_Maddavo's Market Share_1.0
Domestic Appliances 390 408 ? 12795? 2015-01-19 07:47:58
+ Foods
Algae 137 0 21534? - 2015-02-02 09:29:23 # Maddavo_Maddavo's Market Share_1.0
Algae 137 0 3014? - 2015-01-19 07:47:58
Last edited:
I don't do this, but here are my thoughts.
I just upload my .prices file to maddavo, via the web page. There is no explicit "delete all these items".
But I do capture every screen, ALL the station data before I import it into local TD.
My guess is that the 'sparse' stations are a result of user error/misunderstanding of what they ought to be doing. If you look at the comments in a whole maddavo download, the sparse stations normally have an OCR source noted in the commentary.
The only way to deal with this (at maddavo and TD import) IMO is to keep the old commodity data if an update does not contain it. I think this is a valid approach. I have yet to see an item completely disappear from a station commodity market listing. I have seen stock go to 0 and demand too, but not seen a commodity disappear altogether.
Does this present a challenge to asking for trades by age, if commodities have different ages? There would be a need to define 'station age' based on a commodity update mean/standard deviation calculation I think, or consider the age of every commodity, which feels compute intensive.
Last edited:
In EDDB I've edited 409 stations total now, I did quick check against latest station.csv (downloaded just before checking) and while I didn't check all four hundred, I checked enough to give good impression; none of the EDDB data is on Maddavo's station.csv
Why is this, is everything okay?
Why is this, is everything okay?
Yes, something weird was going on there. I think I've fixed the price file generation now. I have tested importing all three files after several regenerations over the last 3 hours and it seems to be OK now.
Specifically the prices at TJURINAS/Herndon Dock are OK in the latest files.
The way the prices merging on MMS works at the moment is LINE-BY-LINE. TD does it STATION-BY-STATION. I decided to do line-by-line because there are many circumstances where I don't get a full station's worth or prices sent to me. EDDN sends prices one at a time. If you listen for a while, then you get the whole station? Not quite. In many cases people are only sending partial stations rather than religiously getting all lines on all screenshots in EliteOCR.
BUT line-by-line processing has a drawback - I cannot test for the absence of a price in order to delete it. ie: If an item is no longer bought/sold at a station then processing of incoming prices doesn't tell me if the item is deleted. That is why the updated.prices file created by TD when you do a manual update with TD will add in every item and put buy=0 and sell=0 for items that are not at the station.
Now most people are using EliteOCR to capture prices, and only capture part-stations, so what to do? How do we get rid of dead items easily?
In an attempt to get rid of these dead items, today I have added a routine that will delete any price that is greater than 14days older than the latest price at a station. Essentially, all the items at a station will be within a 14day window. This will get rid of a dead item at a station, BUT it may also have some undesirable effects. eg: If there's a station that hasn't been visited for 14 days, and someone goes and updates ONE item, then all the others will be deleted. I am hoping that is unlikely but it's something we should monitor.
As more of a manual and immediate method to get rid of a dead item, (as I can't wait 14 days), I have been doing the following:
trade.py update --sub <station> -T
changing the price of the offending item manually to 0 buy 0 sell
increment the timestamp on that line a bit
upload the updated.prices file
I think this has the effect of not changing any of the other price timestamps for the station.
- - - - - Additional Content Posted / Auto Merge - - - - -
Also, in the prices files I have added a comment at the end of each line that is received via EDDN. This shows some info from the sender of the data. This may be useful for us to spot bad data and blacklist it.
Last edited:
Yeah Snake, I have not written anything to grab stations from eddb yet. Themroc said that I had to collect rather than him send. That's OK. He offered to generate a Station.csv file and I said yes but it's not there as yet. I don't fancy writing something to grab that huge Stations.json and then convert it to Station.csv if he can just dump one for us.
If you don't mind me saying so, I think you're over thinking this. I don't think there is a need to delete any data. It's never "dead" it's just "very old". The very old data can remain until a more conscientious visitor grabs the whole station data and updates it. Or are you aware of stations that have suddenly stopped trading in commodities.
I hear you and acknowledge you are concerned about deleting data. But there are circumstances were we need to delete an item at a station. This is because the item is no longer being sold at the station (which does happen), or the item was never sold at the station in the first place. The 2nd can occur because of OCR derps or other human error and is probably the more common of the two but I can't tell the difference. I have personally seen items that were at stations around release time (6-7 weeks ago?) but are not at a station today.
The amount of data that is deleted so far is minimal. ie: in 5000 stations with 260000 prices, there have been 1438 deletions or just over half of a percent. That was 1209 initially and then about 32 every 2 hours.
The onus is on data gatherers to collect whole stations within a 2-week period. That seems fair and achievable to me.
Experience has proven otherwise. 1/ Stations stop selling things, 2/ People submit data for the wrong station, 3/ People submit prices for the wrong item when using update -A... Etc.
I think there are one of two problems:
1. It looks suspiciously like some people are using the "update" command to import ocr data into TD and generate an "updated.prices" file.
My hunch here is that they are doing this because that's the default filename for the "madupload.py" script and they don't realize it takes a filename. They also don't realize that "updated.prices" explicitly contains deletes.
In your logs/past uploads you should be able to find this by looking at users who've submitted the same station several times in fairly rapid succession, where they have, say, "ALGAE 0 0..." and "GOLD 9000 9483 ..." (delete algae, update gold) and then "ALGAE 90 120 ..." and "GOLD 0 0 ..." (update algae, delete gold) in another.
Possible fix: I can make "misc/madupload.py" require a "confirmation code" when you want to upload a .csv that wants to delete more than one item.
2. The question of whether your .prices files are "per station" or "per item": there was a transition period where it became likely that the .prices in the 3h file were only containing items that had been updated, and TD's import method treats missing items as deletions. Can you confirm that the current version emits station-at-a-time?
If not, if there's a good reason to stay with that approach, then we can look at changing the plugin to fill in the blanks, so to speak, but if that's the case, then we'll need you to generate explicit deletes when they happen, and you'll probably want to save them, which means you're going to end up with "0 0 - -" entries to your db for the timestamps, but it shouldn't be an overly large overhead.
Why would people use the update command when EliteOCR generates a .prices file that they can use with the import command?
(If that .prices file has commodities missing, that ought to be present, they are removed from the local database and .prices file when the import is run)
Did not know about madupload.py. How do I invoke it?
Question: If I use trade.py import --plug=maddavo --opt=syscsv --opt=stncsv and my local system.csv and station.csv have entries for systems and stations that maddavo does not yet know about, will they be lost from my local file/database?
(If that .prices file has commodities missing, that ought to be present, they are removed from the local database and .prices file when the import is run)
Did not know about madupload.py. How do I invoke it?
Question: If I use trade.py import --plug=maddavo --opt=syscsv --opt=stncsv and my local system.csv and station.csv have entries for systems and stations that maddavo does not yet know about, will they be lost from my local file/database?
Last edited:
Meh ... it's along the lines of this:
Code:
import json
import re
import requests
systems = {}
req = requests.get('http://eddb.io/archive/v2/systems.json')
jsonData = json.loads(req.content.decode())
with open("Systems.eddb.csv", "w") as out:
for entry in jsonData:
print("'{}',{},{},{},'EDDB','naow'".format(
entry['name'].upper(),
entry['x'],
entry['y'],
entry['z'],
), file=out)
systems[int(entry['id'])] = entry['name'].upper()
req = requests.get('http://eddb.io/archive/v2/stations_lite.json')
jsonData = json.loads(req.content.decode())
with open("Stations.eddb.csv", "w") as out:
for entry in jsonData:
lsFromStar = entry['distance_from_star']
blackMarket = entry['has_blackmarket']
mps = entry['max_landing_pad_size']
lsFromStar = int(lsFromStar) if lsFromStar else 0
if blackMarket is None:
blackMarket = '?'
elif blackMarket is 0:
blackMarket = 'N'
elif blackMarket is 1:
blackMarket = 'Y'
if mps is None:
mps = '?'
elif int(mps) <= 10:
mps = 'S':
elif int(mps) <= 20:
mps = 'M':
else:
mps = 'L'
print("'{}','{}',{},'{}','{}'".format(
systems[int(entry['station_id'])],
entry['name'],
), file=out)Actually, I was so convinced of this I went and knocked up a script for it ... grab the latest td and do
Code:
$ python -m misc.eddb(this runs the misc/eddb.py script as a module, it'll generate tmp/Systems.eddb.csv and tmp/Stations.eddb.csv for your enjoyment)