Ummmm. It's more realistic and also more (futuristic lol) To have a tool to get the prices and tell you where to sell. And to tell you what is available where and how much it sells and buys for. The ingame market tool is horrible. It really needs re-working. I mean seriously even back in the 20's and 30's you could just grab a newspaper and check stocks in the world.
Discussion Has any one experimented with OCR tools for scraping the commodities market prices?
- Thread starter wolverine2710
- Start date
Stupid question: If there are issues with doing the OCR and recognising characters, would a 4k screenshot be a better resolution to use? Would it provide more accurate results than 1920x1080?
wolverine2710
Tutorial & Guide Writer
That is something I was thinking about. I have to run at 1600x900 now to get a half decent frame rate. I thought about using the screenshot folder but I don't want to pollute the users HDD with old images hence hooking a key other than F10.
wolverine2710
Tutorial & Guide Writer
Thats the domain of trading tools NOT OCR. Before FD changed their data access policy Andreas had marketdump (included by Slopeys the BPC) which uploaded the data to EMDN. Effectively every 1-2 seconds the combined commanders using MD and ED where uploading data to EMDN. Other tools relied on this.
What we are trying to do here is to get those market prices back in a legal way so all third party tools can use them.
wolverine2710
Tutorial & Guide Writer
Its great to see the OCR isn't dead and work is ongoing!!!
The F11 route is a nice on as like you said it wouldn't pollute the screenshots directory AND it would make life easier because we have a directory which ONLY holds commodities market data. Easier to automize things as this dir doesn't have other screenshots which don't have valid data and which would bork the OCR.
Is there any interest in iphone app that scans market using camera (augmented reality), data immediately uploaded to the server and OCR processed using trained neural network, same app can give you a hint on where you can sell stuff on stations near you?
I also believe it will be legit from Frontier point of view...
I also believe it will be legit from Frontier point of view...
wolverine2710
Tutorial & Guide Writer
I have to admit, I'm not quite sure how serious you are. It sounds to good to be true. Especially since this post was your first ever here in the FD forum. I'm replying under the assumption you ARE serious.
I think there is interest in ANY tool which can reliably parse the commodities market screen and output market prices. If you can pull that off then please share it with the rest of us.
If I've interpreted your post correctly the iphone is just a means to get a screenshot and the rest is done one a server. Hence if the server part works we can send in F10 or F11 (suggested) created screenshots to your server with a simple program to upload the screenshots.
Edit: The OCr route IS legit. It does NOT violate the new data access policy of FD.
Last edited:
Ozric
Volunteer Moderator
I can see why you wish for that, and don't want to derail your OP entirely, but trying to keep on-topic I think my middle paragraph is pertinent, asking the question "what problem is being solved?". If it's the scant trading in Beta 2, that situation may be resolved in three weeks' time with Beta 3, or later once the game properly gets going. I'd say first define the problem, then find a solution. Are people that sure that trading will be so challenging in the live game that going to the effort of OCR screenscraping will really be worth it?
Looking back to your OP subject line, I presume this was answered the first time another poster replied "yes".
Good luck with which ever way you go. I merely observe that a problem in B2 may no longer be an issue in B3, B4, Bn, or the live game.
You make a good and valid point, which I have made many times before. The trouble is that some people are not prepared to wait and see how to play the game the way it is intended to be played, but want to do it their own way.
Or you could wait for the App that FD are going to release.
I wasn't going to actually save the image to disk (though it currently does just to test) and simply upload the extracted result. Future expansion could be downloading of updated training but to begin with it will just do the uploading.
At somepoint in the future, with an unannounced feature list that may or may not do what we want it to!
wolverine2710
Tutorial & Guide Writer
As you have quoted a commander which mentioned my request not to derail my thread - see OP - I can't respond to your post.
However I'm ALSO very interested in the initially iPhone only app. Btw afaik at this point its not decided by FD if the app will be free or paid.
The from a technical POV interesting part for me is that the app will NOT run on a (windows) computer and can't access ED. Whatever information shown in the app NEEDS to be dowloaded from the server. Or they have part of the PG engine in the app for stuff like text for a star system etc. If we are LUCKY they use a clearly defined web api to get info from the FD servers. A web-api they COULD share with the community. That would be SO nice and useful. If we are less lucky they just sent a network request to the FD server(s) and parse the received info...
Last edited:
Yep, missed that, sorry.
wolverine2710
Tutorial & Guide Writer
No image on PC, even better.
Could you elaborate on what you mean with "downloading up updated training". Would that be a file which can be fed to tesseract so its OCR capabillities for the commodities market is enhanced? Soooo looking forward to your app. Wish you all the luck with it!!!!!
Last edited:
wolverine2710
Tutorial & Guide Writer
I just want this stuff to work - can't really contribute anything much to the collection of data via scraping and/or OCR personally (not got the time to learn such things), but after it has been collected, I would certainly like to be able to consume it - I am a database developer and I know I can do all sorts of wonderful analysis and slicing & dicing of the data once I have a dataset to work with.
wolverine2710
Tutorial & Guide Writer
You might want to send a PM to Slopey. After FD changed their policy he announced that he was planning to give READ only access to his database - fed by Andreas EMDN so it has lots of data. Also IxForres with his web-api has collected lots of EMDN data. I believe he shutdown his project though - after the change. Perhaps Slopey and/or IxForres can provide you with oodles of data. Data you can use to slice an dice and be prepared for things to come.
Yes wonderful things can be done by slicing/dicing this data. One could feed all that data into an open source ELK stack (Elasticsearch, Logstash and Kibana) and have Kibana display the data for you - for free. See also Andreas thread about the use of Kibana. VERY powerful open source stuff.
Last edited:
The only issue I have with Slopey's data, and data from EDMN is that is is well out of date (is EDMN still running?), and by that I mean things like Fish is no longer available at Chango Dock - and FD have changed the balancing of the trading anyway. I'm pretty sure EDMN and Slopey's data has not been updated for Beta 2 - and I would not expect Slopey to start providing up to date data to all and sundry until he's ready to release his own tools.
Meantime, I am already working with out of date and incomplete data in order to design my database, which is a work in progress.
wolverine2710
Tutorial & Guide Writer
Indeed. It IS obsolete data. Slopey relied on the EMDN - when NOT in local mode. Andreas shutdown the firehose of EMDN. I just thought I mentioned it in case you needed data so you could start your project. Up to date data HAS to come from OCR at this point. That is until FD releases an official web-api or gives us market data in some other form. Its amazing to see what the community has done so far and the information deduced by what is seen ingame. Basically mostly done without assistance of FD.
It does not have to be OCR, it just takes a bit more time to fill in the prices. OCR is a distraction at this point. And not much timesaving once you got all the commodities in your database/spreadsheet.
Data sharing is something else. If someone want to get it going start with a way to enter and update data manually. Require a login. Require updating markets by giving share-credits, more credits for markets that have oldest data. Share credits are required for using the trade/search tools for the collected data.
Data sharing is something else. If someone want to get it going start with a way to enter and update data manually. Require a login. Require updating markets by giving share-credits, more credits for markets that have oldest data. Share credits are required for using the trade/search tools for the collected data.