Discussion Has any one experimented with OCR tools for scraping the commodities market prices?

[Stormprooter]

It does not have to be OCR, it just takes a bit more time to fill in the prices. OCR is a distraction at this point. And not much timesaving once you got all the commodities in your database/spreadsheet.

Data sharing is something else. If someone want to get it going start with a way to enter and update data manually. Require a login. Require updating markets by giving share-credits, more credits for markets that have oldest data. Share credits are required for using the trade/search tools for the collected data.

Thrudd has already got this up and running - http://www.elitetradingtool.co.uk/

You can register and submit data manually. AFAIK getting data out is only available through the same website interface, no data feeds via webservices or anything like that for other tools to use.

 

[intra_au]

I have to admit, I'm not quite sure how serious you are. It sounds to good to be true.

It's not THAT difficult, there are apps that translate stuff between different languages when you "see through" the camera.. You don't have to take a picture (e.g. Word Lens)
It's probably not easy to do OCR straight on the iphone but it will be reasonably simple task for the server (same font makes it damn easy), the grid on the screen is quite distinctive, I can pass the content of table (every cell) to the server and deal with it.

I'm just thinking how inconvenient it gonna be for a commander to hold an iphone with app next to the screen while scrolling through the commodity list. Second concern is Frontier API (will make app useless when it's out).
The app, if it's going to be released will not be free, however I think a price within $5 should be reasonable.

 

[Codec]

I put some of my work in bitbucket, if you want to try it out. It uses gocr rather than tesseract to do the OCR, and snips bits out of a currently fixed size) sceenshot.

It might be easier to merge it into other projects, which you are free to do.

https://bitbucket.org/CodecED/edocr/overview

 

[wolverine2710]

I put some of my work in bitbucket, if you want to try it out. It uses gocr rather than tesseract to do the OCR, and snips bits out of a currently fixed size) sceenshot.

It might be easier to merge it into other projects, which you are free to do.

https://bitbucket.org/CodecED/edocr/overview

Thanks for sharing. I will try to have a look at it this evening or tomorrow. Spending to much time on threads and not playing ED at all atm ;-(

 

[wolverine2710]

It's not THAT difficult, there are apps that translate stuff between different languages when you "see through" the camera.. You don't have to take a picture (e.g. Word Lens)
It's probably not easy to do OCR straight on the iphone but it will be reasonably simple task for the server (same font makes it damn easy), the grid on the screen is quite distinctive, I can pass the content of table (every cell) to the server and deal with it.

I'm just thinking how inconvenient it gonna be for a commander to hold an iphone with app next to the screen while scrolling through the commodity list. Second concern is Frontier API (will make app useless when it's out).
The app, if it's going to be released will not be free, however I think a price within $5 should be reasonable.

You seem to have missed a post by graham.reeds. He's also looking into an OCR solution. He has created a tray app for windows which uses F11 to grab the data from the screen or make a screenshot. Then OCR the image and sends the results to a server. I can imagine he may want to share the grab/upload code with us/you. Hence you only have to concentrate on the server side OCR stuff. Is that something you would be interested in?

You state that it is not THAT difficult. That might be entirely true. Fact is I started this thread on the 25th of September 2014 and atm we don't have a working OCR solution - though a few proof concepts exists and work is still going on by a few commanders. For which I want to thank them.

If you are interested in this OCR stuff and think you can make a OCR app please be so kind to do so. We can't have enough knowledge and/or solutions. If server resources are a problem I can imagine there are commanders who can help you with that particular problem. As in when you have everything working on your test server.

Concerning the iPhone app. We don't know what it is going to show. And about being free/paid. I asked Michael Brookes a few weeks ago and at that point FD had not made the decision if it would be free/paid. Things can have changed since then, but I'm unaware of that. If you have have a source for your information, perhaps you can share it with us.

Of course a web-api would make the OCR route obsolete - if it has the functionality we need. But it does NOT exist and iirc FD are only CONSIDERING it - and if so only after the initial ED release. Even if its made obsolete a lot of commanders will have probably enjoyed doing it because they learned something new and/or have helped the community or at least a small part of with a certain need.

 

[mflvs]

I put some of my work in bitbucket, if you want to try it out. It uses gocr rather than tesseract to do the OCR, and snips bits out of a currently fixed size) sceenshot.

It might be easier to merge it into other projects, which you are free to do.

https://bitbucket.org/CodecED/edocr/overview

Hi i have some experince with ocr. and was going to test your project from bitbucket. I was wondering if you could state what resolution / size you are using as you state it is currently fixed to yours

 

[wolverine2710]

Hi i have some experince with ocr. and was going to test your project from bitbucket. I was wondering if you could state what resolution / size you are using as you state it is currently fixed to yours

Forgot to check my own thread....
Codec's resolution is 1920x1080.
Glad to see you onboard commander!!

 

[MadDogMurdock]

For those who wish to participate in a crowd-sourced manual entry tool there is:

http://elitetradingtool.co.uk/

As a manual tool, the output is only as good as the effort made to input.

With ED now being a social multiplayer game, I feel that this is acceptable sharing of info.

Thrudd uses a ranking system for access to more functionality, so you only get to veify and edit prices after making enough queries and then that leads on to station and system updates.

Most of the B2 systems are there so the distances and routes should be reasonably accurate but I'm not so sure about the prices

 

[wolverine2710]

Most of the B2 systems are there so the distances and routes should be reasonably accurate but I'm not so sure about the prices

I've a feeling I know exactly where the B2 systems, coordinates and distances are (largely) coming from

Its a similar process as what we are trying to do here - though atm far more successful. Respecting FD and their changed data access policy. Putting the great minds and knowledge of as many commanders together to see if we can find ways to legally obtain data for which there is a an itching need. If that other process would not have been there, Thrudd would not have had the 573 systems and coordinates which he could use.

I like what he is doing. I'm hoping there will be collaboration between him and other trading tools like trade dangerous. Combining and sharing of legally obtained price info is key to having a fighting change to get some form of up to date prices.

It just happens that here in this thread we try to get prices by OCR-ing.

 

[MadDogMurdock]

I've a feeling I know exactly where the B2 systems, coordinates and distances are (largely) coming from

Its a similar process as what we are trying to do here - though atm far more successful. Respecting FD and their changed data access policy. Putting the great minds and knowledge of as many commanders together to see if we can find ways to legally obtain data for which there is a an itching need. If that other process would not have been there, Thrudd would not have had the 573 systems and coordinates which he could use.

I like what he is doing. I'm hoping there will be collaboration between him and other trading tools like trade dangerous. Combining and sharing of legally obtained price info is key to having a fighting change to get some form of up to date prices.

It just happens that here in this thread we try to get prices by OCR-ing.

Credit where it's due. I spent some time yesterday inputing missing systems from your crowdsourced coordinates effort.

I did use Slopey's for a bit during B1 but ultimately I felt that getting the info for free with no input effort on my part was unrewarding which is why I like the more manual approach even with its less than optimal pricing outputs.

I completely understand the effort to scrape the data in some way being more practical but personally that's not for me.

 

[Colonel Kenney]

None of this would be necessary if they actually had a good trading interface that doesn't require us to use external websites to search for last known prices.

 

[malicious1]

they had a brilliant one inhouse...and they opted for the dumbed down console version

 

[Hiply Rustic]

they had a brilliant one inhouse...and they opted for the dumbed down console version

I think that about summed that up.

 

[Henri]

scraping data will also reveal any undiscovered/secret/pirate bases and other surprises, it would be a mega spoiler.

Also if someone discovers something he can decide to keep it to himself and profit or share the information with others, it is a bit of role playing within the community. if everything is allways known to everyone because of scraping it would kill the fun for everyone.

what will be scraped next? the locations of all undiscovered planets, all undiscovered bases, all the places where there is gold? scrape where you can buy equipment? scrape where you can buy ships?

in the end what do you gain except ruining any chance of discovering anything.

maybe if you do want to cheat, just do it and not post about it anywhere, just do it and hopefully at some point they will supend accounts who do that.

 

[Holven]

I wouldn't count in-game FD-made tools out yet. They keep giving us temporary solutions because we are testing code and functionality. It will be interesting to see how things look at launch. Up until that day, anything and everything can change.

As for gathering data... A friend of mine has an excellent solution... he takes a picture of his screen with his Ipad. Archive of everywhere he has ever been...

And come a huge amount of tradeable worlds on launch... any data retrieved from the system in any way and gathered in databases, will be impossible to keep updated for long. Too many players don't care to use such tools and too many worlds and trade routes.

I understand your wishes and efforts, but I believe they will be ultimately futile, except for making personal logs and see to that core starter worlds will have no market left to trade in, in very short time. As people head off in every direction from the crashed markets of core starting positions, division, distance and pure numbers will likely mess it all up.

 

[wolverine2710]

I know I replied myself to a post which was not primarily suited for this thread. My bad. When I woke up this morning I saw things got derailed. May I ask to concentrate on the matters at hand. As in lets stay ON TOPIC and discuss the OCR-ing of the market prices. . That would be appreciated.

 

[PixelPolish]

they had a brilliant one inhouse...and they opted for the dumbed down console version

Better trade interface is needed - there's no doubt about it. Then there's our onboard computer - doesn't it have some sort of internal storage? WHy can't it save the market data from the stations we have docked with? Then we could check that data at any time, compare prices in our current location, share that data with other people (via in-game means) if we want to, buy that data from others (PCs and NPCs), etc. Current system is not only primitive but also doesn't make any sense. I can't blame people for trying to fix it with 3rd party tools.

 

[graham.reeds]

snip

This isn't about what they had or haven't done. This is about using OCR to parse the commodities board.

 

[Colonel Kenney]

I mean how in the future would they not have something in their computer that KEEPS the data you just looked at? Why wouldn't you be able to reference the data from the station you just came from? Why must we do it manually ourselves?

I think just like the exploration function you could sell and buy updated market info for very small amounts of money like 50cr or something. Then it makes it like a price browser.

 

[Ozric]

I've a feeling I know exactly where the B2 systems, coordinates and distances are (largely) coming from

Its a similar process as what we are trying to do here - though atm far more successful. Respecting FD and their changed data access policy. Putting the great minds and knowledge of as many commanders together to see if we can find ways to legally obtain data for which there is a an itching need. If that other process would not have been there, Thrudd would not have had the 573 systems and coordinates which he could use.

I like what he is doing. I'm hoping there will be collaboration between him and other trading tools like trade dangerous. Combining and sharing of legally obtained price info is key to having a fighting change to get some form of up to date prices.

It just happens that here in this thread we try to get prices by OCR-ing.

I feel I should mention that Thrudd's site has been up and running for a lot longer than that. I've been helping him out on and off since I started in Premium Beta 1, there was no need to try and delve into the programming because we were waiting to see what things were going to be implemented by FD and until that time he started on the collaboration idea. It's never used any of the scraping tools.