That is beside the point of the thread. Create your own thread to argue your point. We are discussing can an OCR reliably read the commodities board.
Discussion Has any one experimented with OCR tools for scraping the commodities market prices?
- Thread starter wolverine2710
- Start date
That is beside the point of the thread. Create your own thread to argue your point. We are discussing can an OCR reliably read the commodities board.
Haven't managed to get 3.00 so I haven't seen if they have done anything to the commodities page.
Have they?
Have they?
wolverine2710
Tutorial & Guide Writer
It seems that all menus including the commodities market now have a tainted glass background. Colors do shine through a bit. A bit like the glass in the showers - so you can't see it all. It MIGHT make things a little easier.
Note: Atm hunting for a better then E-drive FSD for my newly bought Hauler so I CAN go out there again.
Last edited:
Just as an Update: http://imgur.com/a/L8hsy
Thats the accuracy I'm currently achieving. Imgur has screwed up the formatting for my console outputs <.<
But here it is properly formatted:
I have yet to test different Gamma Settings, and resolutions, before releasing anything. And I'm not even sure i will release this as future updates are likely to change the UI more than once.
I did this mostly to keep track of where i've been and what they had to offer.
Thats the accuracy I'm currently achieving. Imgur has screwed up the formatting for my console outputs <.<
But here it is properly formatted:
Code:
goods sell buy cargo demand supply
-------------------- ------ ----- ------- -------- --------
Hydrogen Fuel 101 102 - - 13772
Mineral Oil 139 154 - - 1001
Pesticides 356 - - 8191 -
Clothing 392 - - 2698 -
Consumer Technology 7315 - - 413 -
Dom. Appliances 733 - - 2110 -
Algae 60 77 - - 1942
Animal Meat 1305 1337 - - 7475
Coffee 1328 1361 - - 13
Fish 606 634 - - 395
Fruit and Vegetables 276 294 - - 531
Grain 156 172 - - 45449
Tea 1509 1546 - - 59
Beer 117 131 - - 1297
Liquor 735 - - 32 -
goods sell buy cargo demand supply
----------------------- ------ ----- ------- -------- --------
Water Purifiers 375 - - 1141 -
Agri-Medicines 1307 - - 5597 -
Basic Medicines 471 - - 938 -
Performance Enhancers 7315 - - 390 -
Progenitor Cells 7315 - - 155 -
Animal Monitors 453 - - 53842 -
Aquaponic Systems 346 - - 12578 -
Land Enrichment Systems 5505 - - 10275 -
Leather 118 123 - - 11635
Natural Fabrics 376 393 - - 399
Biowaste 99 - - 23319 -
Non-Lethal Wpns 2091 - - 75 -
Personal Weapons 4680 - - 243 -
Reactive Armour 2322 - - 35 -
goods sell buy cargo demand supply
-------------------- ------ ----- ------- -------- --------
Explosives 453 - - 4660 -
Hydrogen Fuel 101 102 - - 7862
Pesticides 176 190 - - 3220
Clothing 471 - - 2475 -
Consumer Technology 7005 7081 - - 1403
Dom. Appliances 716 - - 1016 -
Animal Meat 1483 - - 175 -
Coffee 1840 - - 408 -
Food Cartridges 250 - - 3625 -
Fruit and Vegetables 432 - - 1332 -
Grain 323 - - 3780 -
Synthetic Meat 168 180 - - 1371
Tea 1747 - - 316 -
Superconductors 7211 - - 1928 -
goods sell buy cargo demand supply
----------------------- ------ ----- ------- -------- --------
Uraninite 883 900 - - 3456
Advanced Catalysers 2827 2859 - - 1
Animal Monitors 260 277 - - 2249
Aquaponic Systems 234 550 - - 245
Auto-Fabricators 3821 3864 - - 2493
Bioreducing Lichen 1081 - - 3286 -
Computer Components 733 - - 1786 -
H.E. Suits 421 - - - -
Land Enrichment Systems 5007 5062 - - 1063
Resonating Separators 5985 6050 - - 1
Robotics 1837 1859 - - 4584
Biowaste 21 27 - - 68
Scrap 49 55 - - 407
Non-Lethal Wpns 1867 1889 - - 644
Personal Weapons 4857 - - - -
Reactive Armour 2128 2153 - - 77
goods sell buy cargo demand supply
------------------- ------ ----- ------- -------- --------
Platinum 18953 - - 147 -
Silver 5489 - - 6983 -
Tantalum 4564 - - 8274 -
Uranium 3152 - - 11302 -
Bertrandite 2580 2610 - - 2
Coltan 1424 1459 - - 67
Gallite 2331 - - 14993 -
Lepidolite 536 559 - - 35
Rutile 484 - - 34209 -
Uraninite 863 908 - - 3456
Advanced Catalysers 2827 2859 - - 1
Animal Monitors 260 277 - - 2249
Aquaponic Systems 234 250 - - 246
Auto-Fabricators 3821 3864 - - 2497
goods sell buy cargo demand supply
-------------------- ------ ----- ------- -------- --------
Hydrogen Fuel 101 102 - - 124462
Pesticides 155 168 - - 124121
Clothing 462 - - 37515 -
Consumer Technology 6153 6234 - - 115535
Dom. Appliances 657 - - 12014 -
Animal Meat 1633 - - 8993 -
Coffee 1531 - - 2967 -
Food Cartridges 243 - - 11974 -
Fruit and Vegetables 454 - - 12727 -
Grain 332 - - 63633 -
Synthetic Meat 198 213 - - 11760
Tea 1821 - - 7535 -
Superconductors 7520 - - 61453 -
goods sell buy cargo demand supply
----------------------- ------ ----- ------- -------- --------
Gallite 2321 - - 327183 -
Rutile 488 - - 737499 -
Advanced Catalysers 2896 2936 - - 691
Animal Monitors 259 277 - - 50882
Aquaponic Systems 232 249 - - 5668
Auto-Fabricators 3503 3550 - - 129623
Bioreducing Lichen 942 969 - - 1794
Computer Components 716 - - 33023 -
H.E. Suits 232 249 - - 28203
Land Enrichment Systems 4944 5009 - - 27500
Resonating Separators 6144 6224 - - 354
Robotics 1760 1785 - - 151888
Biowaste 23 30 - - 989
Scrap 29 33 - - 29579
Non-Lethal Wpns 1503 1524 - - 82046
Reactive Armour 1729 1753 - - 37106I have yet to test different Gamma Settings, and resolutions, before releasing anything. And I'm not even sure i will release this as future updates are likely to change the UI more than once.
I did this mostly to keep track of where i've been and what they had to offer.
wolverine2710
Tutorial & Guide Writer
Don't know what the real numbers are but this looks VERY VERY promising. Please DO consider releasing it. I saw your request for an API in the BPC thread. An alternative to the BPC is trade dangerous by kfsone. Its a command line atm but some cmdrs are building a frontend for it. Its based upon manual input - as in update the trade dangerous.prices table. After that TD picks it up and its stored in the database (speed). If you have this output I'm sure someone is willing to create an importer for TD. Atm there is also a merge website for it. VERY VERY nice work.
How did you do it and what tools did you use?
Concerning pre-processing, imagemagick has a lot of tools (cmd line) to process an image.
This is better than what I have achieved. I was looking at trying a bit of preprocessing with OpenCV.
Can you give details on how you got such good accuracy?
Can you give details on how you got such good accuracy?
I used OpenCV for everything. From pre processing, over finding contours of letters and numbers, up to matching everything with custom trained models fed into K-Nearest.
I tried with tesseract and other OCR Engines, but most of them needed specific pre processing anyway, and since I'm not only doing this to enhance my gameplay experience, but also to learn something, doing it "from sratch" with OpenCV was a no brainer.
For letters, the recognition is quite fuzzy. Medicine may turn out as Meoicine or Mediaine and so on. But since we know all the goods i just use difflib to match the ocr with the expected results. For numbers I implemented a small set of sanity checks (like sell<buy). I'm trying to implement more checks based on average differences for prices from average etc..
Overall the script works quite nicely. I have just finished the code to get notified of a new screenshot, check if its a market shot, and if it is: process and delete it.
The next thing will be to check how everything behaves with different gamma and resolution settings. But atm I'm quite busy, so that will have to wait.
If progress continues I may be releasing it(or wait till gamma and see what FD comes up with), because even with an error rate of ~2-5% (and with implemented filters these errors will be in the least significant numbers) it's a huge improvement over pen&paper
I tried with tesseract and other OCR Engines, but most of them needed specific pre processing anyway, and since I'm not only doing this to enhance my gameplay experience, but also to learn something, doing it "from sratch" with OpenCV was a no brainer.
For letters, the recognition is quite fuzzy. Medicine may turn out as Meoicine or Mediaine and so on. But since we know all the goods i just use difflib to match the ocr with the expected results. For numbers I implemented a small set of sanity checks (like sell<buy). I'm trying to implement more checks based on average differences for prices from average etc..
Overall the script works quite nicely. I have just finished the code to get notified of a new screenshot, check if its a market shot, and if it is: process and delete it.
The next thing will be to check how everything behaves with different gamma and resolution settings. But atm I'm quite busy, so that will have to wait.
If progress continues I may be releasing it(or wait till gamma and see what FD comes up with), because even with an error rate of ~2-5% (and with implemented filters these errors will be in the least significant numbers) it's a huge improvement over pen&paper
I use OCR all the time
I use two clever devices that sit between my ears and slightly above and to the left and right of my nose
I then use a clever interface between my seat and my computer that transfers all the data into an Access Database
Once there it transfers into excel and develops a trading route
I then use the interface between my stick and computer and the optical devices noted earlier to automatically purchase the best items noted by excel
Works really well - is quite a cheap and easy solution and have used it ever since trading was introduced
Occasionaly the seat computer interface plays up and i go chasing dead ends for a while but I am ironing out those bugs
I use two clever devices that sit between my ears and slightly above and to the left and right of my nose
I then use a clever interface between my seat and my computer that transfers all the data into an Access Database
Once there it transfers into excel and develops a trading route
I then use the interface between my stick and computer and the optical devices noted earlier to automatically purchase the best items noted by excel
Works really well - is quite a cheap and easy solution and have used it ever since trading was introduced
Occasionaly the seat computer interface plays up and i go chasing dead ends for a while but I am ironing out those bugs
It's all spreadsheets.
The font is easy to read, it should be trivial to OCR with even ancient/free OCR tools.
But this isn't the issue.
The issue is: someone will memory scan this data regardless of FD's wishes. If the game even becomes 10% of the success they hope for, people will NOT tolerate having to use paper and pencil in 2015. It was fine in 1984 because there was no choice. 30 years later, there are choices.
This is the same issue every MMO has. Scanning memory to get what the players want, rather than what the developers want, has been happening since 1997. It's not going to stop, and nothing FD can do will permanently stop it. Asking people to stop will not stop them. Rotating encryption will not stop them.
Unless you require a borg implant to read market data, it will be scraped, pulled, scanned, read, and turned into a database so people can make money as fast as they want.
While I would never presume to know as much as the mighty FD regarding game design, when every aspect of your game are blatant time sinks, I have observed, over the past 30 years, it creates undesired emergent behavior in your paying customers.
For example, if, hypothetically, the most you can make on any trade is 100 credits per ton, and you need 400 million credits to obtain your desired ship & loadout, it becomes trivial to determine how much cargo & time you need to buy & sell to obtain that ship.
Given the rather alarming numbers produced by such an exercise, it shouldn't be surprising that some players prefer the most efficient use of their time to reach their goals. Given a target audience of over 1000 players, the clever big-brained mammals sitting in the chairs are going to put those big brains to use, either alone or together. Sorry, that's just the way it is.
I believe FD can encourage or discourage player goals *slightly*, but once your audience learns that ALL computer games are spreadsheets with fancy GUI's, well, sometimes some of them just want to read the spreadsheet.
And of course, the last thing; if/when offline mode is added, many of these issues will be moot, because people will simply get what they want offline, and skip online altogether, even if that means having to run the entire game in a VM to get what they want.
The font is easy to read, it should be trivial to OCR with even ancient/free OCR tools.
But this isn't the issue.
The issue is: someone will memory scan this data regardless of FD's wishes. If the game even becomes 10% of the success they hope for, people will NOT tolerate having to use paper and pencil in 2015. It was fine in 1984 because there was no choice. 30 years later, there are choices.
This is the same issue every MMO has. Scanning memory to get what the players want, rather than what the developers want, has been happening since 1997. It's not going to stop, and nothing FD can do will permanently stop it. Asking people to stop will not stop them. Rotating encryption will not stop them.
Unless you require a borg implant to read market data, it will be scraped, pulled, scanned, read, and turned into a database so people can make money as fast as they want.
While I would never presume to know as much as the mighty FD regarding game design, when every aspect of your game are blatant time sinks, I have observed, over the past 30 years, it creates undesired emergent behavior in your paying customers.
For example, if, hypothetically, the most you can make on any trade is 100 credits per ton, and you need 400 million credits to obtain your desired ship & loadout, it becomes trivial to determine how much cargo & time you need to buy & sell to obtain that ship.
Given the rather alarming numbers produced by such an exercise, it shouldn't be surprising that some players prefer the most efficient use of their time to reach their goals. Given a target audience of over 1000 players, the clever big-brained mammals sitting in the chairs are going to put those big brains to use, either alone or together. Sorry, that's just the way it is.
I believe FD can encourage or discourage player goals *slightly*, but once your audience learns that ALL computer games are spreadsheets with fancy GUI's, well, sometimes some of them just want to read the spreadsheet.
And of course, the last thing; if/when offline mode is added, many of these issues will be moot, because people will simply get what they want offline, and skip online altogether, even if that means having to run the entire game in a VM to get what they want.
Yeah, Player want many things, that would ruin a Game when they get it. The MMORPG Genre is the best proof of that, the Genre is ruined because the Devs gives the Players what they want and when not than they cheat... and the Result? This Players cries after some less Weeks how boring the Game is...
Too much Players didn't want to play, they want only be entertained and the Computer should play for them.
And thats the Reason why I don't have any anticipation for the free Online Mode. When I play online, than only with Friends from them I know they want to play a Game as it should be played and not as they want it should be played even when it is against the Game.
But beside that, the Players are right... why should a SpaceSim don't have Ship Computers that could save Data? The Prices did change from time to time, so the older your Data on your Ship are, that more incorrect are they. So I see no big Problem to save this Data at your own Ship. To buy and sell Goods you still need to dock.
Too much Players didn't want to play, they want only be entertained and the Computer should play for them.
And thats the Reason why I don't have any anticipation for the free Online Mode. When I play online, than only with Friends from them I know they want to play a Game as it should be played and not as they want it should be played even when it is against the Game.
But beside that, the Players are right... why should a SpaceSim don't have Ship Computers that could save Data? The Prices did change from time to time, so the older your Data on your Ship are, that more incorrect are they. So I see no big Problem to save this Data at your own Ship. To buy and sell Goods you still need to dock.
Last edited:
What is the issue with people not understanding what the thread is about? Why spend several minutes wasting yours and others time with a pointless diatribe that has nothing to do with the issue at hand? Good to know you use pencil/paper, excel spreadsheet, whatever.
Now I have spent several minutes responding.
- - - - - Additional Content Posted / Auto Merge - - - - -
It would be gratefully received if you would release it as is.
Now I have spent several minutes responding.
- - - - - Additional Content Posted / Auto Merge - - - - -
It would be gratefully received if you would release it as is.
Does anyone know if all station names are unique?
wolverine2710
Tutorial & Guide Writer
They are NOT. Did sent Michael Brookes a couple of question for the crowd source project.
Respone of Michael brookes
Michael Brookes said:
wolverine2710
Tutorial & Guide Writer
A commander, in another thread, stated he has a (near) working OCR solution. He was reluctant to release it because he was not sure it would violate the new FD data access policy. To clear this up I send a PM to Michael Brookes. I asked for permission to share his response and this was granted.
His response:
I HAVE also updated the OP.
Hello Mr Brookes,
I have created the "Has any one experimented with OCR tools for scraping the commodities" thread. Basically a commander takes a screenshot, which is automatically read by an OCR tool which then creates a text file with commodities market prices. This can be used for for example trading tool purposes. For example feeding the text to Trade Dangerous program or Slopeys the BPC. Both are based upon manually inputting of data. Lets make this totally absolutely 100% clear. The end goal will be an EDDN - Elite Dangerous Data Network. Basically Andreas EMDN but filled with data obtained from OCR-ing. NOT by scraping the ED program/protocols. Also when OCR-ing is successful it will most likely be extended to retrieve more information.
When FD's data access policy changed you wrote: "We're no longer allowing tools which scrape the data directly from the game or through its communications. We will be considering an external API for future development, but for now these tools are not permitted."
A commander stated he has a (near) working OCR solution. He is reluctant to release it because he's not sure it would violate the new FD data access policy. There are two possibilities.
With kind regards,
Jan Bessels aka Wolverine2710 aka cmdr Wolverine
I have created the "Has any one experimented with OCR tools for scraping the commodities" thread. Basically a commander takes a screenshot, which is automatically read by an OCR tool which then creates a text file with commodities market prices. This can be used for for example trading tool purposes. For example feeding the text to Trade Dangerous program or Slopeys the BPC. Both are based upon manually inputting of data. Lets make this totally absolutely 100% clear. The end goal will be an EDDN - Elite Dangerous Data Network. Basically Andreas EMDN but filled with data obtained from OCR-ing. NOT by scraping the ED program/protocols. Also when OCR-ing is successful it will most likely be extended to retrieve more information.
When FD's data access policy changed you wrote: "We're no longer allowing tools which scrape the data directly from the game or through its communications. We will be considering an external API for future development, but for now these tools are not permitted."
A commander stated he has a (near) working OCR solution. He is reluctant to release it because he's not sure it would violate the new FD data access policy. There are two possibilities.
- OCR does violate FD's new data access policy.
- OCR does NOT violate FD's new data access policy.
With kind regards,
Jan Bessels aka Wolverine2710 aka cmdr Wolverine
His response:
Michael Brookes said:
I HAVE also updated the OP.
Last edited:
I am so waiting for this. It will be a pain though, if not impossible, to utilize with an OR....
wolverine2710
Tutorial & Guide Writer
Correct.
Slopey's BPC tool pulls the current system from the log file. Using that information in conjunction with the station name would seem to be a viable option.