Page 1 of 14 1234511 ... LastLast
Results 1 to 15 of 198

Thread: Has any one experimented with OCR tools for scraping the commodities market prices?

  1. #1

    Has any one experimented with OCR tools for scraping the commodities market prices?

    Probably most commanders have read this news dated 20140925) by Michael Brookes: "We're no longer allowing tools which scrape the data directly from the game or through its communications. We will be considering an external API for future development, but for now these tools are not permitted."

    This killed EMDN which fed trading tools like Slopey's BPC and a few other trading tool websites. The BPC will be changed so that prices can be entered manually. But of course this is suboptimal.

    Just wondering. Has any one been experimenting with (free) OCR tools to convert a commodities market screenshot into text. If that is feasible it would mean some kind of EMDN would be feasible in the near future. The above would be in compliance with FD if I'm not mistaken.

    Edit: To clear this up I send a PM on the 11th of November 2014 to Michael Brookes. I asked for permission to share his response and was given permission.
    Hello Mr Brookes,

    I have created the "Has any one experimented with OCR tools for scraping the commodities" thread. Basically a commander takes a screenshot, which is automatically read by an OCR tool which then creates a text file with commodities market prices. This can be used for for example trading tool purposes. For example feeding the text to Trade Dangerous program or Slopeys the BPC. Both are based upon manually inputting of data. Lets make this totally absolutely 100% clear. The end goal will be an EDDN - Elite Dangerous Data Network. Basically Andreas EMDN but filled with data obtained from OCR-ing. NOT by scraping the ED program/protocols. Also when OCR-ing is successful it will most likely be extended to retrieve more information.

    When FD's data access policy changed you wrote: "We're no longer allowing tools which scrape the data directly from the game or through its communications. We will be considering an external API for future development, but for now these tools are not permitted."

    A commander stated he has a (near) working OCR solution. He is reluctant to release it because he's not sure it would violate the new FD data access policy. There are two possibilities.

    1. OCR does violate FD's new data access policy.
    2. OCR does NOT violate FD's new data access policy.

    I hope you can give me a response back which states which of the two is true. That way I can communicate it back to the author and the tread in which the question was raised.

    With kind regards,
    Wolverine2710 aka cmdr Wolverine
    His response:
    Originally Posted by Michael Brookes
    As long as the method doesn't alter or access the exe or the game resources then it is fine. So OCR isn't an issue for us, although trying to read the frame buffer directly could be.
    Please do NOT turn this into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". .
    I hope and trust you respect my wishes.

  2. #2
    We're no longer allowing tools which scrape the data directly from the game


    You are mistaken. OCR would be taking data direct from the game.

    This method was also tried (successfully) in the past, so it is feasible, but against FD's wishes.


    You could always learn to use the in-game tools and play properly.
    ----------------------------

  3. #3
    Originally Posted by Dynamo View Post (Source)
    You are mistaken. OCR would be taking data direct from the game.
    Not if you save a screenshot, and then OCR the image file. That's not interfacing to the game one iota.

    However, it doesn't really work due to the colourscheme/background (tried it already, even if you try and filter them out), and more importantly, the market listing is larger than you can get on one page, so you need multiple screenshots, so it becomes a pain.

    Andreas's method wasn't based on OCR. I'm not aware of anyone getting an OCR from image solution working.
    C:\> Cannot find Reality.sys. Universe Halted.

    Slopey's Elite: Dangerous Best Profit Calculators!:
    Now retired - other market tools are available.
    Alleged destroyer of the entire Universe economy.

  4. #4
    Originally Posted by Dynamo View Post (Source)
    You are mistaken. OCR would be taking data direct from the game.

    This method was also tried (successfully) in the past, so it is feasible, but against FD's wishes.


    You could always learn to use the in-game tools and play properly.
    OCR'ing a screenshot would not be interrupting or interfering with the execution of the client so wouldn't be prohibited, as far as I can see. Screenshotting doesn't cause unnecessary tickets, which appears to be what this was all about.
    Cmdr ZenSpider
    Google+ Twitter

  5. #5
    Originally Posted by Slopey View Post (Source)
    Not if you save a screenshot, and then OCR the image file. That's not interfacing to the game one iota.
    FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

    Modders: "Screw you, I do what I want, and I don't care if it destroys the game for lots of people".


    ... seriously? Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what? Because you are too lazy to play the game the way it is meant to be played? Because you enjoy destroying the work of others? Because you like to detract from the joy others get who want to get "the real" experience? How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?

  6. #6
    Originally Posted by Dan-Erik View Post (Source)
    FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

    Modders: "Screw you, I do what I want, and I don't care if it destroys the game for lots of people".


    ... seriously? Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what? Because you are too lazy to play the game the way it is meant to be played? Because you enjoy destroying the work of others? Because you like to detract from the joy others get who want to get "the real" experience? How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?
    Uhuh, yeah. This pretty much covers it.


    Stick with me kid and you'll be farting through silk.

  7. #7
    Originally Posted by Dan-Erik View Post (Source)
    How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?
    Because I'll be continuing in a manner which is permitted by FD.

    I personally want a tool which records prices of systems I'm in. If I have to handball them, that's fine, I'll do that. Now, it seems to me I can share that info in a number of ways, via email, teamspeak, phone, forums, chat rooms, IRC, web sites etc... If I choose to make my handballed numbers available on the web for other people to use, that's my choice, and it's not against any FD t&c.

    If you don't like it, don't use any numbers I publish. But it doesn't impact on your game any more than Ben Ryder's post here publicising a smuggling route. Or are you going to go over there and tell him off for making in-game data public also?

    You play the game how you want, and I'll play it how I want. It just happens that I want to share prices with others, whereas you do not.
    C:\> Cannot find Reality.sys. Universe Halted.

    Slopey's Elite: Dangerous Best Profit Calculators!:
    Now retired - other market tools are available.
    Alleged destroyer of the entire Universe economy.

  8. #8
    Players: "I hate taking off my 3d glass to write stuff with pencil and paper, there must be an easier way to save even the data I alone have seen. Sharing it with the community is also good - we would all win"

    enter Slopey and friends. "Here's a way to automate what you're doing anyway using modern tools."

    FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms. (oh and it did once cause a couple of crashes, but not as many as entering jumpspace, supercruise, or frantic data transfer rates admittedly)"

    Players: "OK, but lets see what adding many more players into the mix does on release. If you're struggling now, its probably not going to get better"
    Otherplayers: "We'll find another way to do it, but keep it quiet this time."

  9. #9
    Frontier Developments should just apply for copyright on all the market data generated by their game, kind of like how the Premier League holds the rights to match fixture data and require that people pay for a license to reproduce said data.

  10. #10
    They will hold the copyright to the data and IP within the game already.

    However, you cannot control information you subsequently release to customers unless you force them to sign an NDA saying they'll never disclose any of the contents of the game, which is obviously unworkable.

    You can't copyright someone typing:

    Sol, Gold, 3000, 120

    into a website, as all those terms are already in the public domain - there is prior art, and you can't copyright common words or "things" generally. (note that UK law varies from the US which is generally sillier). You can however prevent someone printing up and selling Cobra Mk III t-shirts, as those are copyrighted - both the name and the imagery.
    C:\> Cannot find Reality.sys. Universe Halted.

    Slopey's Elite: Dangerous Best Profit Calculators!:
    Now retired - other market tools are available.
    Alleged destroyer of the entire Universe economy.

  11. #11
    Originally Posted by Slopey View Post (Source)

    If you don't like it, don't use any numbers I publish. But it doesn't impact on your game any more than Ben Ryder's post here publicising a smuggling route.
    To play devil's advocate here... it actually does impact the game for people who don't want to use these tools.

    Since the market is variable, and uses everyone's input to calculate the prices... if there are a number of people running the same runs using any tool, they are affecting the economy of the game, which in turn affects those who don't use the tools.

    I don't want to get messed up in the arguments for or against the tools, as personally, I don't particularly care. It won't really affect me being a pirate.. .YARRGH! All your loot is mine regardless of what you carry

    I think that disabling this tools though right now in development is a good idea. It will allow FD to concentrate their efforts on fixing issues caused by their client, rather than getting 3rd party tools mixed up in troubleshooting efforts.

  12. #12
    As this topic has already gone OT and turned into pro/contra trading tools like 15+ others threads I reserve myself the right to go OT also.

    I will retype this at the BPC page but to get even more exposure I type it here.

    There are multipe trading route tools which rely on or use the EMDN like Slopeys's BPC and kfsone's Trade Dangerous (optinal because its normal mode is manual data entry) and the web-api by lx. Why not resurrect the EMDN in a way that is compliant with FD.

    So my request at Slopey. When you release the BPC with manual entered data would it be possible to do the following:

    1. Send the data in EMDN format to your site.
    2. Install EMDN's firehose so mod authors can use the data like they did in the past.
    3. Read your own firehose and then store it in the your database and proceed as normal.


    Of course you could do it the other way around. Upload manual entered data to your central database and make the data available via a firehose in EMDN format.

    The reason for the first suggestion. There are some other trading tools available which allow users to upload data to their website which are then used. Suppose those others would send in EMDN format the data also to your site also. Instead of having manually data spread out at different sitesit is combined into one source. See also two posts in the thread "Elite Advisor - Maximise your trading potential!" where users can manually enter data. In particular post1 and post2. From the last post I quote the following: "A central data repository shared between all sites? Now there's an idea with merit! If anyone's interested in collaborating on such a project then please feel free to drop me a line, either here or via the Elite Advisor Contact Us page."

    In short: There must be a way to combine all spread out data in a 'phoenix' EMDN so all mod authors can benefit.

  13. #13
    Originally Posted by Flappers View Post (Source)
    I think that disabling this tools though right now in development is a good idea. It will allow FD to concentrate their efforts on fixing issues caused by their client, rather than getting 3rd party tools mixed up in troubleshooting efforts.
    On this I wholeheartedly agree. I'm amazed they tolerated it as long as they did, and that's commendable.
    C:\> Cannot find Reality.sys. Universe Halted.

    Slopey's Elite: Dangerous Best Profit Calculators!:
    Now retired - other market tools are available.
    Alleged destroyer of the entire Universe economy.

  14. #14
    Originally Posted by wolverine2710 View Post (Source)
    I will retype this at the BPC page but to get even more exposure I type it here.
    Please don't - duplication is very irritating.

    So my request at Slopey. When you release the BPC with manual entered data would it be possible to do the following:

    [snip]
    Most likely no, but I've not decided yet. It'll depend on if anyone can actually be bothered typing numbers into the BPC - which is a high barrier to cross to start with. It won't happen multiple times per station visit (as you'll likely do it only once) and as it's not automatic I'd expect most people not to bother, so the firehose becomes less attractive compared to when it was used with the marketdump which was repeatedly squriting high volumes of data around. However, that remains to be seen.

    It can be done sans firehose with a back end database - the performance of the amazon database I've been using has been negligible, even at max load. Having the firehose just adds another layer to sending the info to the DB directly - it's duplication on the firehose and listener side. If people want to query the amazon database, I can allow that. Should the traffic get too much then a firehose might be worth it - but with manual entry, most users won't bother and I'd expect the BPC to turn niche instantly.
    C:\> Cannot find Reality.sys. Universe Halted.

    Slopey's Elite: Dangerous Best Profit Calculators!:
    Now retired - other market tools are available.
    Alleged destroyer of the entire Universe economy.

  15. #15
    Originally Posted by Slopey View Post (Source)
    They will hold the copyright to the data and IP within the game already.

    However, you cannot control information you subsequently release to customers unless you force them to sign an NDA saying they'll never disclose any of the contents of the game, which is obviously unworkable.

    You can't copyright someone typing:

    Sol, Gold, 3000, 120

    into a website, as all those terms are already in the public domain - there is prior art, and you can't copyright common words or "things" generally. (note that UK law varies from the US which is generally sillier). You can however prevent someone printing up and selling Cobra Mk III t-shirts, as those are copyrighted - both the name and the imagery.
    However, they can change the EULA and make you agree to it upon installation, like every other piece of software does today, surely?

Page 1 of 14 1234511 ... LastLast