Discussion Has any one experimented with OCR tools for scraping the commodities market prices?

[wolverine2710]

Probably most commanders have read this news dated 20140925) by Michael Brookes: "We're no longer allowing tools which scrape the data directly from the game or through its communications. We will be considering an external API for future development, but for now these tools are not permitted."

This killed EMDN which fed trading tools like Slopey's BPC and a few other trading tool websites. The BPC will be changed so that prices can be entered manually. But of course this is suboptimal.

Just wondering. Has any one been experimenting with (free) OCR tools to convert a commodities market screenshot into text. If that is feasible it would mean some kind of EMDN would be feasible in the near future. The above would be in compliance with FD if I'm not mistaken.

Edit: To clear this up I send a PM on the 11th of November 2014 to Michael Brookes. I asked for permission to share his response and was given permission.

Spoiler

Hello Mr Brookes,

I have created the "Has any one experimented with OCR tools for scraping the commodities" thread. Basically a commander takes a screenshot, which is automatically read by an OCR tool which then creates a text file with commodities market prices. This can be used for for example trading tool purposes. For example feeding the text to Trade Dangerous program or Slopeys the BPC. Both are based upon manually inputting of data. Lets make this totally absolutely 100% clear. The end goal will be an EDDN - Elite Dangerous Data Network. Basically Andreas EMDN but filled with data obtained from OCR-ing. NOT by scraping the ED program/protocols. Also when OCR-ing is successful it will most likely be extended to retrieve more information.

When FD's data access policy changed you wrote: "We're no longer allowing tools which scrape the data directly from the game or through its communications. We will be considering an external API for future development, but for now these tools are not permitted."

A commander stated he has a (near) working OCR solution. He is reluctant to release it because he's not sure it would violate the new FD data access policy. There are two possibilities.

1. OCR does violate FD's new data access policy.
2. OCR does NOT violate FD's new data access policy.

I hope you can give me a response back which states which of the two is true. That way I can communicate it back to the author and the tread in which the question was raised.

With kind regards,
Wolverine2710 aka cmdr Wolverine

His response:

As long as the method doesn't alter or access the exe or the game resources then it is fine. So OCR isn't an issue for us, although trying to read the frame buffer directly could be.

Please do NOT turn this into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". .
I hope and trust you respect my wishes.

 

[Dynamo]

We're no longer allowing tools which scrape the data directly from the game

You are mistaken. OCR would be taking data direct from the game.

This method was also tried (successfully) in the past, so it is feasible, but against FD's wishes.


You could always learn to use the in-game tools and play properly.

 

[Slopey]

You are mistaken. OCR would be taking data direct from the game.

Not if you save a screenshot, and then OCR the image file. That's not interfacing to the game one iota.

However, it doesn't really work due to the colourscheme/background (tried it already, even if you try and filter them out), and more importantly, the market listing is larger than you can get on one page, so you need multiple screenshots, so it becomes a pain.

Andreas's method wasn't based on OCR. I'm not aware of anyone getting an OCR from image solution working.

 

[ZenSpider]

You are mistaken. OCR would be taking data direct from the game.

This method was also tried (successfully) in the past, so it is feasible, but against FD's wishes.


You could always learn to use the in-game tools and play properly.

OCR'ing a screenshot would not be interrupting or interfering with the execution of the client so wouldn't be prohibited, as far as I can see. Screenshotting doesn't cause unnecessary tickets, which appears to be what this was all about.

 

[Dan-Erik]

Not if you save a screenshot, and then OCR the image file. That's not interfacing to the game one iota.

FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

Modders: "Screw you, I do what I want, and I don't care if it destroys the game for lots of people".


... seriously? Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what? Because you are too lazy to play the game the way it is meant to be played? Because you enjoy destroying the work of others? Because you like to detract from the joy others get who want to get "the real" experience? How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?

 

[Joe Spivey]

FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

Modders: "Screw you, I do what I want, and I don't care if it destroys the game for lots of people".


... seriously? Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what? Because you are too lazy to play the game the way it is meant to be played? Because you enjoy destroying the work of others? Because you like to detract from the joy others get who want to get "the real" experience? How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?

Uhuh, yeah. This pretty much covers it.

 

[Slopey]

How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?

Because I'll be continuing in a manner which is permitted by FD.

I personally want a tool which records prices of systems I'm in. If I have to handball them, that's fine, I'll do that. Now, it seems to me I can share that info in a number of ways, via email, teamspeak, phone, forums, chat rooms, IRC, web sites etc... If I choose to make my handballed numbers available on the web for other people to use, that's my choice, and it's not against any FD t&c.

If you don't like it, don't use any numbers I publish. But it doesn't impact on your game any more than Ben Ryder's post here publicising a smuggling route. Or are you going to go over there and tell him off for making in-game data public also?

You play the game how you want, and I'll play it how I want. It just happens that I want to share prices with others, whereas you do not.

 

[Codec]

Players: "I hate taking off my 3d glass to write stuff with pencil and paper, there must be an easier way to save even the data I alone have seen. Sharing it with the community is also good - we would all win"

enter Slopey and friends. "Here's a way to automate what you're doing anyway using modern tools."

FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms. (oh and it did once cause a couple of crashes, but not as many as entering jumpspace, supercruise, or frantic data transfer rates admittedly)"

Players: "OK, but lets see what adding many more players into the mix does on release. If you're struggling now, its probably not going to get better"
Otherplayers: "We'll find another way to do it, but keep it quiet this time."

 

[Armour]

Frontier Developments should just apply for copyright on all the market data generated by their game, kind of like how the Premier League holds the rights to match fixture data and require that people pay for a license to reproduce said data.

 

[Slopey]

They will hold the copyright to the data and IP within the game already.

However, you cannot control information you subsequently release to customers unless you force them to sign an NDA saying they'll never disclose any of the contents of the game, which is obviously unworkable.

You can't copyright someone typing:

Sol, Gold, 3000, 120

into a website, as all those terms are already in the public domain - there is prior art, and you can't copyright common words or "things" generally. (note that UK law varies from the US which is generally sillier). You can however prevent someone printing up and selling Cobra Mk III t-shirts, as those are copyrighted - both the name and the imagery.

 

[Flappers]

If you don't like it, don't use any numbers I publish. But it doesn't impact on your game any more than Ben Ryder's post here publicising a smuggling route.

To play devil's advocate here... it actually does impact the game for people who don't want to use these tools.

Since the market is variable, and uses everyone's input to calculate the prices... if there are a number of people running the same runs using any tool, they are affecting the economy of the game, which in turn affects those who don't use the tools.

I don't want to get messed up in the arguments for or against the tools, as personally, I don't particularly care. It won't really affect me being a pirate.. .YARRGH! All your loot is mine regardless of what you carry

I think that disabling this tools though right now in development is a good idea. It will allow FD to concentrate their efforts on fixing issues caused by their client, rather than getting 3rd party tools mixed up in troubleshooting efforts.

 

[wolverine2710]

As this topic has already gone OT and turned into pro/contra trading tools like 15+ others threads I reserve myself the right to go OT also.

I will retype this at the BPC page but to get even more exposure I type it here.

There are multipe trading route tools which rely on or use the EMDN like Slopeys's BPC and kfsone's Trade Dangerous (optinal because its normal mode is manual data entry) and the web-api by lx. Why not resurrect the EMDN in a way that is compliant with FD.

So my request at Slopey. When you release the BPC with manual entered data would it be possible to do the following:

1. Send the data in EMDN format to your site.
2. Install EMDN's firehose so mod authors can use the data like they did in the past.
3. Read your own firehose and then store it in the your database and proceed as normal.

Of course you could do it the other way around. Upload manual entered data to your central database and make the data available via a firehose in EMDN format.

The reason for the first suggestion. There are some other trading tools available which allow users to upload data to their website which are then used. Suppose those others would send in EMDN format the data also to your site also. Instead of having manually data spread out at different sitesit is combined into one source. See also two posts in the thread "Elite Advisor - Maximise your trading potential!" where users can manually enter data. In particular post1 and post2. From the last post I quote the following: "A central data repository shared between all sites? Now there's an idea with merit! If anyone's interested in collaborating on such a project then please feel free to drop me a line, either here or via the Elite Advisor Contact Us page."

In short: There must be a way to combine all spread out data in a 'phoenix' EMDN so all mod authors can benefit.

 

[Slopey]

I think that disabling this tools though right now in development is a good idea. It will allow FD to concentrate their efforts on fixing issues caused by their client, rather than getting 3rd party tools mixed up in troubleshooting efforts.

On this I wholeheartedly agree. I'm amazed they tolerated it as long as they did, and that's commendable.

 

[Slopey]

I will retype this at the BPC page but to get even more exposure I type it here.

Please don't - duplication is very irritating.

So my request at Slopey. When you release the BPC with manual entered data would it be possible to do the following:

[snip]

Most likely no, but I've not decided yet. It'll depend on if anyone can actually be bothered typing numbers into the BPC - which is a high barrier to cross to start with. It won't happen multiple times per station visit (as you'll likely do it only once) and as it's not automatic I'd expect most people not to bother, so the firehose becomes less attractive compared to when it was used with the marketdump which was repeatedly squriting high volumes of data around. However, that remains to be seen.

It can be done sans firehose with a back end database - the performance of the amazon database I've been using has been negligible, even at max load. Having the firehose just adds another layer to sending the info to the DB directly - it's duplication on the firehose and listener side. If people want to query the amazon database, I can allow that. Should the traffic get too much then a firehose might be worth it - but with manual entry, most users won't bother and I'd expect the BPC to turn niche instantly.

 

[Armour]

They will hold the copyright to the data and IP within the game already.

However, you cannot control information you subsequently release to customers unless you force them to sign an NDA saying they'll never disclose any of the contents of the game, which is obviously unworkable.

You can't copyright someone typing:

Sol, Gold, 3000, 120

into a website, as all those terms are already in the public domain - there is prior art, and you can't copyright common words or "things" generally. (note that UK law varies from the US which is generally sillier). You can however prevent someone printing up and selling Cobra Mk III t-shirts, as those are copyrighted - both the name and the imagery.

However, they can change the EULA and make you agree to it upon installation, like every other piece of software does today, surely?

 

[Armour]

I'm amazed they tolerated it as long as they did

That didn't stop you and others from exploiting their tolerance, apparently. And that's why we can't have nice things.

 

[ShadowGar]

Since the market is variable, and uses everyone's input to calculate the prices... if there are a number of people running the same runs using any tool, they are affecting the economy of the game, which in turn affects those who don't use the tools.

No, people not using the tool are trading information online with what is the best routes for money, those people are hammering the market killing routes.

I noted before where popular routes that were not profitable compared to other routes were being hammered to negative profits. Why would people trade on a route that wasn't as profitable? It's all they knew about at the time. The mass majority of people found a few routes and everyone followed suit trading on those routes. This in turn resulted from people wondering why what shown a profit of 2k is now -400. People using those tools saw the market drops and instead of further hammering an already dead market commodity. They moved on to other systems while a bunch of pen and paper still took the same routes only making half of what they made hours previous thinking it was still the best profits for their time.

People want to say that the markets are not rocket science, but when you have the entire market in front of you and see the massive swings of prices, it is definitely much more complicated to stay ahead of the market even with the tools.

Anyone can find a +whatever number trade and just run that all day to make money. Doing that with the optimal profits based on time is much much harder. I really think that if everyone saw how much the market fluctuates entirely and not in their little routes, they would be more understanding of why the tools are here and what a waste of time they spent trading at some of the more popular places.

Its very interesting that when people on other websites started to talk about market fluctuations and possible flaws in the market because we could see the whole market, its shut down.

I'll buy the crashing issue and causing bug reports. But banning scraping wasn't the solution. This is a single program that was scraping. That crash could have been fixed much faster then adding an API. Or they could have just said that if your running the tool expect crashes and don't complain. But to say its effecting the market more than other methods is an excuse from someone that obviously is not either playing the trading game or just really has convinced themselves that they are right.

Would the tools be allowed if it never caused CLIENTside crashes? I don't think so. Its a threat to people that believe that what I do with my credits right now effect them.

One last thing. These tools allowed a large majority of beta testers to see how the market worked and find all the problems.Now that you are forcing us to go blind, any short or long term effects from poor market implementation will be solely up to FD to find now, which frankly is a waste of their time since we had the community already watching it.

So unless Slopey or others can figure out a way to get the same results from different methods, people will just use illegal methods or get into the information trading game selling the best routes and only keeping the best profitable routes within groups. This opens up a new way to now exploit the market and keep those exploits from the mass majority or players.

I've said about all I'm going to say about this... I'll sit back and wait for the "I told you so moment."

 

[tsh]

These arguments with some players saying what it the right way to play the game to the exclusion of others have destroyed my interest. I was starting to wonder if multi-player added anything, and I think the answer is just more angst.

That was an expensive experience.

 

[dungeonseeker]

However, they can change the EULA and make you agree to it upon installation, like every other piece of software does today, surely?

EULAs are not worth the data they take up. They cannot supercede any local "laws of the land" and the certainly cannot stop you from discussing the game with other people.

 

[Armour]

EULAs are not worth the data they take up. They cannot supercede any local "laws of the land" and the certainly cannot stop you from discussing the game with other people.

But couldn't they force you to play in offline mode exclusively, if you were in violation of their T&Cs?

I mean, Steam can basically steal your money and ban you on the slightest pretext, and they didn't even write those games. So couldn't Frontier do something a lot less drastic? You wouldn't be completely unable to play - you would just be unable to use FD's infrastructure to play online.