Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[kfsone]

So my plan is the server stores all the updates. When you connect you will tell the server the date-time stamp of the last time you (dis)connected and it will catch you up. While you are connected you receive updates.

Don't use a timestamp for this; in particular, connection and disconnection are non-discrete, non-atomic events, and especially they are prone to preemption. It's quite possible that any number of transactions could occur during the time it takes you to tear up a connection and begin receiving non-handshake data; it's also quite possible that any number of transactions could occur between the last packet that was delivered to you and your application becoming aware of a disconnect.

If you're trying to journal transactions, you need to use a serial number that you can guarantee is unique+ordered, and for banking probably sequential. In many cases, this is where you offload to a worker that writes synchronously to the DB, waits for the commit to complete and captures the resulting row id to throw back to the originator.

 

[kfsone]

I have it working. Kibana after 10 minutes shows 560+ messages have been collected. I had to massage the lot (ansible.ymll wrt Pleasrun) a bit to get it working though. Will install a new VM with vagrant/ansible and massage it so that it WILL install correctly under windows in one go.

Next steps:

• Watching Andreas ELK/kibina video and other Kibana info.
• Get info from the archiving services by Askarr and Themroc to fill the NoSQl database ElasticSearch database.
• Delving into Python to achieve the above
• Installing the lot manually or perhaps with ansible on the EDDN server.

Forgot how much fun Linux, programming can be OUTSIDE office hours ;-)

Vagrant can be a lot of fun to play with But are you actually nesting virtual machines instead of just, well, having two virtual machines?

 

[wolverine2710]

Vagrant can be a lot of fun to play with But are you actually nesting virtual machines instead of just, well, having two virtual machines?

I'm using Windows 7 64 bit with 16GB as a host. I run virtualbox under windows. Vagrant (in combination) with the provioner ansible is used to setup the Linux VM. In case of the ELK stack (well really EK stack as LogStash is not used but a collector.py script to put the data in the ElastSearch database) its running Debian 7.7. On the EDDN server we have CentOS 6.6. My plan is to play with the VM a bit. Importing historical data into ElasticSearch (ES) - learning Python in the process. Once I'm confident I have enough experience in setting it all up I'm going to setup an EK/ELK stack on the EDDN server. For personal use first. If I can get ElasticSearch secured enough (nothing out of the box in ElasticSearch but there are tools/methods to get it secure) I probabky want to open it up to the world. Commanders then only need Kibana installed on their system (pretty easy), can connecte to the EDDN ES, have all the ES data and can then play with it. Iirc one can share Kibana dashboard with others.... For real data mining purposes we need more data (OCR-ing gives not enough data) so I'm hoping the upcoming FD api will give us the prices data and a LOT more commanders are uploading that to EDDN.

 

[dwwolf]

Thank you for destroying trading as a profession.
It happened in EvE it will happen here.

 

[Andargor]

Thank you for destroying trading as a profession.
It happened in EvE it will happen here.

Ok I'll bite, you win. I see you didn't read the OP.

 

[Andargor]

Warning! It seems at least one station will be moved in 1.1, unless I am reading this wrong (it might be a special kind of roaming station):

Jacques’ Roaming station is readying to depart
We are looking to test moving an Orbis between systems
The station is Jacques starport in Alshain
We will be moving the station during Fridays update.
It would be good to have a mix of people docked at the station and parked outside before it moves.
Test purchasing goods form the commodities market Particularly “Jaques Quinentian Still” before it moves
and selling them once it has moved.

 

[seeebek]

I saw today that there are still trolls. One of them sent data to EDDN with Platinum sell price of over 19000. Which is 5000 more than best Palladium sell price.

EliteOCR creates unique userID which stays the same(in windows registry) and is not changing even after an update. It means everybody who discovers a troll uploading wrong data can put this persons uploaderID on a blacklist and ban him (from your service). Only people, who know where the userID is, are able to change it and 99.9% of trolls simply don't know it so this should be very effective.

 

[themroc]

I saw today that there are still trolls. One of them sent data to EDDN with Platinum sell price of over 19000. Which is 5000 more than best Palladium sell price.

EliteOCR creates unique userID which stays the same(in windows registry) and is not changing even after an update. It means everybody who discovers a troll uploading wrong data can put this persons uploaderID on a blacklist and ban him (from your service). Only people, who know where the userID is, are able to change it and 99.9% of trolls simply don't know it so this should be very effective.

But best Platinum sell price is around 19k. Whats wrong with that?

 

[Askarr]

EliteOCR creates unique userID which stays the same(in windows registry) and is not changing even after an update. It means everybody who discovers a troll uploading wrong data can put this persons uploaderID on a blacklist and ban him (from your service). Only people, who know where the userID is, are able to change it and 99.9% of trolls simply don't know it so this should be very effective.

Looking through the EDDN code, it also looks like the uploader IP can be passed downstream as a salted & hashed identifier, which could give us another unique ID while being safely anonymous*. Worth discussing with James as to whether we can get this enabled, assuming it's actually a reasonable cryptographic implementation.

I took a look at enabling a keep alive from within the gateway, but still trying to get my head around the greenlet + bottle code. I saw that James added a UTC fix for the timestamp (much appreciated), but I don't think it's been deployed yet (current gateway still seems to send US times)?

* Interesting question over whether uploader ID (whether from EliteOCR or EDDN) is anonymous 'enough' considering one could use a static ID to track where an individual commander has been based on their uploads. I could imagine targets of bounty hunting being less than thrilled to discover their regular routes are mappable in EDDN's data feed. However it would require a lot of correlation to de-anonymise the data - but a determined observer could match up station locations and times where a given player is present, I suppose.

Warning! It seems at least one station will be moved in 1.1, unless I am reading this wrong (it might be a special kind of roaming station):

For any who didn't get the Elite newsletter, this one will be roaming, so that notion that stations & systems are static no longer entirely holds.

 

[seeebek]

But best Platinum sell price is around 19k. Whats wrong with that?

Upps... my bad. I confused Platinum and Palladium. I thought the one was the other. Sorry guys...

 

[DRY411S]

Is there any reason that you don't broadcast the Supply/Demand levels (i.e. L M and H)?

These are important to some work I'm doing with my data source (maddavo). I'm uploading my updates religiously to EDDN, but these values are getting lost on the way to maddavo.

 

[seeebek]

Is there any reason that you don't broadcast the Supply/Demand levels (i.e. L M and H)?

These are important to some work I'm doing with my data source (maddavo). I'm uploading my updates religiously to EDDN, but these values are getting lost on the way to maddavo.

When I did EDDN Export I had problems to get it working, so I just took the minimum of data needed. I can include the Levels in the next version. Please note that many people are still using old versions and it is impossible to get them to upload. Only people who update will also export this data.

 

[Askarr]

Is there any reason that you don't broadcast the Supply/Demand levels (i.e. L M and H)?

Please forgive any accidental pedantry here, but EDDN itself just aggregates JSON. The commodity JSON schema has support for supply/demand levels. Each uploader software (e.g. EliteOCR, RegulatedNoise etc.) is at liberty to choose to upload these values or not (as they are not currently required). I'd suggest reaching out to the relevant owners of the uploader software in question - it may be a thorny OCR issue, or just a feature they've not got to.

(Edit: ninja'd by seeebek far more accurately )

That said, this raises a possible bug/improvement for the next iteration of schema - I'm not sure the current schema should be listing 'demand' as required, when demandLevel, supply, and supplyLevel are not - this gives us a lot of 'demand: 0' entries for all commodities not in demand at a given station. Just a thought though.

Possibly, assuming JSON schema could enforce it (and I lose track of what draft can do what), we might require demand & demandLevel OR supply & suppyLevel.

 

[DRY411S]

When I did EDDN Export I had problems to get it working, so I just took the minimum of data needed. I can include the Levels in the next version. Please note that many people are still using old versions and it is impossible to get them to upload. Only people who update will also export this data.

Thanks for helping me understand. If it appears in a later version, that'd be a great help thanks. What's happening at the moment is that I need the LMH for some work I'm doing. EliteOCR gets it OK for my local Tradeangerous .prices file, but the next day when I get maddavo's data, the LMH are all gone and I lose them.

If others use older versions, then I don't care so much. It's just some very local data I'm playing with that's important to me, but I don't want to lose the ability to get the other crowd sourced data, so I must keep downloading maddavo's updates.

 

[Loriath]

So what is the status of EDDN integration by third party tools?

I have been using EliteOcr to update my preffered trading tool as well as pushing all my scans to EDDN. I can't wait for sites/tools to pull EDDN data.

 

[seeebek]

So what is the status of EDDN integration by third party tools?

I have been using EliteOcr to update my preffered trading tool as well as pushing all my scans to EDDN. I can't wait for sites/tools to pull EDDN data.

The best(and for now only) I know is eddb.io.
It's awsome because it offers basic price search, shows graphs of commodities over time and the best: you can download all the data in json format for your own use (updated daily). Additionally you can contribute additional information on stations like landing pad size and distance from the star.

 

[wolverine2710]

So what is the status of EDDN integration by third party tools?

I have been using EliteOcr to update my preffered trading tool as well as pushing all my scans to EDDN. I can't wait for sites/tools to pull EDDN data.

Quite a few tools use the EDDN data. Far MORE then I could have dreamed off. A partial list can be found on the OP of this thread - its in one of the spoiler tags. Its NOT complete because I haven´t got through the whole to collect all the gems. Aside from what Seebek said there are also archiving services (if you want historical data) AND for Trade Dangerous there is MMS (Maddavo´s Market Share) which subscribes to EDDN. His plugin is part of TD and can be used to upload your .prices file to MMS. It then gets merged with the existing data and you get the latest data back. You can also use it to retrieve the latest .prices file for TD.

If you want all gems you have to read the whole thread. Should you choose to do that I wouldn´t mind if you post the new and not on the OP gems here ;-) That way I can easily update the OP. I´m pretty sure 6++ gems are missing atm.....

 

[Loriath]

Well I see my preferred tool is listed but I don't know if it is actively taking the data. I will have to ask the Author. (not going to name as I am not sure if I missed the information or the tool does not use it).

The idea of crowd sourcing this information is a fantastic idea and a win for the community. I did read most of the thread, not sure if most of it has been understood or absorbed (may not be room since I can't even remember where my keys are at the moment ).

 

[wolverine2710]

Well I see my preferred tool is listed but I don't know if it is actively taking the data. I will have to ask the Author. (not going to name as I am not sure if I missed the information or the tool does not use it).

The idea of crowd sourcing this information is a fantastic idea and a win for the community. I did read most of the thread, not sure if most of it has been understood or absorbed (may not be room since I can't even remember where my keys are at the moment ).

When the FD api is ready AND it will give us at least the basics, ie comodities princes I´m pretty sure EDDN will receive a LOT more and 100% accurate data - explodes in a positive way. I also hope that FD will implement OAuth, OAuth2, OpenID or something similar to prevent poisoning the well. Also extremely useful for every other websites/tools authors which need authentication/authorization. Instead of commanders having to reinvent the wheel FD effectively becomes an authorization server. They could also decide to implement something similar to what EVE online uses - hashkeys I believe a commander mentioned in a PM.

I think Metcalfe´s law applies here as well. When EDDN becomes more valuable (more and better data) I´m sure more and more tool authors are going to use it. Slopey´s BPC and Cmdr Thrudds tool come to mind - and even more creative/talented commanders which create tools for EDDN). It ALSO means EDDN must mature (new Json format which has info for a complete station) and become redundant, scalable (we now have 20 Mbit unmethered and one Athlon CPU provided for free by a hosting provider. If not for them their might not have been an EDDN!!!). Luckily EMDR which is pretty much what EDDN is (and Andreas EMDN which started it all) takes care of that. EDDN is a very basic form of EMDR which resides on one server. The future is looking VERY bright indeed. We just need plenty of good data!

Edit: Commanders who have issues with finding their keys (same goes for glasses) probably don´t have problems with aborbing other info ;-)

 

[seeebek]

I mixed up supplyLevel and demandLevel in EDDN export in version 0.5.2.2. It's corrected now(0.5.2.3). Please update if you export to EDDN.
If you are on the receiving end, please ignore supplyLevel and demandLevel if the version is 0.5.2.2 (they are switched)