Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

Status
Thread Closed: Not open for further replies.

Snuble

S
Is there anyone sharing market data snapshots, thinking in the form of "download market data in a 50LY radius" style of downloads.
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
Snuble said:
Is there anyone sharing market data snapshots, thinking in the form of "download market data in a 50LY radius" style of downloads.
Click to expand...

I've checked in ED3PTT the category EDDN (see OP of the third party tools thread) and I've found the following relevant entries. Copy/paste didn't really work, to much urls. So I'm pointing you to the posts in the third party toolds thread.

Author: Askarr
Tool: No official name, I've named it "Archive of EDDN messages"
See this Post for details.


Author: Pandemic
Tool: Elite History Server
Some info from ED3PTT in the spoiler tag
For those looking for historical EDDN data, I have built a cache system and a new protocol to deal with the problem of sending commodities one at a time. This system is based on protocol buffer, which in testing reduced the bandwidth taken by the messages from about 240 bytes zlib compressed JSON to around 95 bytes uncompressed protocol buffers. In addition, this system supports batching. When you need data, you call the endpoint with the last Unix timestamp of data you receive and it will send you everything since then. The code for the server is located at https://github.com/huadianz/elite-eddn-history and includes the protocol buffer schemas you need to receive the data. In addition, it also contains an example program that shows you how to unpack the protocol buffer binary stream.

The current server I am running is located at http://elite.servers.henryzh.com/api/v0/update/<timestamp>
The API will send you up to the last 1000 missed updates with Unix timestamps in MS >= the timestamp in the system. If you need more you will need to query the server again.

As this project is open source I welcome any suggestions or if you have any code you want to commit, that would be great. As for already planned features:

Update the network using batched commodity updates
Current status server (query the status of a station in its most recent state)
Timeseries server with influxdb
Using commodity IDs instead of names, if we get those from Frontier at any point

Author: Themroc
Tool: eddb - Elite: Dangerous Database
EDDB thread.
I believe there is an API and there are data dumps.

You might consider looking into those and PM-ing the authors for more info.

Note: Its possible that there are commanders who have created similar tools. If that data is freely available please PM me or mention it here. So I can put it in the third party tools thread and ED3PTT.
 
Last edited:

Otis B.

Otis B.
Snuble said:
Is there anyone sharing market data snapshots, thinking in the form of "download market data in a 50LY radius" style of downloads.
Click to expand...
eddb has a record of EDDN messages here. They're not available for download in bulk but if you're using an online tool like eddb or Slopey's BPC then you don't need to download market data manually.
If you're using Trade Dangerous you can download market data from Maddavo's, which includes data from EDDN.

wolverine2710 said:
Its however a pity that most data of EDMC is sent in V1 not V2 and cmdrs haven't upgraded all to the newest version.
Click to expand...
That's because the latest released versions of EliteOCR and E: D Market Connector are still sending v1 because we're waiting on themroc to update eddb to handle v2. *Nudge*.
 
Last edited:

Andargor

Andargor
Otis B. said:
That's because the latest released versions of EliteOCR and E: D Market Connector are still sending v1 because we're waiting on themroc to update eddb to handle v2. *Nudge*.
Click to expand...

No pressure. :D

54250404.jpg
 

Eggplant!

Eggplant!
Hey, any plan for EDDN to support more than just market prices?

For example, the mobile API provides detailed info about what ships and modules are for sale at your docked station. Having a module DB would be nice.
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
Powerplay 1.3 commodities: Does somebody have a full list of new commodities?
So far I know of osmium and sap 8... Are there more new commodities?
I know someone who is very interested in this and needs the info for a new release of his tool ;-)
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
wolverine2710 said:
Powerplay 1.3 commodities: Does somebody have a full list of new commodities?
So far I know of osmium and sap 8... Are there more new commodities?
I know someone who is very interested in this and needs the info for a new release of his tool ;-)
Click to expand...

Just read (in an email) that Anthornet is going to supply the author with the needed information
 
Last edited:

wolverine2710

Tutorial & Guide Writer
wolverine2710
Important announcement from the EDDN team:
Starting the 1st of July 2015 EDDN schema V1 will be discontinued.

On the first of July a new version of the EDDN will be deployed/installed by jamesremuscat and/or AnthorNet. EDDN schema V1 messages send to the EDDN will be rejected. A http error code 426 (Upgrade Required) will be returned to the tools sending the data. Hence no more V1 messages will be distributed by the EDDN.

Steps taken:

  1. The 15th of April. Announcement of a draft for schema V2. Everybody could make suggestions. V2 was also implemented in the EDDN from that date - for testing senders and receivers.
  2. The 5th of June. Schema V2 is finalized and it was announcement that in the near future EDDN would switch to EDDN schema V2.
  3. This week (afaik) all the tool authors have been informed in a 'mailinglist'. The consensus was: "1st of July is OK"
  4. Otis B. has created a "Retire v1 commodities schema" pull request and tested it with a few older clients on his own private server.
  5. Next week Anthornet (together with Otis B.) is going to implement the new EDDN version on a test server. This makes it possible for authors to check how their clients react to the http error code 426 when V1 messages are rejected.
  6. All publicly released tools sending data to the EDDN now have support for V2. Note; Duke of Regulated Noise DJ-version will add V2 support (sender and receiver) this weekend.
  7. EDDB.io supports both V2 and V1. Maddavo supports V2 and V1. Also the tools by Andargor and Anthornet support V2.
  8. This weekend the remaining authors who have stated here in this thread that they receive EDDN messages will get a PM.

Two new additional EDDN schemas
Currently work is ongoing for two new EDDN schemas: Outfitting and ShipYard. A not complete list of commanders working on that are Otis B. , Anthornet and also the commander who implemented the EDDN: jamesremuscat. Depending on tests and input by everyone its possible that those schemas might be implemented in the new EDDN version - for testing/evaluation. Note: A version of ED Market Connector that implements both of these schemas can be found here. If I've interpreted it correctly the new code by Anthornet in the recently release EDDN 0.4 supports multiple relays. Hence the new data could be distributed from there. I would say, let yourself be heart and participate in the construction of the schemas.

I hope that considering all taken precautions that the transistion to the new "EDDN V2 schema" version will go smoothly.

Should there be questions, feel free to ask them.
I'm sure the combined knowledge of the tool authors in the "mailinglist" will be able to provide an answer.

To end this post:
I encourage every tool author who is receiving EDDN data to upgrade to EDDN schema V2 as soon as possible.

Imprtant Updates to this post.
Reason(s): I forgot some crucial information and/or new developments/info became available. I've put it here so that when I mention the phasing out of V1 in other threads commanders reading it here have a better overview without having to read all post after this. Ofc please do read the other post ;-)

20th of June. Post #640. Otis B. creator of Elite: Dangerous Market Connector (EDMC).
"Part of the reason that I support dropping v1 is that most of the publishing apps that are sending v1 are old and don't have support for the E: D 1.3 commodities. Also, according to the stats, a significant number of messages are from people using EliteOCR 0.5, and those people certainly should be encouraged to upgrade to a version of EliteOCR that uses the opencv engine. I'm assuming that dropping v1 support will cause the vast majority of such people to upgrade, rather than give up."

20th of June. Post #643. AntorNet and Otis B. together made the following possible: A publicly accessible EDDN version (it rejects V1 messages and returns HTTP errorcode 426) which can be used by tool authors to see how their tools behave when the EDDN refuses the V1 schema. For more info see post #643.
 
Last edited:

themroc

themroc
wolverine2710 said:
On the first of July a new version of the EDDN will be deployed/installed by jamesremuscat and/or AnthorNet. EDDN schema V1 messages send to the EDDN will be rejected. A http error code 426 (Upgrade Required) will be returned to the tools sending the data. Hence no more V1 messages will be distributed by the EDDN.
Click to expand...

I recommend to analyze http://ross.eddb.io/eddn/log right before the date is due and consider postponing the action if too many v1 messages are still being sent. Of course this shall not encourage creators to not update their tools, but I see alot of old versions still being used. It might not be clever to cut those off if their share is still reasonably high.
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
themroc said:
I recommend to analyze http://ross.eddb.io/eddn/log right before the date is due and consider postponing the action if too many v1 messages are still being sent. Of course this shall not encourage creators to not update their tools, but I see alot of old versions still being used. It might not be clever to cut those off if their share is still reasonably high.
Click to expand...

In the "mailing list" I mentioned all senders (EDAPI,EDMC,EDCE, Maddavo, EOCR, RN) of data and most of the receivers (which have public tools like maddavo) are included. I didn't have your email address (and forgot to ask) so you weren't included, my bad. Please send me a PM and you will be included in it - you can see the old replies also. These authors can inform their user using OP and/or post in their threads. I will also post there. Till the update and shutdown of V1 messages we have approx 11 days left. Should be enough to inform everybody. I can even say 3 days before the 1st create a new thread in the Dangerous Discussion forum to inform all commanders if still a large percentage is V1. At least two sender tools have either an auto update mechanism or a mechanism to send a warning (EDMC only latest versions).

I believe yesterday afternoon or so Seebek released EROCR 0.7 (has auto update function iirc) with V2 support only. If you look at the EDDN-monitor here, select EOCR and click on it you get the details for EOCR and which EOCR version send how much data.

With about 2 hours to go (23:00 here in my TZ) before iirc the stats reset.

Code: 
Schema         V1         V2
Yesterday  7356          22
Today       3819          85

V1: each message is one commodity price.
V2: each message contains complete prices for a whole market. From limited tests I did, I guesstimate a station has between 40-60 commodities. Lets avg that to 50 prices for each message. Hence multiply by 50 to convert from V2 to V1. The same info now looks like this.

Code: 
Schema         V1         V2 converted   Total  Percentage V2
Yesterday  7356          1100               8456  13 %
Today       3819          4250               8068  52,7 $

Hence more then half of the data EOCR sends is now V2 (converted). Yesterday it was 13%. Hence very promising.

When V1 shuts down most if not all clients will create visible errors. Tested by Otis B. for a few older clients. I'm sure those commanders will complain and check the OP (of the tool used) or post a message stating for example EOCR stopped working or stopped sending data to the EDDN. Tool author can point out that they need to update to the latest version.

I only have the EDDN-monitor and can olnly check today and yesterday. I'm sure commanders like you or other receivers with a database can query it to generate nice graphs so we can monitor what percentage of all messages is V1 and what is V2 (converted). Also we can see/monitor how fast/slow commanders start switching to V2 senders. Hence how many V1 messages will be lost and the consequences for receivers like yours, maddavo, RN etc... There will likely be a (hopefully small) dip in data received by receivers when shutting down V1.

TLDR; I'm NOT for postponing the phasing out of V1 messages. Not because commanders are not upgrading. We have 11 days left to inform all commanders.

I hope you understand my POV.

Update: To anyone. If your are a sender of EDDN message or a receiver of them and you were NOT included in the "mailing list" then I apologize for that. In that case send me a PM so I can correct that and can/will be included in the email correspondence.
 
Last edited:

wolverine2710

Tutorial & Guide Writer
wolverine2710
themroc said:
Yes I understand. Let's hope and check stats on the 30th again :)
Click to expand...

I do like to be prepared - as in having data. Perhaps a commander is willing/able to create some .csv files which can be imported into Excel or perhaps someone already has graph generator (like Kibana) running on his/her own system. If I/we could receive that data, that would be cool and very much appreciated.
 

Otis B.

Otis B.
themroc said:
I recommend to analyze http://ross.eddb.io/eddn/log right before the date is due and consider postponing the action if too many v1 messages are still being sent.
Click to expand...
Part of the reason that I support dropping v1 is that most of the publishing apps that are sending v1 are old and don't have support for the E: D 1.3 commodities. Also, according to the stats, a significant number of messages are from people using EliteOCR 0.5, and those people certainly should be encouraged to upgrade to a version of EliteOCR that uses the opencv engine.

I'm assuming that dropping v1 support will cause the vast majority of such people to upgrade, rather than give up.
 
Last edited:
Status
Thread Closed: Not open for further replies.
Back
Top Bottom