Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[wolverine2710]

For now just receiving. As .csv imports are from tools that are able to send data to EDDN usually, I am keeping it to user's decision if they want to share "their" routes or not.

I had then misinterpreted your post it seems, my bad. Indeed most EOCR users will send the data to EDDN. I've changed the entry in ED3PTT (one can still vote for a good name). I've added EDDN to the "used API" section. Quit nice. If you now select EDDN you will see a list of all tools which receive data from it (or send data to it) including yours. The list is growing! Its becoming an eco system on its own.

And YES, Inara looks very nice indeed. If FD only gave us an official ED-API or temp. allow us to use the OCR Error Free tools there would be a BOOM in NEW (because new data) exiting tools. Not to mention FAR FAR more data for the EDDN ;-) EDDN usage by OEF tools is relative small because it doesn't get enough exposure on the FD forums. The redit threads helped a bit it seems....

 

[GelfBride]

Voice controlled EDDN interface

Hi Folks,

I'm writing an app that is driven by voice commands, it going quite well but, i m having trouble tracking down the specs on EDDN interface, i can see a v1 schema, but i see lots of talk about later versions but i can't find any schema for them. Also i would like to hook this app into any other useful services (EDDB, EDCE etc.) so if anyone has the specs on those i'd be grateful. I will also need some help creating different versions of the Grammar and command configs for different langues (i have English and Dutch covered). I will more than likely not have enough time to add all the cool features and fix the bugs so ill likely need some help. If you'd be interested in helping let me know and ill spin up a new thread.

 

[Otis B.]

i'm having trouble tracking down the specs on EDDN interface, i can see a v1 schema, but i see lots of talk about later versions but i can't find any schema for them.

The schemas are at https://github.com/jamesremuscat/EDDN/tree/master/schemas

Or in their canonical locations:
http://schemas.elite-markets.net/eddn/commodity/2
http://schemas.elite-markets.net/eddn/shipyard/1

Note that the draft outfitting schema is not yet in use and is likely to change significantly before being made final.

See http://ross.eddb.io/eddn/log for actual messages in these schemas.

Also i would like to hook this app into any other useful services (EDDB, EDCE etc.) so if anyone has the specs on those i'd be grateful.

EDDB doesn't have an API - see http://eddb.io/api
We're not allowed to talk on these forums about the Companion API that EDMC, EDCE etc use.

 

[wolverine2710]

I apologize upfront for abusing my own thread and going Off Topic.

I have created the thread "POLL. Third party tools thread - upcoming website version - looking for a good name".

The poll is closed. The most votes were for edcodex. The name chosen: EDCodex.

 

[psibernetic]

Hey guys!

I just built a EDDN listener for C# that could help some people get started making things.

https://github.com/psibernetic/EDDN-Listener

 

[wolverine2710]

Hey guys!

I just built a EDDN listener for C# that could help some people get started making things.

https://github.com/psibernetic/EDDN-Listener

NICE. The more tools like this the better. Appreciated. I especially like the fact it has its own repository.

I'm not into .Net and C# so forgive me my ignorance. How do I run your program?
There is a Code file which I assume is the compiled program. In the src directory I see the code.

What exactly does EDDN-Listener show? Currently we have market prices as well as ShipYard info.
For market prices, does it show http://schemas.elite-markets.net/eddn/commodity/2 as well as http://schemas.elite-markets.net/eddn/commodity/2/test (test data)?

I've checked EDCodex (EDC) and found the following entry. Its about EDDN listeners in various languages.
Caveat emperor: They are relative old (PRE Schema V2) so deal ONLY with the now blocked V1 messages. Still could prove useful as well

Spoiler

I got this from the EDDN thread. Its possible I missed one. If you want to develop a tool which is using EDDN data the below should get your started. Note: There are more tools out there using EDDN then code snippets listed below.

Note: There are authors in the thread who said they would be willing to help commanders with code and/or examples on request. So if the below doesn't cover what you need please post a request for it in the EDDN thread. You probably will be surprised....

Tool: (intelligent) EDDN subscriber in python.
Author: kfsone
Annoucement: here.

Tool: EDDN subscriber/listener written in C# (.Net)
Author: maxh2003
Announcement: here

Tool: EDDN subscriber/listener written Python (version 2.x or 3.x is unknown).
Author: K Kinnison
Announcement: here

Tool: (minimal) Node.js listener for EDDN
Author: Andargor aka little red devil...
Announcement: here

 

[psibernetic]

Eddn.Listener is a development library that makes it incredibly simple to access the EDDN relay with on line of code.

There is no end-user application, but I will add an example project that uses it mimicking the EDDN Python Simple Client.

It is a class library that reads from the EDDN relay and uncompresses the messages and then passes that message back to a callback function the developer provides. As such it supports anything EDDN pumps out including commodities and shipyard info as well as any future support.

CMDR RealityEnigma and I are working on more tools and analysis engines, Eddn.Listener is our way of accessing EDDN, so we thought we should share that work.

If .NET developers want to get involved let us know.

 

[wolverine2710]

Eddn.Listener is a development library that makes it incredibly simple to access the EDDN relay with on line of code.

There is no end-user application, but I will add an example project that uses it mimicking the EDDN Python Simple Client.

It is a class library that reads from the EDDN relay and uncompresses the messages and then passes that message back to a callback function the developer provides. As such it supports anything EDDN pumps out including commodities and shipyard info as well as any future support.

CMDR RealityEnigma and I are working on more tools and analysis engines, Eddn.Listener is our way of accessing EDDN, so we thought we should share that work.

If .NET developers want to get involved let us know.

Now I understand. I totally like these kind of libraries. An example would be very nice indeed ;-)
To quote you "CMDR RealityEnigma and I are working on more tools and analysis engines".
Very very nice. If not a secret what are you exactly planning?

 

[Roguey]

hi there guys, I am still having a few problems - with the ZMQ side. It seems after awhile my code disconnects or starts missing messages from EDDN. I wonder if ive set something wrong;

// --------------------------------------
// ZMQ connection
// --------------------------------------
	  
$context 	= new ZMQContext();
$subscriber = $context->getSocket(ZMQ::SOCKET_SUB);

// Connect to the first publicly available relay.
$subscriber->connect("tcp://eddn-gateway.elite-markets.net:9500");

// Disable filtering.
$subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, "");

print "\n";

while (true)
{

    // Receive raw market JSON strings.
   $market_json 		= gzuncompress($subscriber->recv());

	// Un-serialize the JSON data to a named array.
    $json_data			= json_decode($market_json, true);

Also should we use gateway or relay? I wasnt too sure what the difference was. many thanks,

ps. will equipment be added to EDDN? I have support for ships already.

 

[Artie]

Set receive timeout:
$subscriber->setSockOpt(ZMQ:: SOCKOPT_RCVTIMEO, $timeout);

Set php script time limit (PHP can have some issues with it when script run from web):
set_time_limit(0);

...and setSockOpts before connect. It should solve your problems.

 

[Roguey]

Set receive timeout:
$subscriber->setSockOpt(ZMQ:: SOCKOPT_RCVTIMEO, $timeout);

Set php script time limit (PHP can have some issues with it when script run from web):
set_time_limit(0);

...and setSockOpts before connect. It should solve your problems.

Thanks for the help. Ill give that ago and hopefully thats it (I added SOCKOPT_RCVTIMEO and moved the connect afterwards). I already had the php script time set to zero.

ill update this thread if its working later on

 

[Roguey]

well sorry to say that its still doing the same thing. I left it running for an hour and a bit, and checking the logs its not posted for the past hour. This is what I have at the moment;

// --------------------------------------
// ZMQ connection
// --------------------------------------
	  
$context 	= new ZMQContext();
$subscriber = $context->getSocket(ZMQ::SOCKET_SUB);

// Disable filtering.
$subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, "");

// set timeout
$subscriber->setSockOpt(ZMQ:: SOCKOPT_RCVTIMEO, -1);

// Connect to the first publicly available relay.
$subscriber->connect("tcp://eddn-gateway.elite-markets.net:9500");


print "\n";

while (true)
{

    // Receive raw market JSON strings.
   $market_json 		= gzuncompress($subscriber->recv());

	// Un-serialize the JSON data to a named array.
    $json_data			= json_decode($market_json, true);

Works for a bit, then stops posting. I can get it to resume if I close it, then restart the code. Im wondering if its something to do with the PHP.

 

[Artie]

Weird... you can try to set timeout to some fixed value like 15 minutes, check if FALSE is returned on recv() and in that case restart a connection. I think there is nice example in PHP with similar solution in EDDN archive. It's rather workaround of your problem than solution, but at least something.
It doesn't seem to be primarily PHP related, I had no such problems with it.

 

[Otis B.]

ps. will equipment be added to EDDN? I have support for ships already.

There's a month-old pull request for the outftting schema here: https://github.com/jamesremuscat/EDDN/pull/22
and a sample message here: https://gist.github.com/Marginal/d4fab8b73a828c33e459

 

[Roguey]

Weird... you can try to set timeout to some fixed value like 15 minutes, check if FALSE is returned on recv() and in that case restart a connection.

Hi there, I tried what you said and still the same. After an hour and half, my PHP stopped listening. I wrote some code to tell me if recv() returned with false, and nothing came up.

I did go through php.ini to see if there was any limits, and set them to either 0, or -1.

Although if I do get my listener working, its not ideal as I need to have my pc on to listen for EDDN information (my PC is on only a certain amount of hours a day). I cant run ZMQ on the server, as its not mine. So I need something to listen to EDDN traffic then POST the data to my site, which in-return updates the database. Although I would be happy if the listener wouldn't stop after a certain amount of time, and just freeze up. Although if it works for a few hours, then it helps a lot.

There's a month-old pull request for the outftting schema here: https://github.com/jamesremuscat/EDDN/pull/22
and a sample message here: https://gist.github.com/Marginal/d4fab8b73a828c33e459

Ah, that would be good if its put into use. As I can receive that information.

 

[GelfBride]

Hey guys!

I just built a EDDN listener for C# that could help some people get started making things.

https://github.com/psibernetic/EDDN-Listener

Also c# wanna team up?

 

[Snuble]

Still noone offer dump of data received "from date time" "to date time"?

What about station data for systems, is there anyone offering that data parseable same as Redwizzard do for systems?

 

[wolverine2710]

Still noone offer dump of data received "from date time" "to date time"?

In this thread commanders mentioned that they have created archiving services for EDDN. The data is accessible with an API. I think I mentioned them a few times here but never put them in the OP, my bad. Below is a list of archiving services that I'm aware of. Not sure if they have a "from date time" "to date time" function in their API's.

I will update the OP and link to this post.

Archiving services - Using the export function of EDCodex.

Spoiler

Elite History Server
-----------------------------------------------------------
Categories : EDDN
authors : Pandemic
Status : Active
Compatible with release : Active
Sources available : Active
Supported languages : English

Elite History Server.

For those looking for historical EDDN data, I have built a cache system and a new protocol to deal with the problem of sending commodities one at a time. This system is based on protocol buffer, which in testing reduced the bandwidth taken by the messages from about 240 bytes zlib compressed JSON to around 95 bytes uncompressed protocol buffers. In addition, this system supports batching. When you need data, you call the endpoint with the last Unix timestamp of data you receive and it will send you everything since then. The code for the server is located at https://github.com/huadianz/elite-eddn-history and includes the protocol buffer schemas you need to receive the data. In addition, it also contains an example program that shows you how to unpack the protocol buffer binary stream.

The current server I am running is located at http://elite.servers.henryzh.com/api/v0/update/
The API will send you up to the last 1000 missed updates with Unix timestamps in MS >= the timestamp in the system. If you need more you will need to query the server again.

As this project is open source I welcome any suggestions or if you have any code you want to commit, that would be great. As for already planned features:


Update the network using batched commodity updates
Current status server (query the status of a station in its most recent state)
Timeseries server with influxdb
Using commodity IDs instead of names, if we get those from Frontier at any point


Links :
Source - https://github.com/huadianz/elite-eddn-history

---

Archive of EDDN messages.
-----------------------------------------------------------
Categories : EDDN
authors : Askarr
Status : Active
Compatible with release : Active
Supported languages : English

Archive of messages broadcast on the firehose of EDDN.

Type of Value added service (VAS) for EDDN: Archiving.

Announcement here (https://forums.frontier.co.uk/showthread.php?t=57986&p=1500490&viewfull=1#post1500490) , with code sample of how to use it. More info about usage here (https://forums.frontier.co.uk/showt....php?t=57986&p=1560417&viewfull=1#post1560417.
Note: You must first ask for an key before you can use it.

Comment: No official name for the tool has been given by Askarr.


Links :
Thread - https://forums.frontier.co.uk/showthread.php?t=57986&p=1500490&viewfull=1#post1500490

---

Database store for EDDN
-----------------------------------------------------------
Categories : EDDN
authors : thardie
Status : Active
Compatible with release : Active
Supported languages : English

Database store for EDDN. Suitable for Trade Dangerous.

In an attempt to bridge the gap between where EDDN is now, and systems like trade dangerous, I've implemented my first go at storing and publishing data from EDDN. If it gets enough interest, I'll add more features. Let's see what response this gets.

You can get a trade dangerous-ish dump of my database from - It's real-time (IE, calling this URL reads the data from the database, not a pregenerated file). The database is updating from EDDN in real time.

http://eddndb.orcas.net/cgi-bin/tdexport.prices

Let me know what you think.

Links :
Other - http://eddndb.orcas.net/cgi-bin/tdexport.prices

Update: Added "Archiving services for EDDN - with an API" section with spoiler tags to the OP.

 

[wolverine2710]

What about station data for systems, is there anyone offering that data parseable same as Redwizzard do for systems?

Not totally sure what kind of data you are looking for but I think the following two services are perhaps providing what you want/need.

Maddavo's Market Share (MMS). Its used by Trade Dangerous but Maddavo also has a website where you can download the files from. Station.csv is probably what you are looking for. Part of the contents in the spoiler tag. It currently has 40874 stations in it. Maddavo also uses the ShipYard information send by EDDN. That data is coming from ED Market Connector. Its a so called OCR Error Free aka mobile api base tool which uses the web-api which powers the offical ED iPhone app. For more info about these tools see the OP of my third party tools thread.

Spoiler

unq:name@System.system_id,unq:name,ls_from_star,blackmarket,max_pad_size,market,shipyard,modified,outfitting,rearm,refuel,repair
'1 G. CAELI','Smoot Gateway',4761,'N','L','Y','Y','2015-05-06 17:03:00','Y','Y','Y','Y'
'1 GEMINORUM','Collins Settlement',2598,'Y','M','N','N','2015-05-25 15:10:48','Y','Y','Y','N'
'1 GEMINORUM','Zholobov Orbital',3430,'Y','M','N','N','2015-06-21 23:25:43','N','Y','Y','Y'
'1 HYDRAE','Hieb Orbital',160,'N','M','N','N','2015-05-25 15:10:48','N','Y','Y','Y'
'1 HYDRAE','Afanasyev Port',200,'N','M','N','N','2015-05-25 15:10:49','N','Y','Y','Y'
'1 HYDRAE','Hutton Port',279,'N','L','Y','Y','2015-06-30 07:34:00','Y','Y','Y','Y'
'1 HYDRAE','Yurchikhin Port',369,'N','M','N','N','2015-06-21 23:11:54','N','N','Y','Y'
'1 HYDRAE','Arber Terminal',664,'N','M','N','N','2015-06-21 23:11:54','N','N','Y','Y'
'1 HYDRAE','Feustel Port',664,'N','L','Y','Y','2015-06-30 07:46:00','Y','Y','Y','Y'
'1 HYDRAE','Voss Hub',823,'Y','L','N','Y','2015-05-25 15:10:49','Y','Y','Y','Y'
'1 HYDRAE','Hornby Dock',1457,'N','L','N','Y','2015-05-25 15:10:49','Y','Y','Y','Y'
'1 HYDRAE','Whitney Station',2106,'N','M','N','N','2015-06-21 23:11:54','N','N','Y','Y'
'1 I CENTAURI','Armstrong Ring',1093,'Y','L','Y','Y','2015-06-21 23:04:19','Y','Y','Y','Y'
'1 I CENTAURI','Ampere Dock',1498,'Y','L','Y','Y','2015-05-01 04:33:33','Y','Y','Y','Y'
'1 KAPPA CYGNI','Kinsey Ring',2359,'N','M','Y','N','2015-06-21 23:31:26','N','N','Y','Y'
'1 KAPPA CYGNI','Wohler Port',3520,'Y','L','Y','Y','2015-06-21 23:31:26','Y','Y','Y','Y'
'1 KAPPA CYGNI','Hauck Enterprise',4832,'Y','M','N','N','2015-06-21 23:31:26','N','Y','Y','Y'
'10 ARIETIS','Archimedes Base',0,'N','?','Y','N','2015-05-13 10:27:32','N','N','Y','Y'
'10 ARIETIS','Chadwick Station',0,'N','?','Y','N','2015-05-13 11:56:42','Y','N','Y','Y'

Note: There are stations/outpost for which no one so far has filled in all the information -> market by a ? character

The API page of eddb.io. Systems.json has content like, see spoiler tag.

Spoiler

{
"id": 1,
"name": "1 G. Caeli",
"x": 80.90625,
"y": -83.53125,
"z": -30.8125,
"faction": "Empire League",
"population": 6544826,
"government": "Patronage",
"allegiance": "Empire",
"state": "None",
"security": "Medium",
"primary_economy": "Industrial",
"needs_permit": 0,
"updated_at": 1430931668,
"power_control_faction": null
},
{
"id": 2,
"name": "1 Geminorum",
"x": 19.78125,
"y": 3.5625,
"z": -153.8125,
"faction": "1 Geminorum Liberals",
"population": 141727,
"government": "Democracy",
"allegiance": "Federation",
"state": "None",
"security": "Medium",
"primary_economy": "Terraforming",
"needs_permit": 0,
"updated_at": 1430931879,
"power_control_faction": null
},

I hope this info was useful.

 

[Artie]

Hi there, I tried what you said and still the same. After an hour and half, my PHP stopped listening. I wrote some code to tell me if recv() returned with false, and nothing came up.

...

Then there comes two options to my mind:
1. if it really hangs on recv(), it's ZMQ related issue and you can try another/newer version. When fixed timeout set, recv() should return FALSE even when network issues, etc.
2. Your script hangs/break somewhere else and it slipped out of your attention. I can only recommend to track down problematic place using log file(s). So dump to file what exactly is returned from recv(), what exactly is returned from uncompress, etc. Eventually, you can also set more "sensitive" error logging in PHP and check your error logs (you should also notice script timouts there).