Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[Andargor]

Nice

Why the $ in "$schema"?

{'header': {'softwareVersion': 'v3.14', 'softwareName': 'My Awesome Market Uploader', 'uploaderID': 'abcdef0123456789'}, '$schema': 'http://schemas.elite-markets.net/eddn/commodity/1', 'message': {'categoryName': 'Metals', 'buyPrice': 1024, 'timestamp': '2014-11-17T12:34:56+00:00', 'stationStock': 7, 'systemName': 'Eranin', 'stationName': 'Azeban Orbital', 'demand': 42, 'sellPrice': 1138, 'itemName': 'Gold'}}

 

[jamesremuscat]

Why the $ in "$schema"?

JSON Schema itself uses $ to indicate that the "schema" property of a JSON schema is 'special' (http://json-schema.org/latest/json-schema-core.html#anchor22); I changed the property name to "schemaRef" since we're talking instances not schemas here, but the $ is kept because it's still a 'special' property, not part of the data.

 

[wolverine2710]

Since many different formats can be fed to EDDN, can there at least be a "version" or "schema" field which would help apps interpret what they are getting? E.g. { schema: "EliteOCR.3.0.0.6", ... }.

This would give all developers the flexibility they need and keep things modular on both ends.

- - - - - Additional Content Posted / Auto Merge - - - - -

Also, I have posted a REQUEST for player and market data dump from the local game client using the Web API format (or other).

There have been similar threads. Dump JSON/XML info in a file whenever data is received from the FD-servers. Example: opening the market etc. But this one is original. Just use what is already there, iOS web-api instead of reinventing the wheel. It would solve lots of problems. Problem atm that the servers are sized/dimensioned for only iPhone users. As in a relative small group. But they can adjust for that. I just don't think they are going to do it. Best is to write Michael Brookes a PM and suggest it. Now that ED is released they might have some more time for this kind of things.

- - - - - Additional Content Posted / Auto Merge - - - - -

Cool stuff. BTW, I might redistribute the feed using Node.js/Socket.IO to help browser-based, event-driven tools. I might even archive for historical market data. I'll let the project advance a bit before I head into that.

Looks good. EDDN is Alpha and it will take a while before the infrastructure (senders/receivers and their imports into trading tools) is in place. Hopeful for the future!!

 

[Andargor]

Can I also suggest that in the message format there is a "test" flag. I sent a bunch of test POSTs, and realized that I might be polluting someone's data. It would be therefore easy to weed out the tests.

- - - - - Additional Content Posted / Auto Merge - - - - -

And also thinking out loud here, but couldn't we have some sort of signature (à la PGP) to certify that the message comes from known tools? To avoid Hurl.it spam, for example.

 

[NeilF]

Can I also suggest that in the message format there is a "test" flag. I sent a bunch of test POSTs, and realized that I might be polluting someone's data. It would be therefore easy to weed out the tests.

Or have a parallel URL/database with one live and one test? People can then develop/test as much as they like on the test environment?

 

[Andargor]

Or have a parallel URL/database with one live and one test? People can then develop/test as much as they like on the test environment?

Or simply another channel. Currently the subscription channel in ZMQ is "". (blank)

 

[Errantthought]

Have any of the primary market data tool devs committed to support? I was thinking we got Trade Dangerous, might be wrong tho. What about Slopey?

 

[Andargor]

Have any of the primary market data tool devs committed to support? I was thinking we got Trade Dangerous, might be wrong tho. What about Slopey?

Too lazy to dig it up, but I believe all the major ones have (Slopey, Thrudd, Seeebek, etc.). There's really no reason not to, it benefits everyone (as wolfie, no sorry wolvie likes to say )

 

[wolverine2710]

Have any of the primary market data tool devs committed to support? I was thinking we got Trade Dangerous, might be wrong tho. What about Slopey?

As Andargor said I've contacted all trading tools authors I know of (see my 3rd party tools thread in the signature or look in my profile). There are three which are in business if I'm not mistaken. Kfsone (TD) is in. Cmdr Thrudd is in. Slopey lately stated on his OP: "and EDDM updating will be added in due course (nobody seems to be using it at the moment, but Wolvie is excited as usual )". Also Maddavo (merge tool for TD) is in - if needed.

EDDN for now is setup for commodity prices but it can be used to relay all kinds of dynamic data.

EDDN in my/our view is just a basic service where others can built upon. Commanders who have far more time and have brilliant ideas. Store data in a database and make that available to others. For example for users who are not 24/7 connected to the EDDN firehose (data stream) but want to setup their own ELK stack - and need data of the last hour/day/week. See these threads by Andreas which illustrate the use of Kibana (the K in ELK stack). "[Video] Market data visualization with EMDN and ELK", "Economist Newsletter #1: Insight into the current market dynamics" and "Economist Newsletter #2: The Eranin food crisis". Another example is providing TD commanders with a fresh (up to date) .prices file. What maddavo's merge website/tool for TD currently provides. Another example: Provide a web-api to get data. The possibilities are endless. Please do list your ideas for other possibilities here....

James and I like the Unix/Linux philosophy. Highly specialized programs (which do one thing but do it extremely well) which are connected together.The Unix pipe mechanism. Where the output of one command is the input of another program - multiple commands can be strung together (piped) to provide in a one-liner very powerful functionality. Unix commanders know what I mean. EDDN is heavily based on EMDN (by Andreas) which again is heavily based upon EMDR from EVE. Currently its running in Alpha on one system but it can be made as redundant/scalable as needed - see the EMDR link for a high level overview of EMDR. Not that I think we will be running out of our 20Mbps line bandwidth any time soon......
Wrt to the scalability of EMDR - from the EMDR site: EMDR delivers nearly a million messages a day to a number of different projects around the world.

Edit 1: Note to self. Update OP to state that EDDN is in Alpha now.
Edit 2: OP has been changed. Its in the Announcements section at the top of the OP.

 

[jamesremuscat]

OK - as promised, I'll be adding some validation in to the gateway this afternoon. When this happens there are a few things anyone sending data to the network will need to be aware of:

• You MUST include a '$schemaRef' property in the root object of your message.
• The $schemaRef MUST be one of the following:
  • http://schemas.elite-markets.net/eddn/commodity/1 for commodity price data messages
  • http://schemas.elite-markets.net/eddn/commodity/1/test for testing commodity price data messages
  • http://example.com for testing arbitrary JSON - no further validation is applied

Those using the network to receive data SHOULD check the $schemaRef of a message to determine how (or even if) to process it.

The commodity message schema (now, excitingly, at its canonical location) is quite lax at the moment - I will keep the example at https://github.com/jamesremuscat/EDDN/wiki schema-valid as any changes are made. In particular, additional properties within message or header will not cause an error. (Though if there's something you think should be there, it makes sense to have that thing properly defined in a schema somewhere! So let me know and I can add it properly.)

If you throw some JSON at the gateway that it doesn't recognise, it'll return an HTTP 400 code with an error message that could probably be more helpful but should point out what's wrong.

 

[NeilF]

OK - as promised, I'll be adding some validation in to the gateway this afternoon. When this happens there are a few things anyone sending data to the network will need to be aware of:

• You MUST include a '$schemaRef' property in the root object of your message.
• The $schemaRef MUST be one of the following:
  • http://schemas.elite-markets.net/eddn/commodity/1 for commodity price data messages
  • http://schemas.elite-markets.net/eddn/commodity/1/test for testing commodity price data messages
  • http://example.com for testing arbitrary JSON - no further validation is applied

Yeh! Me like clever!

Or have a parallel URL/database with one live and one test? People can then develop/test as much as they like on the test environment?

 

[gazelle]

So the "buyPrice" and "sellPrice" is the reverse of the columns?

• buy column = sellPrice
• sell column = buyPrice

And no field for the demand/supply level?

 

[CMDRKNac]

Ok so...

Don't make systemName required IMO for 'commodity schema', you can let the clients if needed to figure that one if they need to. EliteOCR doesn't include system name as for now and that's where most of the data will come from.

Every field that can be null should be required? We have two ways to deal with it, make them required and each user will POST 'null' for the value when the value doesn't exist (I guess that would pass the validation), or just not make them required at all. Both 'stationStock' and 'demand' can be null. Even in-game 'sell' (which would be 'buyPrice' here) can be null (a station demands a good and offers a price for it -bid in trading terms-, but if it does not have that good in stock it WON'T have a sellPrice -ask in trading terms-.

 

[jamesremuscat]

So the "buyPrice" and "sellPrice" is the reverse of the columns?

• buy column = sellPrice
• sell column = buyPrice

And no field for the demand/supply level?

The buy/sell fields should be the same as the UI labels e.g. sellPrice = sell to market, buyPrice = buy from market. FD's confusing nomenclature here is a pain, but I guess it's more straightforward to stay consistent with it.

I've added optional fields for supply/demand levels (which should be one of Low, Med, High).

Ok so...

Don't make systemName required IMO for 'commodity schema', you can let the clients if needed to figure that one if they need to. EliteOCR doesn't include system name as for now and that's where most of the data will come from.

Every field that can be null should be required? We have two ways to deal with it, make them required and each user will POST 'null' for the value when the value doesn't exist (I guess that would pass the validation), or just not make them required at all. Both 'stationStock' and 'demand' can be null. Even in-game 'sell' (which would be 'buyPrice' here) can be null (a station demands a good and offers a price for it -bid in trading terms-, but if it does not have that good in stock it WON'T have a sellPrice -ask in trading terms-.

I note that EliteOCRReader has started populating the systemName field. I'm not sure that station names are unique galaxy-wide so I get the feeling we need both station and system.

For stationStock, null is zero: "we don't have any in stock". But you're correct that that entails that buyPrice (buy from market) can be null, which implies it should be optional in the schema.

For demand, null I suppose is equivalent to "no demand" e.g. zero.

So, I think I should keep stationStock and demand as required (and zero if the UI reads null), but make buyPrice optional (absent if the UI reads null).

Are there any scenarios where a station will refuse to state a price to buy goods from you (assuming they're listed on the commodity screen)?

 

[donpost]

Ok so...

Don't make systemName required IMO for 'commodity schema', you can let the clients if needed to figure that one if they need to. EliteOCR doesn't include system name as for now and that's where most of the data will come from.

The only problem with that is if there are two stations with the same name but different systems, although I think that's unlikely.
.
In other news... if anyone can tell me why I am not receiving messages using this code I'd be eternally grateful. It definitely connects but gets stuck on the last line waiting for a message when I know for a fact that messages are being sent out.

Dim context As New ZMQ.Context
Dim subscriber = context.Socket(SocketType.SUB)
Dim emptyArray As Byte() = System.Text.Encoding.Default.GetBytes("")
subscriber.SetSockOpt(ZMQ.SocketOpt.SUBSCRIBE, emptyArray)
subscriber.Connect("tcp://eddn-gateway.elite-markets.net:9500")
Dim response = subscriber.Recv()

 

[wolverine2710]

I note that EliteOCRReader has started populating the systemName field. I'm not sure that station names are unique galaxy-wide so I get the feeling we need both station and system.

In the crowd source coordinates project it was found out - and acknowledged by Michael Brookes iirc - that station names or NOT unique galaxy-wide.

 

[CMDRKNac]

The buy/sell fields should be the same as the UI labels e.g. sellPrice = sell to market, buyPrice = buy from market. FD's confusing nomenclature here is a pain, but I guess it's more straightforward to stay consistent with it.

I agree is a pain... honestly I think every app should us e standard trading nomenclature: ASK/BID, the way is done right now is confusing.

I note that EliteOCRReader has started populating the systemName field. I'm not sure that station names are unique galaxy-wide so I get the feeling we need both station and system.

Alright then! I can live with that, the problem it makes direct export from EliteOCR harder unless they include 'system name' in the output (which they should IMO, but is up to them to add it or not, probably is planned).

For stationStock, null is zero: "we don't have any in stock". But you're correct that that entails that buyPrice (buy from market) can be null, which implies it should be optional in the schema.

For demand, null I suppose is equivalent to "no demand" e.g. zero.

Again I'm posting this because how EliteOCR deals with it (which is an extension on how is represented in game). Techncially, zero and null are different things. It's not hard for me to add an other if statement and attach an integer 0 to a value if it does not exist and I guess everybody can do the same, but it reduced 'straightforwardness' from EliteOCR. I guess is a matter of us as a community comming to standard on how to deal with it which is what we are doing now.

So, I think I should keep stationStock and demand as required (and zero if the UI reads null), but make buyPrice optional (absent if the UI reads null).

Are there any scenarios where a station will refuse to state a price to buy goods from you (assuming they're listed on the commodity screen)?

Yes, when you don't have that commodity in your ship storage! Which is most of the time, a price only shows when you have the good in your ship and can sell it, most of the output from OCR scans will have no value in that field.

 

[wolverine2710]

The buy/sell fields should be the same as the UI labels e.g. sellPrice = sell to market, buyPrice = buy from market. FD's confusing nomenclature here is a pain, but I guess it's more straightforward to stay consistent with it.
I've added optional fields for supply/demand levels (which should be one of Low, Med, High).

I note that EliteOCRReader has started populating the systemName field. I'm not sure that station names are unique galaxy-wide so I get the feeling we need both station and system.
For stationStock, null is zero: "we don't have any in stock". But you're correct that that entails that buyPrice (buy from market) can be null, which implies it should be optional in the schema.
For demand, null I suppose is equivalent to "no demand" e.g. zero.
So, I think I should keep stationStock and demand as required (and zero if the UI reads null), but make buyPrice optional (absent if the UI reads null).
Are there any scenarios where a station will refuse to state a price to buy goods from you (assuming they're listed on the commodity screen)?

I'm not 100% sure but can check with TornSoul who created TGC (The great collector) for the crowd source coordinates project. Its web-api is using JSON for uploading info also. I think he HAD to use null in a field instead of leaving out fields. Otherwise TGC borked. That was because of the .Net framework and C# he is using. Are their any C# commanders here who have some insight in this or perhaps can even check/verify this with code?

 

[CMDRKNac]

Dim context As New ZMQ.Context
Dim subscriber = context.Socket(SocketType.SUB)
Dim emptyArray As Byte() = System.Text.Encoding.Default.GetBytes("")
subscriber.SetSockOpt(ZMQ.SocketOpt.SUBSCRIBE, emptyArray)
subscriber.Connect("tcp://eddn-gateway.elite-markets.net:9500")
Dim response = subscriber.Recv()

I don't know how the binding you are using works, but if you are certain that messages are being (are you sure about this 100%? most of the time you would should be receiving nothing), I would look at the encoding line maybe you have to specify 'utf-8', usually those things give trouble when dealing with different types of languages and system platforms. Hope that helps.

 

[jamesremuscat]

Yes, when you don't have that commodity in your ship storage! Which is most of the time, a price only shows when you have the good in your ship and can sell it, most of the output from OCR scans will have no value in that field.

Does the OCR scan only get applied to the panel on the right, then? Because sell/buy is listed for most things in the big table on the left...