Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

Status
Thread Closed: Not open for further replies.

MagmaiKH

MagmaiKH
CMDRKNac said:
Eeeer not is not, I wasn't talking about system coordinates. I was asking about the STATION NAMES pertaining to confirmed systems. Anyway, got a system more or less working already requiring the input of the same station name from different users several times before it gets added to the db.

Did some clean up of erroneous station names that were being added blindly. Hopefully we can integrate it with EliteOCR in a few days so data will come a bit cleaner, but more cleaning up is probably necessary (have around 2400 station names compiled so far).

BTW if anyone is wondering there are tons of duplicate station names, so definitively they are not unique. (to make things worse.)
Click to expand...


My bad, sorry.
How do you envision this working with EliteOCR?
It seems like EliteOCR (or alt) always has to send the data and the receivers have to implement the data fusion algorithms.
And more to the point, there's already bad OCR tools spewing bad data so the receivers already have to deal with that.
 

MagmaiKH

MagmaiKH
jamesremuscat said:
Created https://github.com/jamesremuscat/EDDN/issues/2 to track this.
Click to expand...

I'd like to propose we start working on the next generation of messages and those messages should have strict schema.
(I know XML & XML schemas well but not JSON unfortunately.)

I would like to see a single message for an entire station. This is how we would know if a commodity disappeared from the station.
This is most important for data-healing. Right now if someone scans a station and it charfs and generates, say, "Bio-f3ul" there's no way for the system to remove it.
If a single message was sent for the whole station then we could take that as a complete list of commodities and expire the old ones.

This would also pave-the-way to start adding station meta-data; it is particularly important to know if a station has a large landing pad or not.
It would be nice to know the Ls from the star and the amenities available, black-market, et. al. Perhaps factions.
Data could be intentionally set/left as null to indicate the information is not known or provided (then the receiving tools would know it's not updated and could fuse with prior history).

It would also be nice to have a System message with star, planet, moon, & asteroid data.
I'd like to know the star type (to determine if it's scoopable, if it's hard to scoop/dwarf).
I'd like to catalog/search for pristine metallic reserves.

Thoughts?

Astro Analytics
 

Andargor

Andargor
MagmaiKH said:
I'd like to propose we start working on the next generation of messages and those messages should have strict schema.
(I know XML & XML schemas well but not JSON unfortunately.)
Click to expand...

I disagree. EDDN should be minimal and general purpose since it will become the common target for all data producers (EliteOCR, star charting, WhatsThatStationName, MyLittlePonyInSpaceHeatMap, etc.), and the common source for all consumers (BPC, TD, ELK, etc.), including any intermediate transformations which shall also be sources (caching/buffering, content validation, station names/commodity names, localization, etc.). Any changes to the schema will have immediate impacts both upstream and downstream.

Hence, if you wish to provide "clean" data, tap into EDDN, perform your transformations and provide it as a source to consumers using a strict schema. All the while you can rest assured that you will be fed by EDDN using a stable schema.

Yes, this means there will be many different transformations and sources that will pop up, but this ensures the greatest speed to usability and all different concerns can be expressed in parallel by different tools. The most useful will survive, and EDDN will march on.
 

Andargor

Andargor
Thinking ahead about malicious use of EDDN (bad data intentionally fed), perhaps JWS (JSON Web Signature) can be considered? It's a draft IETF recommendation. Clients sign a message (an optional object field can be added to the schema). Downstream consumers may decide to use it and filter out messages without proper signatures, or use the stream as-is.

There's one implementation that I could find in javascript.

Again, it would be optional.

EDIT: Here's a python implementation.
 
Last edited:

wolverine2710

Tutorial & Guide Writer
wolverine2710
Andargor said:
Thinking ahead about malicious use of EDDN (bad data intentionally fed), perhaps JWS (JSON Web Signature) can be considered? It's a draft IETF recommendation. Clients sign a message (an optional object field can be added to the schema). Downstream consumers may decide to use it and filter out messages without proper signatures, or use the stream as-is.

There's one implementation that I could find in javascript.

Again, it would be optional.

EDIT: Here's a python implementation.
Click to expand...

Thanks for your continued feedback and suggestions. I have to read something about it to see if its usable for clients like TD - where everybody runs its locally.
Currently having serious RL issues but hopefully they will be gone in 2+ days. I hoped they would be gone by now ;-( Here is fingers crossed.

If there are TD users amongst us with Python knowledge. You might want to consider creating a program or plugin for Trade Dangerous which uploads the .prices file to EDDN. Of course only relevant (not obsolete prices) ones. In the misc folder of TD there is an experimental prices exporter which transforms a .prices file into a JSON structure. We NEED to keep feeding EDDN.
 
Last edited:

Snake Man

S
Err don't you mean the other way around? :S

There are huge number of feeding posts into EDDN through EliteOCR and RegulatedNoise (and that one other place with long name sorry I forgot specifics), yet none of the actual trading tools are using the feed. Now don't get me wrong, I love EDDN, I love watching the almost constant stream of data coming through, but... what I'd love more than watching it, would be using it.

I'm coding my own little tool for my own little use, but what we really, desperately, need is these released trading tools to start use EDDN :(
 

themroc

themroc
Snake Man said:
yet none of the actual trading tools are using the feed. Now don't get me wrong, I love EDDN, I love watching the almost constant stream of data coming through, but... what I'd love more than watching it, would be using it.
Click to expand...

For two weeks I'm working on a trading website which is using EDDN. I'm still on data correction and general finishing. I guess within the next two days a public release is very likely.
 

Andargor

Andargor
Snake Man said:
Err don't you mean the other way around? :S

There are huge number of feeding posts into EDDN through EliteOCR and RegulatedNoise (and that one other place with long name sorry I forgot specifics), yet none of the actual trading tools are using the feed. Now don't get me wrong, I love EDDN, I love watching the almost constant stream of data coming through, but... what I'd love more than watching it, would be using it.

I'm coding my own little tool for my own little use, but what we really, desperately, need is these released trading tools to start use EDDN :(
Click to expand...

It does take a little while for developers to start using the stream. Also, as I believe the case is with BPC, there is some concern with the quality of the feed (i.e. OCR reliability, mangled station and commodity names, and such), so it doesn't use it, yet. When more pieces will be in place to take the feed and clean it up, we should see a surge in tools.

Currently, I believe the most important piece will be for semi-static info, which is system, station and commodity name cleanup and validation. I'm not sure who is working on taking the EDDN feed and performing validation, and then offering that as a source.
 
Last edited:

themroc

themroc
Andargor said:
I'm not sure who is working on taking the EDDN feed and performing validation, and then offering that as a source.
Click to expand...

In fact, I am. A REST api is in place already and will soon be available for the public. I am cleaning the data meticulously and this is still taking some time. Stay tuned!
 

Askarr

A
I've set up a system whereby all EDDN commodity traffic that passes the primary schema is being stored in a DynamoDB database in AWS (effectively, Cassandra NoSQL). If this is of interest to others for historical queries or recovering lost data, I'm happy to provide read access to other users.
 

Walter Harriman

Walter Harriman
Askarr said:
I've set up a system whereby all EDDN commodity traffic that passes the primary schema is being stored in a DynamoDB database in AWS (effectively, Cassandra NoSQL). If this is of interest to others for historical queries or recovering lost data, I'm happy to provide read access to other users.
Click to expand...
I'm very much interested in such a thing for a personal project. I've been reading the stream myself and dumping station and system data to a mysql database but I've been unable to make the python script work reliably (I don't actually know python, which is a problem :))
 

Askarr

A
Walter Harriman said:
I'm very much interested in such a thing for a personal project. I've been reading the stream myself and dumping station and system data to a mysql database but I've been unable to make the python script work reliably (I don't actually know python, which is a problem :))
Click to expand...
Sure. This is purely the recorded incoming JSON as Dynamo attributes. There's no validation or parsing beyond ensuring the JSON schema validates. A clean stream is a later project, or others may already be providing it by then :)

For simplicity, and since I've not had a chance to wrap a REST API around the storage as yet, any user can request to query the NoSQL DB direct via AWS SDKs. I'd strongly encourage people to use the high level SDKs such as Python, because they'll handle a host of issues, such as retrying queries, backing off if throttled, all that good stuff.

Obviously let me know if data seems missing; this is relatively new code.

Example Python AWS SDK query code:
Code: 
import boto.dynamodb2
from boto.dynamodb2.table import Table

# I'll give out read-only access keys on request, but please don't share them; that way I can track who is using what data
# and block individual access if someone abuses the system.
# Note boto doesn't need this; you can put the keys as files on your system. Just provided here for completeness.
awsKey = "YOURKEYGOESHERE"
awsSecret = "YOURSECRETGOESHERE"
    
raw_data_table = Table('eddn-raw-commodity-json', connection = boto.dynamodb2.connect_to_region('eu-west-1', aws_access_key_id=awsKey, aws_secret_access_key=awsSecret))

# Return data for day 2015-01-10 between epoch times in microseconds. Compared to gatewayTimestamp, each epoch time numeral would be:
# (gatewayDateTime - 1970,1,1).total_seconds() * 1000000
# IMPORTANT: Please query only data you need, and store that data rather than re-querying if at all possible, so as to be fair on other users.
#
# This is only one way to query the data. This assumes you want to get a day's worth or part of a day, partitioned by timestamps.

rows = raw_data_table.query_2(gatewayDay__eq="2015-01-10",
                                               gatewayEpochMicro__between=[1420848000000000, 1423526400000000],
                                               consistent=False)
for row in rows:
    print row["json"]
    # etc. or do more interesting things with row than straight access
 
Last edited:

wolverine2710

Tutorial & Guide Writer
wolverine2710
AWESOME commanders. Nice to see that commanders ARE taking the data, do meaningful stuff with it it AND giving it back to the community ;-) RL issues here (still). In the next days I will go through this thread and collect all tools/code and place them in an EDDN tools section on the OP. Python is NOT a requirement for ØMQ (zeroMQ). It has support for more then over 40 languages.

What would be very cool if we could have a collection of very simple ØMQ firehose subscribers (like src/eddn/client.py in the EDDN code) which could be used by potential new tool authors as a starting point so they can use there own preferred language. It would lower the threshold for them. Perhaps some of the existing tool authors who already use the EDDN stream in their code could create and publish such a simple firehose subscriber program. In the end we will have a collection of ready to use quality subscribers for most common (or uncommon) languages. Imho I think there is a need for them ;-)

Edit: @Askarr. COOL tool.
 
Last edited:

themroc

themroc
Today I'm making my EDDN-driven site online:

eddb - Elite Dangerous Database

It's by far not finished and will get a bunch of new features within the next weeks. However, the actual state makes it usefull already I believe.

Initially I wanted to post a new thread in this Guides and Tutorials section of the forum, but it's for approved topic submitters only. Anyone can tell me whom I have to talk to in order to get the approvement?
 

Andargor

Andargor
themroc said:
Today I'm making my EDDN-driven site online:

eddb - Elite Dangerous Database

It's by far not finished and will get a bunch of new features within the next weeks. However, the actual state makes it usefull already I believe.

Initially I wanted to post a new thread in this Guides and Tutorials section of the forum, but it's for approved topic submitters only. Anyone can tell me whom I have to talk to in order to get the approvement?
Click to expand...

Most excellent!
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
themroc said:
Today I'm making my EDDN-driven site online:

eddb - Elite Dangerous Database

It's by far not finished and will get a bunch of new features within the next weeks. However, the actual state makes it usefull already I believe.

Initially I wanted to post a new thread in this Guides and Tutorials section of the forum, but it's for approved topic submitters only. Anyone can tell me whom I have to talk to in order to get the approvement?
Click to expand...

Sending a PM to a moderator should do to the trick. In the past I've PM-ed with Bret C. which turned out fine.

May I ask why you want to create a thread in the "Guides and Tutorials" sub forum? Imho the "Dangerous Discussion" or "Gameplay and Features" are more appropriate/useful. Especially the first one will get you more exposure. Right, so why is this thread in the "Guides and Tutorials" you might ask? When ED was released all Gamma forums were archived (made read-only) and when I ask to move this thread, it was moved to this sub forum by a moderator. Imho not the best place but I got some a few extra perks (see avatar) which work here so I'm quite ;-) Also commanders seem to be able to find this thread (its also in my signature).
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
themroc said:
Today I'm making my EDDN-driven site online:

eddb - Elite Dangerous Database

It's by far not finished and will get a bunch of new features within the next weeks. However, the actual state makes it usefull already I believe.

Initially I wanted to post a new thread in this Guides and Tutorials section of the forum, but it's for approved topic submitters only. Anyone can tell me whom I have to talk to in order to get the approvement?
Click to expand...

Superb work. Thanks very much for your effort and using the EDDN data ;-) I've been so frank to put eddb in the top of the OP of my 3rd party tools thread (also here in this sub forum) in the announcements section. I've also put it in the TODO section, will be moved to the correct section later.....

I've noticed you use TGC (The Great Collector) aka edsc api for coordinates (and stations which are also in there to some extend). TGC's goal was to store all kind of static data in it but we haven't heart of author TornSoul since the 16th of December 2014. So not sure what the future of TGC is. An API to store and retrieve stations/platform info would be great - ideally with distance from the star. Perhaps this is of interest to you. Because there is no TGC with static data in all kind of tools data is stored (segmentation). You might want to have a look at Trade Dangerous. An (open source) command line tool written in Python which basically is an engine to be used by others - in a GUI or website. It does all the heavy lifting like solving the NP hard problem. It has MANY feature and is FAST. You could use that as a base for eddb if you deem that useful. More importantly right now, quite a few commanders update TD with station data and systems (with an economy iirc but not 100% sure). In TD's system.csv there are 21390 systems and the stations.csv holds 3947 stations. Your systems page states "Total systems known: 19892. Systems with stations: 2305" so the TD info might be useful to you ;-) Keep up the good work...

Edit: In what language is your tool/API written?
 
Last edited:
Status
Thread Closed: Not open for further replies.
Back
Top Bottom