Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[AnthorNet]

Was buried in ED3PTT work. Didn't have time check it out, my bad.

In the past with V1 aside from the fact that each and every commodity was a single message (for backwards compatibility) there were quite a few discussions about things which were missing or could be done better. NOW is your change to requirement known again.

I do have a proposal for a requirement. Atm there are relative few messages send to EDDN and published by it. When the ED API's arrives this hopefully will explode. Currently the free hosting provided by Vivio Technologies gives us a 20 Mbps unmetered internet connection. I believe this will change in the future and we will get more bandwidth.

Suppose the ED API is there and EDDN will be flooded with messages and a lot of them could/will be the same. As in each time a commander opens the commodities market the prices for that station/outpost are uploaded - ofc the commander has to be running a tool which uploads to EDDN. If more commanders arrive at that station more or less at the same time the prices will be the same data will be transmitted. Supppose the current 20Mbps is full but EDDN can handle it. That is NOT to say that commanders who want to connect to EDDN for their public/private tools can handle it. Not everyone has a lot of bandwidth.

Like I said a lot of data could be the same, it will be received by EDDN BUT it doesn't mean EDDN has to publish all those duplicates. Suppose a hash is created (there are some extremely fast ones out there) for all the commodities and and their prices. When the message is received by EDDN is creates a hash also. If the hashes are the same it should not be send out. This will make it possible that commanders with less bandwidth can still connect to EDDN and it also means saving bandwidth.

Ofc it means extra work for EDDN, coding wise and CPU wise. The aim of EDDN is to keep things simple. Do you see merits in my suggestion? Please us all know what you think....

There is a dupe message function that is currently commented in the code but could be update to suits our needs.
We'll have to investigate that, but it's mainly a hash cache, to see if a message has been sent for the last 5 minutes or so.

 

[wolverine2710]

There is a dupe message function that is currently commented in the code but could be update to suits our needs.
We'll have to investigate that, but it's mainly a hash cache, to see if a message has been sent for the last 5 minutes or so.

I stand corrected. Thanks for the PM explaining things AND of course updating this thread.

 

[wolverine2710]

Should I have time this evening (not that likely) I will try to go through the whole thread and check what requests have been made for a V2 and if those are in the current V2 or not. Not that they should be in your V2 but its good to be aware of all the things which were mentioned in the past. Its always good to know if things from the past are still valid or that time has proven they are for example NOT needed anyway.

EDIT: Should someone with time want to do this. FEEL FREE as it would be much appreciated and it would save me lots of time...

 

[jamesremuscat]

EDDN is currently down; our hosting provider seems to be suffering a DDoS attack. No ETA at the moment, and nothing I can do from here. Sorry for the inconvenience!

 

[wolverine2710]

EDDN is currently down; our hosting provider seems to be suffering a DDoS attack. No ETA at the moment, and nothing I can do from here. Sorry for the inconvenience!

Woke up and checked the status page. In the last hour the gateway and relay processed 1185 messages. Also used another tool to check for incoming messages. Seems to be working fine again and it looks as if they handled the DDoS attack.

 

[seeebek]

EDDN is currently down; our hosting provider seems to be suffering a DDoS attack. No ETA at the moment, and nothing I can do from here. Sorry for the inconvenience!

I really hope EliteOCR didn't cause a situation similar to DDoS. I mean one user sends on average 50 items at a time. If there were very many users sending their data EDDN could get maybe up to 1000 parallel requests...

Edit: Do you have any statistics how many messages EDDN received over time (maximum messages received at a time)

 

[AnthorNet]

I really hope EliteOCR didn't cause a situation similar to DDoS. I mean one user sends on average 50 items at a time. If there were very many users sending their data EDDN could get maybe up to 1000 parallel requests...

Edit: Do you have any statistics how many messages EDDN received over time (maximum messages received at a time)

Other than stats in the status page, we do not have any maximum messages received at a time.

 

[Andargor]

What is the process for getting schemas approved and implemented? Other than the new one I proposed, there is the "commodity as an array" one which I believe is being worked on, and which would alleviate the issues seeebek is talking about. I can draft that if needed, tell me how to help.

How can new schemas get up and running?

 

[AnthorNet]

For getting a new schema approved and implemented, they have to be dynamic and useful.
The commander infos are dynamic true, but are they useful to anyone yet ?

Furthermore, I don't think personal infos should pass to EDDN ?

James or Wolverine may have a better response than mine as I'm not responsible for that ^^

 

[Snake Man]

If EDDN has two 'relays' or something that one link/port will feed us the normal market data and the another, totally separate feeds whatever lore user combat rankings/explorations... then I'm totally fine it with (I don't care what the other link feeds).

However it should be obvious that the existing EDDN commodity market data feed should not be polluted with anything else.

 

[AnthorNet]

If EDDN has two 'relays' or something that one link/port will feed us the normal market data and the another, totally separate feeds whatever lore user combat rankings/explorations... then I'm totally fine it with (I don't care what the other link feeds).

However it should be obvious that the existing EDDN commodity market data feed should not be polluted with anything else.

Well the second D from EDDN doesn't exactly mean commodities
And that why you have a $schemaref so everyone can grab what he needs and only the feeds they want to fetch and use.

In the future, not so far (eg: PowerPlay) new datas will probably have to be updated through EDDN.

 

[Andargor]

EDCE has just been released. Please keep an eye out on upcoming EDDN traffic, hopefully it will not explode.

 

[wolverine2710]

EDCE has just been released. Please keep an eye out on upcoming EDDN traffic, hopefully it will not explode.

Headsup (url). Relevant for EDDN listerners/subscribers. See also the posts following it.

Update: I´ve removed the url(s) to be in compliance with the FD forum rules. Apply google-fu.
For more information about and an overview of OCR error Free (OEF) aka mobile api based tools see this FD thread.
Relevant thread: Any news on the official ED API ?

 

[seeebek]

Headsup. Relevant for EDDN listerners/subscribers. See also the posts following it.

The thread disappeared. Probably after people from Slopey's thread reported it.

 

[Andargor]

The EDCE thread has been closed by the moderators, pending Michael's decision. The thread has not been deleted, I will keep you posted.

EDIT: I'm pleading with Michael to tolerate it for a while.

 

[Askarr]

I wanted to point out something about the potential duplicates filtering idea. It's actually quite useful when tracking the data to see duplicates; multiple reports of the same data from different sources actually increases confidence that the data is correct. Even with a full FD API, we can't rule out data tampering.

I don't disagree that having duplicates filtered out is useful for certain use cases and lower bandwidth - it absolutely is. I just think EDDN would be better retaining its original firehose intent. Why not have additional streams for different types or quality of data, including a firehose version of EDDN (current today) and a filtered version of EDDN (filters duplicates, perhaps cleans up garbage data).

I would envision something like this:

• Data sources send data to EDDN firehose
• EDDN firehose is connected to by large bandwidth cleanup sources and archives
• These re-publish to a clean stream that filters duplicates
• Low bandwidth users connect to that stream, knowing they're getting cleaner more concise data but potentially lacking some nuances

Cleanup code could even append extra data such as 'I've seen X messages like this in the past 24 hours' e.g. counting the hashes.

I'd also caution over how filtering is applied depending on level. There's a huge difference semantically between filtering identical messages out (this exact message has been sent multiple times) and filtering identical data out (regardless of source, current price of Station X Commodity Y is Z)

Just a random early-Sunday-morning thought.

 

[Andargor]

I wanted to point out something about the potential duplicates filtering idea. It's actually quite useful when tracking the data to see duplicates; multiple reports of the same data from different sources actually increases confidence that the data is correct. Even with a full FD API, we can't rule out data tampering.

I don't disagree that having duplicates filtered out is useful for certain use cases and lower bandwidth - it absolutely is. I just think EDDN would be better retaining its original firehose intent. Why not have additional streams for different types or quality of data, including a firehose version of EDDN (current today) and a filtered version of EDDN (filters duplicates, perhaps cleans up garbage data).

I would envision something like this:

• Data sources send data to EDDN firehose
• EDDN firehose is connected to by large bandwidth cleanup sources and archives
• These re-publish to a clean stream that filters duplicates
• Low bandwidth users connect to that stream, knowing they're getting cleaner more concise data but potentially lacking some nuances

Cleanup code could even append extra data such as 'I've seen X messages like this in the past 24 hours' e.g. counting the hashes.

I'd also caution over how filtering is applied depending on level. There's a huge difference semantically between filtering identical messages out (this exact message has been sent multiple times) and filtering identical data out (regardless of source, current price of Station X Commodity Y is Z)

Just a random early-Sunday-morning thought.

We've had a few discussions in december on the topic, just search "hash" in the thread. AnthorNet mentioned that there already is a hash function in EDDN, but it's disabled. James was concerned about the load on the server. I can't find it, but I think I suggested this be done by the upstream before submitting to EDDN, and that would mean adding a field in the header for the schema.

It could be an optional field, and technically we could allow any hashing method since we are trying to suppress dupes from a same source. However, I think it would be good practice to agree on a specific method (e.g. MD5, SHA-1, FNV-1, whatever). It would have to be something easily available in all programming languages, to encourage community devs to use it.

EDIT: The hash in the header would be a hash of the "message" field/object

 

[Snake Man]

For that you can just check EDDB.

 

[AnthorNet]

I wanted to point out something about the potential duplicates filtering idea. It's actually quite useful when tracking the data to see duplicates; multiple reports of the same data from different sources actually increases confidence that the data is correct. Even with a full FD API, we can't rule out data tampering.

I don't disagree that having duplicates filtered out is useful for certain use cases and lower bandwidth - it absolutely is. I just think EDDN would be better retaining its original firehose intent. Why not have additional streams for different types or quality of data, including a firehose version of EDDN (current today) and a filtered version of EDDN (filters duplicates, perhaps cleans up garbage data).

I would envision something like this:

• Data sources send data to EDDN firehose
• EDDN firehose is connected to by large bandwidth cleanup sources and archives
• These re-publish to a clean stream that filters duplicates
• Low bandwidth users connect to that stream, knowing they're getting cleaner more concise data but potentially lacking some nuances

Cleanup code could even append extra data such as 'I've seen X messages like this in the past 24 hours' e.g. counting the hashes.

I'd also caution over how filtering is applied depending on level. There's a huge difference semantically between filtering identical messages out (this exact message has been sent multiple times) and filtering identical data out (regardless of source, current price of Station X Commodity Y is Z)

Just a random early-Sunday-morning thought.

The dedupe function is just filtering exact same message from a 5 minute timestamp.
Don't think it will filter much ^^

What it dedupe is for example when a user in EliteOCR fires twice the export to EDDN button with the same table.

Thus it does not change the original plan.
This could be monitored before being put in production.

 

[Snake Man]

EliteOCR prevents exporting same commodity to EDDN twice. When you export, it puts that eddn boolean field to true for all commodities exported.