Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[themroc]

Part of the reason that I support dropping v1 is that most of the publishing apps that are sending v1 are old and don't have support for the E: D 1.3 commodities. Also, according to the stats, a significant number of messages are from people using EliteOCR 0.5, and those people certainly should be encouraged to upgrade to a version of EliteOCR that uses the opencv engine.

I'm assuming that dropping v1 support will cause the vast majority of such people to upgrade, rather than give up.

That's a point

 

[wolverine2710]

Part of the reason that I support dropping v1 is that most of the publishing apps that are sending v1 are old and don't have support for the E: D 1.3 commodities. Also, according to the stats, a significant number of messages are from people using EliteOCR 0.5, and those people certainly should be encouraged to upgrade to a version of EliteOCR that uses the opencv engine.

I'm assuming that dropping v1 support will cause the vast majority of such people to upgrade, rather than give up.

Totally forgot to mention that in my post. We did discuss it in the "mailing list" though. That is another good/urgent reason to switch to the latest version of the sender tools. Publicly released receiver tools which are used by others like eddb.io (used by a lot of other tools) and Maddavo (for Trade Dangerous users) support receiving V1 and V2 messages so no data is lost when switching. I'm going to put your information in my original post here.

Note: If you are able to do so please update your EDMC thread and urge commanders to upgrade. Please do point them to my post (#635) here in this thread. If you aren't able to et me know and I will do it for you ;-)

 

[wolverine2710]

New development. I mentioned in my first post (#635) the following: "Otis B. of EDMC has created a "Retire v1 commodities schema" pull request and tested it with a few older clients on his own private server." AntorNet who did marvelous work for the now live EDDN v0.4 has the EDDN running on his own server, for testing and coding. The server is publicly accessible. The original plan was that he would update his version next Tuesday (he's on holiday) so all tool authors (senders and receivers) could test their tools. So they can check how older version of their sender tools will behave - and share that knowledge with their users on the authors OP or a post. As stated before when V1 messages are rejected and HTTP error 426 ("Upgrade Required") will be returned. But he found time to install the code (pull request by Otis B.) this morning. He has informed all authors in the "mailing list".

TLDR; AntorNet and Otis B. together made the following possible: A publicly accessible EDDN version which can be used by tool authors to see how their tools behave when the EDDN refuses the V1 schema.

Note: AntorNet has shared this information with the "mailing list" (ML) members. Most of the sender/receiver tool authors are on it. If you have a tool and want to check how it behaves when V1 is shutdown please send me a PM. I will add you to the ML so you have the info (ip address/port numbers) which makes it possible for you to test your tool.

 

[wolverine2710]

I've finished sending a "heads up" about shutting down the EDDN schema V1 to the last three commanders who I had not written before. Some have private services and some have a public value added service for other commanders. This weekend I've also been contacted by a commander who has a private tool but hasn't post in this thread before. He's on board, will add V2 support to his tool and he's on his request been added to the "mailing list"

I've also had contact with Duke of "Regulated Noise DJ-version *All in one* Trading Tool". His tool can OCR market screenshots. He is sending to and receiving data from the EDDN. He is going to support V2 as well. In his Github repository he already has implemented a new version of RN-DJ which is sending only V2 messages. After that his HD crashed, thankfully backups available. When his system is restored he is going to implement receiving V2 messages. I'm very hopeful that the new version will be ready and released before the 1st of July.

I strongly believe ALL commanders who have a tool/service for or are using the EDDN have been personally contacted by now. As long as it was mentioned here in this thread ;-)

Yesterday 23,015 V1 messages were send vs 912 V2 messages. As one V2 message is equal to approx 50 V1 messages that results in about 45.600 messages. Hence almost twice as many V2 messages then V1 messages were sent. Hopefully this will grow this week ;-) Well 9 days left to update/upgrade.

I'm assuming and hoping that commanders which have a public tool and a thread for that have or are going to inform their users about the shutdown and will advice them to use their latest tool. Quite a few have already done so. Thanks commanders ;-)

 

[wolverine2710]

Got a response from Askarr which I send a PM. As of the 22th of June both his tools support schema V2.

• EDDN RealTime Feed. Very cool. 3D view. Description: "See the systems light up as market data arrives."

• Archive of EDDN messages. Announcement.

That leaves 2 commanders who haven't PM-ed back. Thardie and Pandemic.
Both have an EDDN value added service. They both store eddn message which can be used by others, most likely tool authors.
Thardie has as a service for the end-user that by calling this url you receive a trade dangerous .prices back. Didn't test it.
Pandemic has a trading tool called elite-trade-analyzer. Github repo here. It needs Apache Spark. Last update repo 4 monhts ago.

 

[wolverine2710]

Using the EDDN-monitor found this wrt ration between V1 and V2 being received and distributed by the EDDN. As before I count one V2 message as 50 V1 messages as a V2 message contains the prices for a complete market.

• June the 20th: V2 messages 66%.
• Yesterday: V2 messages 73%.
• Today till now (16:51 GMT+1). V2 messages 80%.

I don't have a database so I've to rely on what I can daily see in the EDDN-monitor.
The trend I notice is a positive one ;-)

Note: Seebek of EliteOCR has informed his users yesterday and asked them to update to V0.70 which has V2 (only) support.

 

[Askarr]

Got a response from Askarr which I send a PM. As of the 22th of June both his tools support schema V2.

Sorry I didn't post here, it's been crazy at work. To confirm, both the DynamoDB storage and the visual feed should be compatible and processing V2 commodity schemas. As it's a pure JSON archive (it literally just stores the received JSON that matches a valid schema over time), I didn't have much to change & users should interact in the same way as before - you just get V2 JSON instead.

 

[AnthorNet]

I don't have a database so I've to rely on what I can daily see in the EDDN-monitor.
The trend I notice is a positive one ;-)

You know you only had to ask
For your pleasure: http://eddn.ed-td.space/schemas.html

Put a projected of 50, but I think it is more an average of 60.

 

[wolverine2710]

You know you only had to ask
For your pleasure: http://eddn.ed-td.space/schemas.html

Put a projected of 50, but I think it is more an average of 60.

Perfect. I did ask it here in general ;-)

Very very busy lately AND also was trying to reduce the SPAM (implementation requests) in our email conversations ;-)
As you have the data curious what the real avg is. I can then update my posts here and feel free to use the real avg in your very nice looking graph.

 

[AnthorNet]

Perfect. I did ask it here in general ;-)

Very very busy lately AND also was trying to reduce the SPAM (implementation requests) in our email conversations ;-)
As you have the data curious what the real avg is. I can then update my posts here and feel free to use the real avg in your very nice looking graph.

We were both near the average !
The average calculated on my database is: 55.0287

Updated the projected in the graph.

 

[Duke]

Hi guys,

as Wolverine asked me I will give you a small status report on the compatibility of RegulatedNoise with EDDN schema v2:
It is fully working and I'm doing only some test now. The coming release of RN accepts incoming v1 and v2 messages
and is sending only v2 messages. I will release it during this weekend, so it should be no problem to shutdown the v1
functionality on EDDN on July, 1st.

 

[NeutronCorean]

EDDN seems down in the moment ?

At least TD crashes with the message that i cant connect to it.

 

[wolverine2710]

EDDN seems down in the moment ?

At least TD crashes with the message that i cant connect to it.

According to the EDDN-monitor data is being distributed as I write this. Double checked with eddb.io (eddn section). Though there seems to be a gap between 2015-06-27 05:54:41 and 2015-06-27 10:22:27. Commanders with a receiver and a database did you notice something strange?

Wrt Trade Dangerous. I didn´t know that TD has EDDN receiver capability. Only part I know of is the EDAPI pligin which send data to EDDN. EDAPI gets its data from the FD-servers using the web-api as used by the official iPhone app.

Update: For more information about and an overview of OCR error Free (OEF) aka mobile api based tools see this FD thread.

 

[themroc]

Double checked with eddb.io (eddn section). Though there seems to be a gap between 2015-06-27 05:54:41 and 2015-06-27 10:22:27. Commanders with a receiver and a database did you notice something strange?

Some logs:

[eddn] [01:27:15] ... Receiving message: {"header": {"softwareVersion": "0.6.0.10", "gatewa ...
[eddb] [01:27:15] success: Indite in Cabrera Dock in LHS 1541
[eddn] [01:37:15] ... Reconnecting to relay
[eddn] [01:47:15] ... Reconnecting to relay
[eddn] [01:57:15] ... Reconnecting to relay
[eddn] [02:07:16] ... Reconnecting to relay
[eddn] [02:17:16] ... Reconnecting to relay
[eddn] [02:27:16] ... Reconnecting to relay
[eddn] [02:37:16] ... Reconnecting to relay
[eddn] [02:47:16] ... Reconnecting to relay
[eddn] [02:57:16] ... Reconnecting to relay
[eddn] [03:03:26] ... Receiving message: {"header": {"softwareVersion": "0.6.0.10", "gatewa ...
[eddb] [03:03:27] success: Bertrandite in Cabrera Dock in LHS 1541

And further down:

[eddn] [03:54:41] ... Receiving message: {"header": {"softwareVersion": "0.6.0.10", "gatewa ...
[eddb] [03:54:41] success: Fish in Thompson Ring in Juduni
[eddn] [04:04:42] ... Reconnecting to relay
[eddn] [04:14:42] ... Reconnecting to relay
[eddn] [04:24:42] ... Reconnecting to relay
[eddn] [04:34:42] ... Reconnecting to relay
[eddn] [04:44:42] ... Reconnecting to relay
[eddn] [04:54:42] ... Reconnecting to relay
[eddn] [05:04:42] ... Reconnecting to relay
[eddn] [05:14:42] ... Reconnecting to relay
[eddn] [05:24:42] ... Reconnecting to relay
[eddn] [05:34:42] ... Reconnecting to relay
[eddn] [05:44:43] ... Reconnecting to relay
[eddn] [05:54:43] ... Reconnecting to relay
[eddn] [06:04:43] ... Reconnecting to relay
[eddn] [06:14:43] ... Reconnecting to relay
[eddn] [06:24:43] ... Reconnecting to relay
[eddn] [06:34:43] ... Reconnecting to relay
[eddn] [06:44:43] ... Reconnecting to relay
[eddn] [06:54:43] ... Reconnecting to relay
[eddn] [07:04:43] ... Reconnecting to relay
[eddn] [07:14:43] ... Reconnecting to relay
[eddn] [07:24:43] ... Reconnecting to relay
[eddn] [07:34:44] ... Reconnecting to relay
[eddn] [07:44:44] ... Reconnecting to relay
[eddn] [07:54:44] ... Reconnecting to relay
[eddn] [08:04:44] ... Reconnecting to relay
[eddn] [08:14:44] ... Reconnecting to relay
[eddn] [08:22:27] ... Receiving message: {"header": {"softwareVersion": "1.5.1.2", "gateway ...
[eddb] [08:22:27] success: Updating 58 listings in <White Ring> in system <LHS 3802>.

My EDDN listener automatically reconnects if it did not receive any message for 10 minutes. It has been proving bullet proof to prevent shadow connections. So yeah, no data received

 

[wolverine2710]

I have send an email to jamesremuscat and AnthorNet to inform them.

@NeutronCorean. Thanks for reporting, apreciated.

Update: It turned out AnthorNet had actually send me an email informing me about the outage. Missed that one, my bad.

 

[wolverine2710]

AnthorNet created a stats page for schema, which you can find here.
He very recently added a percentages stats graph. Very nicely done btw ;-)

Well I have brilliant news to report. In the last 7 days (20th of June till the 27th of June) the percentage of V2 messages has gone from 61.8% to 91%. Today the 28th is still on going (18:15 GMT+1 atm) the percentage of V2 messages has gone up to 94,2%. The latest increase is no doubt related to the fact that Duke on the 26th released a new version of his popular and great RN-DJ tool (one of its features is a very good OCR-ing solution). Atm all OCR Eror Free tools in their latest release support schema V2 only. Same for the OCR-ing tools. Also nearly all publicly released tools support receiving schema V2. All major ones DO support V2.

The only two tools which have not V2 receiving capability are the tools by Thardi and Pandemic. At least they haven´t responded to my PM´s. Not sure if they are still palying ED. Pandemic has created 5 post in the FD forum, last post was 03/02/2015 was here in this thread, 11:57 PM. thardie has created 3 posts in the FD forum, last post 23/02/2015, 5:25 AM was here in this thread.


Copy/paste from post #645:

That leaves 2 commanders who haven't PM-ed back. Thardie and Pandemic.
Both have an EDDN value added service. They both store eddn message which can be used by others, most likely tool authors.
Thardie has as a service for the end-user that by calling this url you receive a trade dangerous .prices back. Didn't test it.
Pandemic has a trading tool called elite-trade-analyzer. Github repo here. It needs Apache Spark. Last update repo 4 monhts ago.

I truly appreciate the great work they have done wrt to their Value Added Services (VAS). Thardi´s service where you can get download a .prices file for TD is probably no longer up to date since Power Play introduced new commodities. His service connects to the EDDN and hence will receive data from the OCR Error Free tools. These tools get the data from the web-api and those commodity names are different then used ingame in ED. So have to be translated for TD. Hence unless he has updated his service for PP the TD .prices can´t be used any longer. I can´t know how much their archiving services are used - their audience are tool authors. I have his feeling their services won´t effect normal ED commanders. So far haven´t seen posts here that commanders/tool authors have stated they are using those VAS. Perhaps when they notice that their services no longer receive V1 messages that they contact this thread, though like I said I PM-ed them.

TL;DR The schema V2 usage has gone up to approx 94%. Personally I think dropping of the V1 schema support in the EDDN starting the 1st of July as was planned and communicated is safe to do.

Note: Hopefully after the switch I can stop with these kind of large posts ;-) The life of an perfectionist is not a good one ;-(

 

[wolverine2710]

Got a PM back from Thardie today - see last post. He´s going to try and find some time to update his tool. He´s not sure if anyone is using his tool though. Which really is a shame because its a wonderful tool for TD users. Perhaps when he had promoted his tool more or when I had mentioned it in the TD thread it would have been used more. I think his excellent tool totally go snowed under. Even I had forgotten about his tool. BUT its in the website version (not released) of the third party tools thread, so who knows...

So that leaves Pandemic which has a pure archiving tool. Typically main audience are tool authors. Has anyone used his tool?

 

[Andargor]

Got a PM back from Thardie today - see last post. He´s going to try and find some time to update his tool. He´s not sure if anyone is using his tool though. Which really is a shame because its a wonderful tool for TD users. Perhaps when he had promoted his tool more or when I had mentioned it in the TD thread it would have been used more. I think his excellent tool totally go snowed under. Even I had forgotten about his tool. BUT its in the website version (not released) of the third party tools thread, so who knows...

So that leaves Pandemic which has a pure archiving tool. Typically main audience are tool authors. Has anyone used his tool?

You realize you've used the word "tool" 9 times in this post?

Keep up the good work!

 

[Otis B.]

I'm seeing sporadic timeouts sending to EDDN today. Anyone else?

 

[AnthorNet]

I'm seeing sporadic timeouts sending to EDDN today. Anyone else?

Can't say, but will be investigated