Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[DRY411S]

*me* Shakes head in disbelief that ANYBODY feels they have the right to complain about a little downtime on a service that is provided completely free of charge by volunteers.

To those volunteers, I beg you to take the positives out of this. i.e. that it is so important to people they feel they should complain. You have made their lives richer by doing what you do.

 

[seeebek]

I have a technical question for james or wolverine.

I want to speed up the EDDN export in EliteOCR and yesterday somebody in eddb.io IRC channel told me to try parallel requests. I didn't try it yet and I though I should ask here first. My questions:
1. Did anybody try it out?
2. Is there a limit of connections from one ip? Do people get blocked if there are too many connections (DDoS prevention)?
3. How high is the chance of dropped data?
4. What would be a recommended amount of requests from one user at the same time?
5. Do you have any recommendations, tips or alternative ideas?

 

[jamesremuscat]

I have a technical question for james or wolverine.

I want to speed up the EDDN export in EliteOCR and yesterday somebody in eddb.io IRC channel told me to try parallel requests. I didn't try it yet and I though I should ask here first. My questions:
1. Did anybody try it out?
2. Is there a limit of connections from one ip? Do people get blocked if there are too many connections (DDoS prevention)?
3. How high is the chance of dropped data?
4. What would be a recommended amount of requests from one user at the same time?
5. Do you have any recommendations, tips or alternative ideas?

1. As far as I'm aware, nobody has tried this yet;
2. Not currently; if people get silly, then maybe!
3. I would hope, zero; the HTTP gateway is set up to be multi-threaded so should JustWork(TM)
4. I don't really have any feel for it, but ten is probably OK, one hundred is probably not.
5. A better solution - one that's already been raised in this thread - would be a schema that lets you submit all the data items you have at once, rather than one message per commodity. It's on the backlog!

 

[AnthorNet]

Hey guys !
Any news on new shema specifically a new one with commodities array ?

Would be great to have support from website, as for now having nearly 60 post requests in one page update(eg station prices update),
is a little to much loading for end user.
We could make it on a separate process, by adding queue, but an Array would be much more effective I think.

I've read all the post and didn't remember seeing advance on that matter.

Thx

 

[jamesremuscat]

Any news on new schema specifically a new one with commodities array ?

I've not done any work towards this yet. If anyone out there feels like jumping in to do so, please feel free

 

[jamesremuscat]

Just a heads-up that I'll be taking the EDDN down for a software upgrade at some point in the next day or two. Total interruption to service should be less than ten minutes.

EDIT:

The upgrade was completed successfully and with minimal service interruption.

New features are mostly behind-the-scenes admin related, making the network components easier to deploy and administer, but there is also now a status page available at http://eddn-gateway.elite-markets.net/ showing message throughput of the network.

AnthorNet and I have been collaborating on a v2 schema that lets you submit multiple commodities in one message. The draft is up on GitHub: https://github.com/jamesremuscat/EDDN/blob/master/schemas/commodity-v2.0-draft.json. It's also installed in the gateway so software authors can move to testing using it if they wish - usual /test hack applies. The schema is subject to change until we declare it stable.

 

[Snake Man]

EDDN Status page will be handy for those moments where one wonders if the network down, if it just super quiet in Elite galaxy or is it in our end, thank you for providing this page, appreciated.

 

[seeebek]

I released a new version of EliteOCR and it has asynchronous export to EDDN. Basically everything tries to go at once. It would be helpful if you could observe if there are no negative effects because of this.
I will concentrate on testing the new schema in the future.

 

[AnthorNet]

I released a new version of EliteOCR and it has asynchronous export to EDDN. Basically everything tries to go at once. It would be helpful if you could observe if there are no negative effects because of this.
I will concentrate on testing the new schema in the future.

Wich version should be monitored ?

 

[seeebek]

Wich version should be monitored ?

EliteOCR 0.6 it creates many parallel requests

 

[AnthorNet]

EliteOCR 0.6 it creates many parallel requests

I launched a client monitoring only that version, will tell you as soon I receive results.

 

[jamesremuscat]

I released a new version of EliteOCR and it has asynchronous export to EDDN. Basically everything tries to go at once. It would be helpful if you could observe if there are no negative effects because of this.
I will concentrate on testing the new schema in the future.

Thanks for the heads-up - I'll continue to monitor the server and shout if it starts getting sad, but I'm expecting it to be fine.

 

[Andargor]

I'm running some tests, and trying to post the following to EDDN gives me an error 400. I must be blind, any ideas?

EDIT: nevermind, the version was numeric instead of a string.

{
	"header": {
		"uploaderID": "Andargor TEST",
		"softwareName": "EDCE",
		"softwareVersion": 0.1
	},
	"$schemaRef": "http://schemas.elite-markets.net/eddn/commodity/1/test",
	"message": {
		"stationStock": 19308,
		"sellPrice": 197,
		"demand": 0,
		"itemName": "Explosives",
		"timestamp": "2015-04-27T23:47:52.823093",
		"stationName": "Rozhdestvensky Horizons",
		"systemName": "La Tenha",
		"buyPrice": 212
	}
}

 

[Andargor]

Request for new schema for commanders. This is a schema that allows commanders to report their general status, location and ranks. Useful for such things as heatmaps, exploration logs, and other social aspects.

Example JSON:

{
	"header": {
		"uploaderID": "Andargor TEST",
		"softwareName": "EDCE",
		"softwareVersion": 0.1
	},
	"$schemaRef": "http://schemas.elite-markets.net/eddn/commander-light/1",
	"message": {
		"name": "Andargor",
		"docked": true,
		"rank": {
			"combat": 5,
			"trade": 5,
			"explore": 6,
			"crime": 0,
			"service": 0,
			"empire": 10,
			"federation": 10
		},
		"lastSystem": {
			"name": "La Tenha",
			"faction": "Federation"
		},
		"lastStarport": {
			"name": "Rozhdestvensky Horizons",
			"faction": "Federation"
		},
		"timestamp": "2015-04-27T23:54:18.367129"
	}
}

Proposed schema (hopefully I didn't mess it up too badly):

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"id": "http://schemas.elite-markets.net/eddn/commander-light/1#",
	"type": "object",
	"additionalProperties": false,
	"properties": {
		"$schemaRef": {
			"id": "#$schemaRef",
			"type": "string",
			"additionalProperties": false
		},
		"header": {
			"id": "#header",
			"type": "object",
			"additionalProperties": true,
			"properties": {
				"uploaderID": {
					"id": "#uploaderID",
					"type": "string",
					"additionalProperties": false
				},
				"softwareName": {
					"id": "#softwareName",
					"type": "string",
					"additionalProperties": false
				},
				"softwareVersion": {
					"id": "#softwareVersion",
					"type": "string",
					"additionalProperties": false
				},
				"gatewayTimestamp": {
					"id": "#gatewayTimestamp",
					"type": "string",
					"format": "date-time",
					"description": "Timestamp upon receipt at the gateway. If present, this property will be overwritten by the gateway; submitters are not intended to populate this property.",
					"additionalProperties": false
				}
			}
		},
		"message": {
			"id": "#message",
			"type": "object",
			"additionalProperties": true,
			"properties": {
				"name": {
					"id": "#name",
					"type": "string",
					"additionalProperties": false
				},
				"docked": {
					"id": "#docked",
					"type": "boolean",
					"additionalProperties": false
				},
				"rank": {
					"id": "#rank",
					"type": "object",
					"properties": {
						"combat": {
							"id": "#combat",
							"type": "integer",
							"additionalProperties": false
						},
						"trade": {
							"id": "#trade",
							"type": "integer",
							"additionalProperties": false
						},
						"explore": {
							"id": "#explore",
							"type": "integer",
							"additionalProperties": false
						},
						"crime": {
							"id": "#crime",
							"type": "integer",
							"additionalProperties": false
						},
						"service": {
							"id": "#service",
							"type": "integer",
							"additionalProperties": false
						},
						"empire": {
							"id": "#empire",
							"type": "integer",
							"additionalProperties": false
						},
						"federation": {
							"id": "#federation",
							"type": "integer",
							"additionalProperties": false
						},
						"required": ["combat",
						"trade",
						"explore",
						"crime",
						"service",
						"empire",
						"federation"]
					},
					"additionalProperties": false
				},
				"lastSystem": {
					"id": "#lastSystem",
					"type": "object",
					"properties": {
						"name": {
							"id": "#name",
							"type": "string",
							"additionalProperties": false
						},
						"faction": {
							"id": "#faction",
							"type": "string",
							"additionalProperties": false
						},
						"required": ["name",
						"faction"]
					},
					"additionalProperties": false
				},
				"lastStarport": {
					"id": "#lastStarport",
					"type": "object",
					"properties": {
						"name": {
							"id": "#name",
							"type": "string",
							"additionalProperties": false
						},
						"faction": {
							"id": "#faction",
							"type": "string",
							"additionalProperties": false
						},
						"required": ["name",
						"faction"]
					},
					"additionalProperties": false
				},
				"timestamp": {
					"id": "#timestamp",
					"type": "string",
					"format": "date-time",
					"additionalProperties": false
				}
			},
			"required": ["name",
			"docked",
			"rank",
			"lastSystem",
			"lastStarport",
			"timestamp"]
		}
	},
	"required": ["$schemaRef",
	"header",
	"message"]
}

 

[Andargor]

I released a new version of EliteOCR and it has asynchronous export to EDDN. Basically everything tries to go at once. It would be helpful if you could observe if there are no negative effects because of this.
I will concentrate on testing the new schema in the future.

Seems to run fine here. Pretty cool to see EDDN update so fast.

 

[AnthorNet]

EliteOCR 0.6 it creates many parallel requests

Working well, and receiving faster than before.


@Andagor: Think new schemas was discussed, and told that only prices should be passed to EDDN.
Eventually we could have another fork with much infos, I'd like to push systems/stations updates in the loop.
Even if they arr static, for now they are almost dynamic for database as we dont' have all infos, this could change with frontier API when it comes

 

[AnthorNet]

Hey guys,
As it was said by James two weeks ago, the new v2 schema is being tested.

I do not planning on modify it again, so it could be considered as STABLE.
If you have any objection, or modifications to push, please contact us, and we will see what can be done.

 

[wolverine2710]

Working well, and receiving faster than before.


@Andagor: Think new schemas was discussed, and told that only prices should be passed to EDDN.
Eventually we could have another fork with much infos, I'd like to push systems/stations updates in the loop.
Even if they arr static, for now they are almost dynamic for database as we dont' have all infos, this could change with frontier API when it comes

If I'm not mistaken the (original) plan for EDDN was to be limited to Dynamic Data. Static data like coordinates, kind of landings pads for a station is the domain of things like TGC and its new implementation by a commander. Also it can be stored in tools like EDDB. Aside from prices all other kinds of Dynamic Data could be used for EDDN.

For me the scheme suggested by Andargor concerns Dynamic Data. I do however think that this kind of personal data should be an opt-in option for tools like heat maps. Otherwise it would be possible to track the whereabouts of a certain commander. I haven't looked at all the heatmap like tools so don't know what they currently show/publish to the outside world. The same opt-in principle should be true for all other kind of schemes/data which are personal.

Just my 2 euro cent. What do others think.

 

[wolverine2710]

Hey guys,
As it was said by James two weeks ago, the new v2 schema is being tested.

I do not planning on modify it again, so it could be considered as STABLE.
If you have any objection, or modifications to push, please contact us, and we will see what can be done.

Was buried in ED3PTT work. Didn't have time check it out, my bad.

In the past with V1 aside from the fact that each and every commodity was a single message (for backwards compatibility) there were quite a few discussions about things which were missing or could be done better. NOW is your change to requirement known again.

I do have a proposal for a requirement. Atm there are relative few messages send to EDDN and published by it. When the ED API's arrives this hopefully will explode. Currently the free hosting provided by Vivio Technologies gives us a 20 Mbps unmetered internet connection. I believe this will change in the future and we will get more bandwidth.

Suppose the ED API is there and EDDN will be flooded with messages and a lot of them could/will be the same. As in each time a commander opens the commodities market the prices for that station/outpost are uploaded - ofc the commander has to be running a tool which uploads to EDDN. If more commanders arrive at that station more or less at the same time the prices will be the same data will be transmitted. Supppose the current 20Mbps is full but EDDN can handle it. That is NOT to say that commanders who want to connect to EDDN for their public/private tools can handle it. Not everyone has a lot of bandwidth.

Like I said a lot of data could be the same, it will be received by EDDN BUT it doesn't mean EDDN has to publish all those duplicates. Suppose a hash is created (there are some extremely fast ones out there) for all the commodities and and their prices. When the message is received by EDDN is creates a hash also. If the hashes are the same it should not be send out. This will make it possible that commanders with less bandwidth can still connect to EDDN and it also means saving bandwidth.

Ofc it means extra work for EDDN, coding wise and CPU wise. The aim of EDDN is to keep things simple. Do you see merits in my suggestion? Please us all know what you think....

 

[wolverine2710]

Just a heads-up that I'll be taking the EDDN down for a software upgrade at some point in the next day or two. Total interruption to service should be less than ten minutes.

EDIT:

The upgrade was completed successfully and with minimal service interruption.

New features are mostly behind-the-scenes admin related, making the network components easier to deploy and administer, but there is also now a status page available at http://eddn-gateway.elite-markets.net/ showing message throughput of the network.

AnthorNet and I have been collaborating on a v2 schema that lets you submit multiple commodities in one message. The draft is up on GitHub: https://github.com/jamesremuscat/EDDN/blob/master/schemas/commodity-v2.0-draft.json. It's also installed in the gateway so software authors can move to testing using it if they wish - usual /test hack applies. The schema is subject to change until we declare it stable.

I've finally updated the OP with this information, my bad...