Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

Status
Thread Closed: Not open for further replies.

jamesremuscat

J
MagmaiKH said:
On related topics ...

Data scrubbing should be the primary responsibility of the OCR tool.

Data scrubbing should be the secondary responsibility of the receiving client.

Allowing a client or even a special client to rebroadcast data seems like a bad idea architecturally. It's begging for "email bomb" sort of problems where 2 scrubbers keep sending each other scrubbed data back and forth which piles up as more POST come through until the pipe is full and behind the capacity of any end-client-node.
Click to expand...

I pretty much agree with you there. As you say later: SEP. :)

MagmaiKH said:
I am not impressed with 0MQ. It's a massive load of complexity to solve a rather simple problem and at best the C# implementations are buggy, at worst the underlying architecture is invalid.
Click to expand...
I can't speak to the C# implementations, but (barring some cross-version incompatibility) the Python one seems robust. Indeed, 0MQ is the foundation of a similar network for another internet spaceship game and currently handles some 200 messages/minute with a large number of uploaders and subscribers... For reference, I just did a count and there are currently around 450 subscribers connected to EDDN.

MagmaiKH said:
Making the commodity the first order object is a design flaw that needs to be corrected to improve EDDN services. I think the first-order object should be the station. A complete set of commodities is expected and this allows additional station information to be included. Right now there's a ton of redundant data sent with the station name, dates, et. al. sent over and over again for each commodity.
Click to expand...

Completely agree on this - the only thinking in the v1 schema design was to remain close to the EMDN format to try and preserve as much client compatibility as we could. Someone else has already suggested a more normalised schema for v2 that addresses the points you raise. At some point, I might have time to work on it! ;)

MagmaiKH said:
Each station update sent out should be accompanied by a generated GUID (or UUID) that can be used to track that transaction. Another message on the EDDN should be to declare that GUID of data no-good. Broadcast that data to the clients and if the client gets enough votes it can purge the bad data associated with that GUID.
Click to expand...

That's actually a pretty nice idea. Clients who don't care about message quality can simply ignore the "no-good" messages, those that want to keep track can. Would there be anything to prevent someone flooding the network with malicious "no-good" messages? (I mean, any more than they could already flood bad market data messages...)
 
Last edited:

maxh2003

M
jamesremuscat said:
I can't speak to the C# implementations (of 0MQ)
Click to expand...

Virtually no bugs reported on RN with the EDDN subscriber functionality - just one person who seemed to get empty messages at one point, haven't seen it fail at all other than that. Maybe I'm doing something right - or wrong - that I'm not aware of. It uses ZeroMQ via nuget, IIRC. Code is on GitHub for anyone who wants it: https://github.com/stringandstickytape/RegulatedNoise/blob/master/RegulatedNoise/EDDN.cs

+1 for "No-Good" invalidation messages, that's a sound idea.
 

CMDRKNac

C
@maddavo, this is a code snippet to you do it with python 3.4 (or any other python 3):

Code: 
import zmq, zlib

context = zmq.Context()
subscriber = context.socket(zmq.SUB)
    
subscriber.setsockopt(zmq.SUBSCRIBE, b"")
#subscriber.setsockopt(zmq.RCVTIMEO, 65000)
subscriber.connect("tcp://eddn-relay.elite-markets.net:9500")
    
while True:
        try:
            live_market_json = zlib.decompress(subscriber.recv())
        except:
            print('EDDN is down')
            return
        
        live_market_data = json.loads(live_market_json)        
        if live_market_data['$schemaRef'] == 'http://schemas.elite-markets.net/eddn/commodity/1':
...

Probably your problem comes that in the example if I recall right gevent binding was used, this is not supported in Python 3, but you actually don't need gevent to use zmq (it's optional and probably for most needs not needed, there are alternatives to gevent for Python 3 which are maintained also).
You only need to specify that you are receiving byte objects and not string objects (if you get to know Python 3 you will find out more about the two different types).

EDIT: I see you solved it. Still will leave this here in case somebody runs into similar problems.
 
Last edited:

maddavo

maddavo
CMDRKNac said:
@maddavo, this is a code snippet to you do it with python 3.4 (or any other python 3):

...snip...

Probably your problem comes that in the example if I recall right gevent binding was used, this is not supported in Python 3, but you actually don't need gevent to use zmq (it's optional and probably for most needs not needed, there are alternatives to gevent for Python 3 which are maintained also).
You only need to specify that you are receiving byte objects and not string objects (if you get to know Python 3 you will find out more about the two different types).

EDIT: I see you solved it. Still will leave this here in case somebody runs into similar problems.
Click to expand...

Thanx for that. More examples helps to get my head around it. Rather than setsockopt, I had to use setsockopt_string:

Code: 
    context = zmq.Context()
    subscriber = context.socket(zmq.SUB)


    subscriber.setsockopt_string(zmq.SUBSCRIBE, "")
    subscriber.connect('tcp://eddn-relay.elite-markets.net:9500')

Now I've come across an issue with recv. In the docs I see it is blocking by default. I want to use it in non-blocking mode. I don't know the syntax of doing that - probably due to lack of Python knowledge. Is it something like this?? :

Code: 
            rawmsg = subscriber.recv(,,,ZMQ_DONTWAIT)           #         check for a message
            if rawmsg != EAGAIN :                               #         if there's a message then
                  #do stuff
or basically how do you check for the EAGAIN error flag. Should I use a "Try" statement - haven't 'tried' those yet.
 

CMDRKNac

C
You have to look for the zeromq documentation, it should say it there. But if you are registering a parameter into the configuration to the running app you probably need to do something like:

Code: 
subscriber.setsockopt(zmq.SUBSCRIBE, b"")
...
subscriber.setsockopt(zmq.DONTWAIT)

(copy paste won't work probably as I don't know the exact argument to pass, but you get the idea)
First you register the application (the listening socket), then in the context of the running instance of the application you register additional parameters through calling methods of the class object. This way of doing things is very common across several python modules and libraries (well in many OO languages really but).

P.S: Notice that using subscriber.setsockopt(zmq.SUBSCRIBE, b"") instead of using subscriber.setsockopt_string(zmq.SUBSCRIBE, "") are equivalent in practice. In the first you are telling the subscriber that it will be receiving byte encoded strings not other type of objects, in the other you are telling the subscriber that what it's receiving is a string object.
 
Last edited:

seeebek

seeebek
Hi,
so I implemented German and French in EliteOCR. It should now always upload to EDDN in English (exceptions are rare commodities).

If you are interested how the commodities were translated to those languages here is my commodities.json (I hatethe person who made the french translation).

{ "TOBACCO": {
"fra": "TABAC",
"rare": "False",
"deu": "TABAK"
},
"COFFEE": {
"fra": "CAFÉ",
"rare": "False",
"deu": "KAFFEE"
},
"BERTRANDITE": {
"fra": "BERTRANDITE",
"rare": "False",
"deu": "BERTRANDIT"
},
"GOLD": {
"fra": "OR",
"rare": "False",
"deu": "GOLD"
},
"BAUXITE": {
"fra": "BAUXITE",
"rare": "False",
"deu": "BAUXIT"
},
"ADVANCED CATALYSERS": {
"fra": "CATALYSEURS COMPLEXES",
"rare": "False",
"deu": "FORTSCHR. KATALYSATOREN"
},
"SUPERCONDUCTORS": {
"fra": "SUPRACONDUCTEURS",
"rare": "False",
"deu": "SUPRALEITER"
},
"FISH": {
"fra": "POISSON",
"rare": "False",
"deu": "FISCH"
},
"GALLITE": {
"fra": "GALLITE",
"rare": "False",
"deu": "GALLIT"
},
"ATMOSPHERIC PROCESSORS": {
"fra": "PROCESSEURS ATMOSPHÉRIQUES",
"rare": "False",
"deu": "ATMOSPHÄRENPROZESSOREN"
},
"FOOD CARTRIDGES": {
"fra": "CARTOUCHES ALIMENTAIRES",
"rare": "False",
"deu": "NAHRUNGSKARTUSCHEN"
},
"TITANIUM": {
"fra": "TITANE",
"rare": "False",
"deu": "TITAN"
},
"COBALT": {
"fra": "COBALT",
"rare": "False",
"deu": "COBALT"
},
"BATTLE WEAPONS": {
"fra": "ARMES MILITAIRES",
"rare": "False",
"deu": "KRIEGSWAFFEN"
},
"BIOREDUCING LICHEN": {
"fra": "LICHEN BIORÉDUCTEUR",
"rare": "False",
"deu": "BIOREDUZIERENDE FLECHTEN"
},
"WINE": {
"fra": "VIN",
"rare": "False",
"deu": "WEIN"
},
"PROGENITOR CELLS": {
"fra": "CELLULES SOUCHES",
"rare": "False",
"deu": "VORLÄUFERZELLEN"
},
"CLOTHING": {
"fra": "VÊTEMENTS",
"rare": "False",
"deu": "KLEIDUNG"
},
"WATER PURIFIERS": {
"fra": "PURIFICATEURS D'EAU",
"rare": "False",
"deu": "WASSERREINIGER"
},
"BASIC MEDICINES": {
"fra": "MÉDICAMENTS SIMPLES",
"rare": "False",
"deu": "ALLGEMEINE MEDIKAMENTE"
},
"PLATINUM": {
"fra": "PLATINE",
"rare": "False",
"deu": "PLATIN"
},
"ALUMINIUM": {
"fra": "ALUMINIUM",
"rare": "False",
"deu": "ALUMINIUM"
},
"AUTO-FABRICATORS": {
"fra": "DISPOSITIFS D'AUTOFABRICATION",
"rare": "False",
"deu": "FABRIKATOREN"
},
"NATURAL FABRICS": {
"fra": "FIBRE TEXTILE NATURELLE",
"rare": "False",
"deu": "NATURFASERN"
},
"RUTILE": {
"fra": "RUTILE",
"rare": "False",
"deu": "RUTIL"
},
"POLYMERS": {
"fra": "POLYMÈRES",
"rare": "False",
"deu": "POLYMERE"
},
"URANINITE": {
"fra": "URANINITE",
"rare": "False",
"deu": "URANINIT"
},
"EXPLOSIVES": {
"fra": "EXPLOSIFS",
"rare": "False",
"deu": "SPRENGSTOFFE"
},
"REACTIVE ARMOUR": {
"fra": "PROTECTION RÉACTIVE",
"rare": "False",
"deu": "REAKTIVRÜSTUNG"
},
"MICROBIAL FURNACES": {
"fra": "HAUTS FOURNEAUX MICROBIENS",
"rare": "False",
"deu": "MIKROBIELLE ÖFEN"
},
"TANTALUM": {
"fra": "TANTALE",
"rare": "False",
"deu": "TANTAL"
},
"PERFORMANCE ENHANCERS": {
"fra": "PRODUITS DOPANTS",
"rare": "False",
"deu": "LEISTUNGSSTEIGERER"
},
"COMPUTER COMPONENTS": {
"fra": "COMBINAISONS HAZMAT",
"rare": "False",
"deu": "COMPUTERTEILE"
},
"COLTAN": {
"fra": "COLTAN",
"rare": "False",
"deu": "COLTAN"
},
"CHEMICAL WASTE": {
"fra": "MATÉRIAUX RADIOACTIFS",
"rare": "False",
"deu": "CHEMIEMÜLL"
},
"PESTICIDES": {
"fra": "PESTICIDES",
"rare": "False",
"deu": "PESTIZIDE"
},
"ANIMAL MEAT": {
"fra": "VIANDE",
"rare": "False",
"deu": "TIERFLEISCH"
},
"BIOWASTE": {
"fra": "DÉCHETS ORGANIQUES",
"rare": "False",
"deu": "BIOMÜLL"
},
"LIQUOR": {
"fra": "SPIRITUEUX",
"rare": "False",
"deu": "SPIRITUOSEN"
},
"ROBOTICS": {
"fra": "ROBOTS",
"rare": "False",
"deu": "ROBOTER"
},
"COPPER": {
"fra": "CUIVRE",
"rare": "False",
"deu": "KUPFER"
},
"SYNTHETIC FABRICS": {
"fra": "TISSU SYNTHÉTIQUE",
"rare": "False",
"deu": "CHEMIEFASERN"
},
"RESONATING SEPARATORS": {
"fra": "SÉPARATEURS À RÉSONANCE",
"rare": "False",
"deu": "RESONANZABGRENZER"
},
"HYDROGEN FUEL": {
"fra": "CARBURANT À BASE D'HYDROGÈNE",
"rare": "False",
"deu": "WASSERSTOFF-TREIBSTOFF"
},
"PERSONAL WEAPONS": {
"fra": "ARMES DE POING",
"rare": "False",
"deu": "PERSÖNLICHE WAFFEN"
},
"LEATHER": {
"fra": "CUIR",
"rare": "False",
"deu": "LEDER"
},
"AGRI-MEDICINES": {
"fra": "AGRI-MÉDICAMENTS",
"rare": "False",
"deu": "AGRAR-MEDIKAMENTE"
},
"SCRAP": {
"fra": "FERRAILLE",
"rare": "False",
"deu": "SCHROTT"
},
"INDITE": {
"fra": "INDITE",
"rare": "False",
"deu": "INDIT"
},
"GALLIUM": {
"fra": "GALLIUM",
"rare": "False",
"deu": "GALLIUM"
},
"LEPIDOLITE": {
"fra": "LÉPIDOLITE",
"rare": "False",
"deu": "LEPIDOLITH"
},
"IMPERIAL SLAVES": {
"fra": "EMPIRE ESCLAVES",
"rare": "False",
"deu": "IMPERIALE SKLAVEN"
},
"NARCOTICS": {
"fra": "NARCOTIQUES",
"rare": "False",
"deu": "DROGEN"
},
"SEMICONDUCTORS": {
"fra": "SEMI-CONDUCTEURS",
"rare": "False",
"deu": "HALBLEITER"
},
"NON-LETHAL WEAPONS": {
"fra": "ARMES INCAPACITANTES",
"rare": "False",
"deu": "NICHTLETALE WAFFEN"
},
"URANIUM": {
"fra": "URANIUM",
"rare": "False",
"deu": "URAN"
},
"CONSUMER TECHNOLOGY": {
"fra": "ÉQUIPEMENT DE LOISIR",
"rare": "False",
"deu": "UNTERHALTUNGSELEKTRONIK"
},
"LITHIUM": {
"fra": "LITHIUM",
"rare": "False",
"deu": "LITHIUM"
},
"SILVER": {
"fra": "ARGENT",
"rare": "False",
"deu": "SILBER"
},
"BERYLLIUM": {
"fra": "BÉRYLLIUM",
"rare": "False",
"deu": "BERYLLIUM"
},
"MINERAL EXTRACTORS": {
"fra": "EXTRACTEURS DE MINERAI",
"rare": "False",
"deu": "MINERALEXTRAKTOREN"
},
"PALLADIUM": {
"fra": "PALLADIUM",
"rare": "False",
"deu": "PALLADIUM"
},
"DOMESTIC APPLIANCES": {
"fra": "MAQUILLAGE",
"rare": "False",
"deu": "HAUSHALTSGERÄTE"
},
"COMBAT STABILISERS": {
"fra": "STABILISATEURS DE COMBAT",
"rare": "False",
"deu": "KAMPFSTABILISATOREN"
},
"INDIUM": {
"fra": "INDIUM",
"rare": "False",
"deu": "INDIUM"
},
"ANIMAL MONITORS": {
"fra": "SYSTÈMES DE SURVEILLANCE",
"rare": "False",
"deu": "TIERÜBERWACHUNG"
},
"SYNTHETIC MEAT": {
"fra": "VIANDE SYNTHÉTIQUE",
"rare": "False",
"deu": "KÜNSTLICHES FLEISCH"
},
"TEA": {
"fra": "THÉ",
"rare": "False",
"deu": "TEE"
},
"MINERAL OIL": {
"fra": "HUILE MINÉRALE",
"rare": "False",
"deu": "MINERALÖL"
},
"BEER": {
"fra": "BIÈRE",
"rare": "False",
"deu": "BIER"
},
"POWER GENERATORS": {
"fra": "GÉNÉRATEURS",
"rare": "False",
"deu": "STROMERZEUGER"
},
"SLAVES": {
"fra": "ESCLAVES",
"rare": "False",
"deu": "SKLAVEN"
},
"FRUIT AND VEGETABLES": {
"fra": "FRUITS ET LÉGUMES",
"rare": "False",
"deu": "OBST UND GEMÜSE"
},
"AQUAPONIC SYSTEMS": {
"fra": "SYSTÈMES AQUAPONIQUES",
"rare": "False",
"deu": "AQUAPONIKSYSTEME"
},
"GRAIN": {
"fra": "CÉRÉALES",
"rare": "False",
"deu": "GETREIDE"
},
"H.E. SUITS": {
"fra": "COMBINAISONS DE PROTECTION",
"rare": "False",
"deu": "SCHUTZANZÜGE"
},
"CROP HARVESTERS": {
"fra": "MOISSONNEUSES",
"rare": "False",
"deu": "ERNTEMASCHINEN"
},
"ALGAE": {
"fra": "ALGUES",
"rare": "False",
"deu": "ALGEN"
},
"MARINE EQUIPMENT": {
"fra": "ÉQUIPEMENT AQUAMARIN",
"rare": "False",
"deu": "MARITIMAUSSTATTUNG"
},
"LAND ENRICHMENT SYSTEMS": {
"fra": "SYSTÈMES D'ENRICHISSEMENT DES",
"rare": "False",
"deu": "LANDANREICHERUNGSSYSTEME"
}
}
 
Last edited:

Andargor

Andargor
seeebek said:
"LAND ENRICHMENT SYSTEMS": {
"fra": "SYSTÈMES D'ENRICHISSEMENT DES",
"rare": "False",
"deu": "LANDANREICHERUNGSSYSTEME"
}
Click to expand...

Nice work! This sound wrong, though. I don't have the French version, but could it be "Systèmes d'enrichissement des terres"?
 

maddavo

maddavo
After a crash course in Python, I now have EDDN price updates coming into the Maddavo shared prices data. It is good to be LIVE! Haha.

Now I have a question about sending data. It occurs to me that it might be useful to rebroadcast data so that listeners that weren't listening at a certain time can get the data that they missed. Would it be useful to rebroadcast data? I'm thinking that maybe recent data could be rebroadcast regularly for a while and older data would be rebroadcast with less regularity. ie: something like this:

data within 3 hours - rebroadcast every 15 mins
data within 3-12 hours - rebroadcast every hour (maybe broken up into 3 x 3 hour lots every 20 mins)
data within 12-24 hours - rebroadcast every 2 hours (maybe broken up into 4 x 3 hour lots every 30mins)

in this way EDDN would be a kind of tickertape and you could rebuild the last 24hours of data by listening for 2 hours.

OR is this a bad idea? and should I just send non-EDDN received data as I get it?
 

seeebek

seeebek
I got informed by a french user that I made mistakes in the french commodities. This is the right way:
Clothing = MAQUILLAGE
Consumer Technology = VÊTEMENTS
Domestic Appliances = ÉQUIPEMENT DE LOISIR

I will update it in elite OCR for the next version.
 
Last edited:

themroc

themroc
maddavo said:
After a crash course in Python, I now have EDDN price updates coming into the Maddavo shared prices data. It is good to be LIVE! Haha.

Now I have a question about sending data. It occurs to me that it might be useful to rebroadcast data so that listeners that weren't listening at a certain time can get the data that they missed. Would it be useful to rebroadcast data? I'm thinking that maybe recent data could be rebroadcast regularly for a while and older data would be rebroadcast with less regularity. ie: something like this:

data within 3 hours - rebroadcast every 15 mins
data within 3-12 hours - rebroadcast every hour (maybe broken up into 3 x 3 hour lots every 20 mins)
data within 12-24 hours - rebroadcast every 2 hours (maybe broken up into 4 x 3 hour lots every 30mins)

in this way EDDN would be a kind of tickertape and you could rebuild the last 24hours of data by listening for 2 hours.

OR is this a bad idea? and should I just send non-EDDN received data as I get it?
Click to expand...

I believe other archive services should take care of this....
 

Snake Man

S
maddavo said:
After a crash course in Python, I now have EDDN price updates coming into the Maddavo shared prices data. It is good to be LIVE! Haha.

Now I have a question about sending data. It occurs to me that it might be useful to rebroadcast data so that listeners that weren't listening at a certain time can get the data that they missed. Would it be useful to rebroadcast data? I'm thinking that maybe recent data could be rebroadcast regularly for a while and older data would be rebroadcast with less regularity. ie: something like this:

data within 3 hours - rebroadcast every 15 mins
data within 3-12 hours - rebroadcast every hour (maybe broken up into 3 x 3 hour lots every 20 mins)
data within 12-24 hours - rebroadcast every 2 hours (maybe broken up into 4 x 3 hour lots every 30mins)

in this way EDDN would be a kind of tickertape and you could rebuild the last 24hours of data by listening for 2 hours.

OR is this a bad idea? and should I just send non-EDDN received data as I get it?
Click to expand...
Your text was all very light grey color, could not see jack.
 

Askarr

A
maddavo said:
Now I have a question about sending data. It occurs to me that it might be useful to rebroadcast data so that listeners that weren't listening at a certain time can get the data that they missed. Would it be useful to rebroadcast data? I'm thinking that maybe recent data could be rebroadcast regularly for a while and older data would be rebroadcast with less regularity.
Click to expand...
As I've posted before, I'm already providing a basic archive of EDDN data for those that wish access to missed data. Right now it's direct AWS SDK access, so you need to get a key from me. Later on (hopefully soon) it'll be a public REST API with caching so you can ask for old data without access controls. Currently distracted building wolverine's Christmas tree.

Archiving & buffering are better off handled as separate problems to be solved. Clients shouldn't re-broadcast old data ever (to the same endpoint). Reasoning:

EDDN is a firehose of data with a unique gatewayTimestamp, provided by EDDN. A rebroadcast will have a different gatewayTimestamp. This is a bad idea. It changes the fundamental notion of 'when did this piece of data cross EDDN's network?' The gatewayTimestamp is guaranteed to uniquely represent an instant in 'EDDN timeline'. Changing that would be monumentally bad for any historical analysis, load balancing, etc.

There are different needs for archiving & buffering. Are you just coping with load balance problems? Do you need days or weeks or months of data for historical analysis? How should the data be stored, indexed, queried? These are best solved at the layer dealing with the problem.

Right now, if you're handling just balancing traffic, you can use NoSQL and the gatewayTimestamp as a unique identifier to know, for certain, that this message is a unique message received at that time. If you had multiple subscribers all sending you that data, you know you can reliably filter them down to a single entry.
 
Last edited:

Snake Man

S
I agree.

Maddavo, I also have the archive beginning from 2014-12-16T12:43:00 date so you can build a base-database for TD/your site.
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
maddavo said:
After a crash course in Python, I now have EDDN price updates coming into the Maddavo shared prices data. It is good to be LIVE! Haha.

Now I have a question about sending data. It occurs to me that it might be useful to rebroadcast data so that listeners that weren't listening at a certain time can get the data that they missed. Would it be useful to rebroadcast data? I'm thinking that maybe recent data could be rebroadcast regularly for a while and older data would be rebroadcast with less regularity. ie: something like this:

data within 3 hours - rebroadcast every 15 mins
data within 3-12 hours - rebroadcast every hour (maybe broken up into 3 x 3 hour lots every 20 mins)
data within 12-24 hours - rebroadcast every 2 hours (maybe broken up into 4 x 3 hour lots every 30mins)

in this way EDDN would be a kind of tickertape and you could rebuild the last 24hours of data by listening for 2 hours.

OR is this a bad idea? and should I just send non-EDDN received data as I get it?
Click to expand...

For starters: GREAT work.
Like others have said, rebroadcasting doesn't fit in the design of EDDN. Distributing older (missed data ) is something for additional services. One service is by Askarr and hopefully many will follow. It would be great if you send the .prices uploaded by TD users to your merge tool to EDDN as well. It would of course be best ONLY to upload recent data -or better just what a commander changed if you can detect that.

Edit: Missed that Snake Man also has an archive.
 
Last edited:
Status
Thread Closed: Not open for further replies.
Back
Top Bottom