that would be so awesome!
Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.
- Thread starter wolverine2710
- Start date
- Status
that would be so awesome!
It sure would make life so much easier. At the moment I'm having to rely on 3rd party data for station names, economies etc. and I'm currently trying to teach Tesseract-OCR to read price/station/system info from screenshots.
Only data I'm interested in is data I have already come across in-game, so a xml dump of data as you enter a station, system, etc. would be extremely useful to me.
wolverine2710
Tutorial & Guide Writer
I stand corrected. What cmdr's log tool? I checked your profile but can't seem to find anything. Is it perhaps a private tool which is not publicly available?
wolverine2710
Tutorial & Guide Writer
wolverine2710 said:
I totally agree BUT its possible to reduce the amount of manual work/steps. It probably will reduce the number of (good) data send to EDDN. Let me try to explain. Lets suppose you have created an EDDN validator service and its working splendidly. Correcting names using ASP/Levenshtein and throwing away commodities based on the OCR-ed prices which are in the range they should have been (for a certain system/station/time). The checks can be simple/barebone or really complicated checks. At the end you return the prices to EDDN.
I've said it before, I'm a true believer of the Unix/Linux way. Especially the pipe mechanism. A complex (monolithic, hard to maintain) program is divided into smaller parts (programs). Each program performs an action on the input received and outputs it to the next program. The output of one program becomes the input of the next program in the pipeline. The Unix tools are in principle all barebone (not speaking about stuff like Perl and possibly awk), do a very specific task and do it extremely well. Combine them an you can create complex programs with it. I've done that in the past lost of times - record is 15+ tools in the pipeline. Its just plain fun ;-) James and myself are trying to apply the same philosophy to EDDN. EDDN is NOT a big, monolithic, hard to maintain, complex beast of a program. EDDN is small, has limited functionality, conceptually beautiful (thanks Andreas), does a simple task BUT does it well and relies on other ""add value services" to become (more) useful. Unix pipe mechanism.....
If you deem it worthwhile you could extend that to the .csv/.bpc files generated by OCR tools. You receive a file, correct it (and or ditch/remove commods or flag them as bad) and then send it back to the sender. The OCR tool before showing a screen where commanders HAVE to manually check the OCR-ed screen (EOCR/RN shows whats possibly wrong with a commodity) can use the send back data to make adjustments and directly flag what is wrong. That way less manual checks have to be done. If a validation tool is working (near) perfectly they could also decide (an option) to NOT show a commander a correction screen but simply generate a .csv/.bpc file and send the data to EDDN. The point I'm trying to make: A commander can be brilliant at OCR-ing, not so at validating stuff. A commander can be brilliant at validating, not so at OCR-ing. You get the picture. Combine the tools created by the commanders and the end result is BETTER then what they both achieve indipendently.
Disadvantage of pipes running across the internet is that when one program dies (a validator) the chain normally breaks. You and/or other validator service commanders could go one step further. If you create your validator in such a way that it is NOT part of a big monolithic programs but a small standalone program. A program which contains the logic and relies on data from your databases (for range etc). Then that program does NOT have to live/reside on your server - it can run locally on the PC of a commander. The data can be gathered from your server and cached in for example sqlite (when eddb dies the local validator still functions. You and others can take this as far as one like. OCR output is in multiple format (.csv files). Hence you have to support multiple file format. One could just say NO, I only support ONE format - ie the EDDN JSON format. When commanders/tools want to send data to the validator it has to be in that format. Other commanders could make a tool which supports multiple .csv format (and .bpc) and output a JSON structure which is supported by you. There could be separate tools for uploading to EDDN etc etc. Unix pipes. Starting/executing small programs can make things a bit slower, thats true but the advantages are great.
In an ideal world: A brilliant commander makes a superb OCR solution (better then all the rest) but can't do anything with it because (s)he is not a programmer and can't create JSON, can't upload to EDDN, can't create a .bpc/TD .prices file etc. Should that commander have a toolbox with useful stuff (s)he could still make a complete tool............ I know of commanders with OCR solutions but can't take it to the next level. The viability of the above ofc relies on the language/framework used.
I know I get carried away with this kind of stuff but perhaps one might take the above in consideration when creating tools. Tools/programs which are absolutely appreciated - even when they happen to be beasts ;-)
Last edited:
Yes it's a private tool. Not necessarily by choice but currently I don't see how anyone else could benefit from it. Here's a couple of screenshots:
So far I've only been a user of EDDN and a lot of other data collected by other Commanders (system coordinates, stations, what have you..), but hopefully if I get the OCR working with some accuracy, I might actually upload stuff as well. Of course it won't be much compared to what the other OCR tools put out but at least it's something.
EDIT: And the point of this tool of mine is to provide some missing functionality to the game: a log, a quick note-taking thing that's not pen and paper (not visible on the screenshots), a map of places you've been to and places you want to go to. It's running on a laptop next to me while I play, and it's designed to look like another panel on the game.
Last edited:
wolverine2710
Tutorial & Guide Writer
The map view looks nice. Was there anything missing in the rest of the tools which caused you to create your own tool?
Curious about your OCR tool and the results. I'm probably telling you nothing new but names can be checked against a dictionary (levenshtein distance/ASP) to correct them. Sometimes it does not because two commods are to familiar. Wrt numbers, you might want to have a look at "EliteBrainerous: the standalone "incredible neural OCR engine" powering EliteOCR". It does numbers only. From the thread: As my long term goal is to have this be an all-round solution for anyone working on integrating OCR into their EliteDangerous app, it's spun off into it's own project!. With default HUD colors and a resolution of 1920x1080 he has achieved > 99.99% accuracy. He's using machine learning (neural networks). Not sure about the language, could be Java. I know it can also be used from within other languages. In fact RegulatedNoise (written in a different language then EB) written by maxh2003 is doing exactly that. I believe EOCR and RN both use Tesseract for OCR-ing the strings/names.
Last edited:
wolverine2710
Tutorial & Guide Writer
Has anyone created a subscribe client for EDDN using VBscript. If so I know of a commander who would love to see your code!!
Out of curiosity: What languages are the current EDDN subscribers using to get data for their own tools?
I would love to see a collection of simple subscriber clients (in as much different languages as possible) which could be used by other potential tool authors. It gives them a head start so they only have to concentrate on building their own tool instead of getting (possibly/potentially) frustrated in getting zeroMQ working ;-)
Out of curiosity: What languages are the current EDDN subscribers using to get data for their own tools?
I would love to see a collection of simple subscriber clients (in as much different languages as possible) which could be used by other potential tool authors. It gives them a head start so they only have to concentrate on building their own tool instead of getting (possibly/potentially) frustrated in getting zeroMQ working ;-)
Last edited:
I guess I figured I'd never find a tool that had all the functionality I wanted (and might want in the future), so it proved easier to make my own. Plus I was away from home for a couple of weeks (and therefore couldn't play the game) so it gave me something to do
I was thinking of trying the .traineddata files from EliteBraineous as a starting point (hopefully save me a lot of work), and maybe use the data provided in tradedangerous's Item.csv and Category.csv to check the validity of the results. It's the numbers that were going to be the problem anyway but if zxctypo's code is that accurate it gives me hope this isn't a futile project.
I think one of the great things about the various efforts here is that even if FD provide an XML export, it will still require some sort of tool to upload & some sort of crowd aggregator to be truly useful to trading tools. In that regard I consider EDDN to be an idea that survives a significant number of changes to Elite. I also think a lot of interesting projects will come out of having the sheer volume of data available over time (which is why my current operation simply stores every piece of valid JSON coming through). Right now most tools are focused on the immediate 'how do I make money in the next few hours' - if there's enough data coverage over time, I could envisage being able to trend prices, observe commander behaviour, see what commodities are in flux, etc. You often don't know what patterns and insights you'll discover until you look at the data.
On languages: my little projects so far have been in Python because I find that the most efficient* language for me to put together simple test code and then polish it up to something I can run on a server without crashing out (without feeling like I'm doing my day job).
Two of the reasons I chose my current dump of EDDN data to be in AWS is that it's easy & cheap, and that they have great SDKs (that I don't have to write) in a significant number of languages; access control is straightforward and I can create a key & secret I can hand out to any commander interested that has purely read-only access to that DB.
* Efficient, that is, in development. Usually several hours lost in finding out which particular package, pip, egg, easy_install, compiler voodoo I have to pull to get some module needed by another module, but hey, that's Python
On languages: my little projects so far have been in Python because I find that the most efficient* language for me to put together simple test code and then polish it up to something I can run on a server without crashing out (without feeling like I'm doing my day job).
Two of the reasons I chose my current dump of EDDN data to be in AWS is that it's easy & cheap, and that they have great SDKs (that I don't have to write) in a significant number of languages; access control is straightforward and I can create a key & secret I can hand out to any commander interested that has purely read-only access to that DB.
* Efficient, that is, in development. Usually several hours lost in finding out which particular package, pip, egg, easy_install, compiler voodoo I have to pull to get some module needed by another module, but hey, that's Python
Last edited:
wolverine2710
Tutorial & Guide Writer
I think one of the great things about the various efforts here is that even if FD provide an XML export, it will still require some sort of tool to upload & some sort of crowd aggregator to be truly useful to trading tools. In that regard I consider EDDN to be an idea that survives a significant number of changes to Elite. I also think a lot of interesting projects will come out of having the sheer volume of data available over time (which is why my current operation simply stores every piece of valid JSON coming through). Right now most tools are focused on the immediate 'how do I make money in the next few hours' - if there's enough data coverage over time, I could envisage being able to trend prices, observe commander behaviour, see what commodities are in flux, etc. You often don't know what patterns and insights you'll discover until you look at the data.
On languages: my little projects so far have been in Python because I find that the most efficient* language for me to put together simple test code and then polish it up to something I can run on a server without crashing out (without feeling like I'm doing my day job).
Two of the reasons I chose my current dump of EDDN data to be in AWS is that it's easy & cheap, and that they have great SDKs (that I don't have to write) in a significant number of languages; access control is straightforward and I can create a key & secret I can hand out to any commander interested that has purely read-only access to that DB.
* Efficient, that is, in development. Usually several hours lost in finding out which particular package, pip, egg, easy_install, compiler voodoo I have to pull to get some module needed by another module, but hey, that's Python
Nice to see you like what was initially Andreas (EMDN) idea - which was based on Eve's EMDR. I've just taken the liberty to try to revive it and with with the excellent work of implementer James this has been succesful. It has taken me by surprise to see how many commanders are actually creating tools for EDDN. Especially given the fact EDDN is atm a toddler and things can and will change. There are far more commanders setting up tools (private/public) then with EMDN (that I'm ware of). So I guess there is a need for EDDN. In the future when Thrudd, TD and Slopey will also upload to EDDN there will be a lot more data ;-) Luckily EOCR and RN are already outputting their OCR-ed results to EDDN.
I DO agree with "if there's enough data coverage over time, I could envisage being able to trend prices, observe commander behaviour, see what commodities are in flux, etc. You often don't know what patterns and insights you'll discover until you look at the data". I'm hoping for quite a lot of variety in the new forthcoming tools.
Wrt tools: Hopefully tomorrow I have time to go through the whole thread and determine what tools are out there atm and create an overview of it. Makes life easier for commanders new to the EDDN party ;-) Personally an ELK stack on the EDDN server is my goal for the near future. That way I get insight in the amount of data, number of uploads, bandwidth usage etc. Kibana is ideal for that - it seems....
Note: I make a living as a Java software engineer but I'm learn Python - just for EDDN and TD.
Edit: " I can create a key & secret I can hand out to any commander interested that has purely read-only access to that DB". THAT is VERY interesting and generous!!!
Last edited:
Hi all, this is for me. @wolverine2710 thanks for posting on my behalf. There is a zmq binding for VB, but VB is different from VBScript. If there is a binding that works with VBScript then I can get going very quickly. If you can help me that would be great. Otherwise, I also need help with the following...
Looking at some forums about zmq performance I get the feeling that Python is good. I need to learn Python and I had a look at @jamesremuscat's example code. I can't get it to work. I have Python 3.4.1 and it was written for 2.7 . I have other things I use that need the version of Python I have. Has anyone got any EDDN example code that works with Python 3?
wolverine2710
Tutorial & Guide Writer
You can have multiple Python versions installed. Normally one is set in your PATH. You can select a specific version by using the Absolute path for the python exe. Hence you run a python script with either Python 2.7.x or 3.4.x. I hope this helps a bit....
Last edited:
I'm using C# and NetMQ (as opposed to ZeroMQ).
There are issues with silent time-outs and disconnects.
To overcome this I periodically reconnect every 5 minutes.
On related topics ...
Data scrubbing should be the primary responsibility of the OCR tool.
Data scrubbing should be the secondary responsibility of the receiving client.
Allowing a client or even a special client to rebroadcast data seems like a bad idea architecturally. It's begging for "email bomb" sort of problems where 2 scrubbers keep sending each other scrubbed data back and forth which piles up as more POST come through until the pipe is full and behind the capacity of any end-client-node.
I am not impressed with 0MQ. It's a massive load of complexity to solve a rather simple problem and at best the C# implementations are buggy, at worst the underlying architecture is invalid.
Making the commodity the first order object is a design flaw that needs to be corrected to improve EDDN services. I think the first-order object should be the station. A complete set of commodities is expected and this allows additional station information to be included. Right now there's a ton of redundant data sent with the station name, dates, et. al. sent over and over again for each commodity. Each station update sent out should be accompanied by a generated GUID (or UUID) that can be used to track that transaction. Another message on the EDDN should be to declare that GUID of data no-good. Broadcast that data to the clients and if the client gets enough votes it can purge the bad data associated with that GUID.
Those are the improvements I need from EDDN to make QAA better. Data scrubbing is SEP - someone else's problem.
Data scrubbing should be the primary responsibility of the OCR tool.
Data scrubbing should be the secondary responsibility of the receiving client.
Allowing a client or even a special client to rebroadcast data seems like a bad idea architecturally. It's begging for "email bomb" sort of problems where 2 scrubbers keep sending each other scrubbed data back and forth which piles up as more POST come through until the pipe is full and behind the capacity of any end-client-node.
I am not impressed with 0MQ. It's a massive load of complexity to solve a rather simple problem and at best the C# implementations are buggy, at worst the underlying architecture is invalid.
Making the commodity the first order object is a design flaw that needs to be corrected to improve EDDN services. I think the first-order object should be the station. A complete set of commodities is expected and this allows additional station information to be included. Right now there's a ton of redundant data sent with the station name, dates, et. al. sent over and over again for each commodity. Each station update sent out should be accompanied by a generated GUID (or UUID) that can be used to track that transaction. Another message on the EDDN should be to declare that GUID of data no-good. Broadcast that data to the clients and if the client gets enough votes it can purge the bad data associated with that GUID.
Those are the improvements I need from EDDN to make QAA better. Data scrubbing is SEP - someone else's problem.
I'm using C# and NetMQ, though not doing much with the data yet.
The code is currently mostly in MainWindowViewModel.cs though it will move out of there in the future. Anyone that needs it is free to use it.
As others have said, you can use them side by side with the right config. I haven't seen any 0MQ binding for Python 3 as I suspect it'd need a rewrite of how the eventing is working under the hood (see gevent & Python 3).
Look into virtualenv. I am still getting the hang of it myself but it's quite a clever idea for separation of packages & concerns. It's a bit like containers/Docker for Python.
I hadn't actually heard of EMDN, but then I really only started paying attention to the Elite forums way back at Kickstarter and then later closer to launch so I probably just missed it.
I think this is one of those things where 'if you build it, they will come'. Heck, I'm playing around with it just to teach myself some new tricks & play with some big data that isn't loaded with non-disclosure agreements. I liked your idea about a 3D real time map. Looking into that now.
No worries. Right now, I envisage a few commanders wanting data, and one of the great things about AWS's design is that it's a managed Cassandra system I don't have to maintain with a bunch of high level SDKs on top. They handle throttling access and access control for me, and it's tiny money compared to running a whole bunch of nodes myself. For example, they tend to charge for data transfer once it's hitting 1GB per month. Current EDDN data from December is <100MB total. As long as people don't download it all every day, costs should be fine.
Another great feature and why I'm prepared to make it available is that AWS essentially say to any read access 'the owner of the DB is paying for X reads a second. Your reads are too fast, slow down, and we'll give you the data more slowly'. So as long as people don't re-request the entire DB a lot & the data transfer costs don't explode, I'm happy to make it available for access.
Edit: I should emphasise it's built for historical queries, i.e. the indexing & querying is designed around 'I want to find out what EDDN data happened between X & Y times (even if Y is Now)' - I'm anticipating people will pull data down and store it in their own systems for more complex queries. I'm aiming just to keep a record of all EDDN traffic at this time.
At some point I'll probably wrap it with an API and caching layer so the data will be genuinely freely available, but one thing at a time
Last edited:
Well, I have some progress...
I managed to get a listener working with Python 3.4.1 and connected to EDDN and parsed the data to something usable for me:
woot :-D
Still alot to learn and lots of head scratching, but at least some progress.
I managed to get a listener working with Python 3.4.1 and connected to EDDN and parsed the data to something usable for me:
Code:
C:\>listener.py
@Korro Kung/Lonchakov Orbital
# ERROR: Demand and Stock are 0.
Hydrogen Fuel 90 94 ? 135802? 2015-01-15 05:02:37
# ERROR: Demand and Stock are 0. Kdro Kung Pellets
Clothing 221 239 ? 3129? 2015-01-15 05:02:37
Consumer Technology 7561 0 5145? - 2015-01-15 05:02:37
Domestic Appliances 418 439 ? 1876? 2015-01-15 05:02:37
Algae 268 0 97185? - 2015-01-15 05:02:37
Animal Meat 1647 0 6647? - 2015-01-15 05:02:37
Coffee 1647 0 5004? - 2015-01-15 05:02:37
Fish 588 0 25572? - 2015-01-15 05:02:37
Food Cartridges 40 52 ? 5101? 2015-01-15 05:02:37
Fruit And Vegetables 478 0 18056? - 2015-01-15 05:02:37
Grain 345 0 70199? - 2015-01-15 05:02:37
# ERROR: Buy less than Sell. Mdkdjing Beast Feast
Synthetic Meat 400 0 11659? - 2015-01-15 05:02:37
Tea 1848 0 6788? - 2015-01-15 05:02:37
Polymers 277 0 683294? - 2015-01-15 05:02:45
Semiconductors 1168 0 92477? - 2015-01-15 05:02:45
Superconductors 6615 0 4792? - 2015-01-15 05:02:45
Beer 307 0 45377? - 2015-01-15 05:02:45
Liquor 517 543 ? 25? 2015-01-15 05:02:45
Wine 400 0 34863? - 2015-01-15 05:02:45
Atmospheric Processors 256 270 ? 86420? 2015-01-15 05:02:45
Crop Harvesters 1836 1870 ? 308132? 2015-01-15 05:02:45
Marine Equipment 3699 3765 ? 184496? 2015-01-15 05:02:45
Mineral Extractors 438 460 ? 126779? 2015-01-15 05:02:45
Power Generators 418 439 ? 5634? 2015-01-15 05:02:45
# ERROR: Buy less than Sell. Volkhab See Drdnes
Water Purifiers 170 183 ? 878158? 2015-01-15 05:02:45
Basic Medicines 189 203 ? 4233? 2015-01-15 05:02:45
Performance Enhancers 7561 0 5207? - 2015-01-15 05:02:52
Progenitor Cells 7561 0 1386? - 2015-01-15 05:02:52
Aluminium 495 0 1338040? - 2015-01-15 05:02:52
Beryllium 9127 0 54085? - 2015-01-15 05:02:52
Cobalt 918 0 69805? - 2015-01-15 05:02:52
Copper 648 0 402339? - 2015-01-15 05:02:52
Gallium 5862 0 156674? - 2015-01-15 05:02:52
Gold 10383 0 98324? - 2015-01-15 05:02:52
Indium 6436 0 23312? - 2015-01-15 05:02:52
Lithium 1908 0 144385? - 2015-01-15 05:02:52
Palladium 14339 0 76514? - 2015-01-15 05:02:52
Platinum 19851 0 5751? - 2015-01-15 05:02:52
Silver 5473 0 160121? - 2015-01-15 05:02:52
Tantalum 4500 0 89867? - 2015-01-15 05:02:52
Titanium 1298 0 548991? - 2015-01-15 05:02:52
Uranium 3155 0 394336? - 2015-01-15 05:02:52
Auto-Fabricators 4296 0 20607? - 2015-01-15 05:02:55
Computer Components 356 375 ? 33386? 2015-01-15 05:02:55
H.E. Suits 428 0 293283? - 2015-01-15 05:02:55
Robotics 2203 0 27675? - 2015-01-15 05:02:55
Leather 307 0 723144? - 2015-01-15 05:02:55
Natural Fabrics 588 0 129870? - 2015-01-15 05:02:55
Synthetic Fabrics 315 0 477326? - 2015-01-15 05:02:55
Biowaste 11 14 ? 14438? 2015-01-15 05:02:55
Scrap 27 31 ? 155548? 2015-01-15 05:02:55
Non-Lethal Weapons 2203 0 957? - 2015-01-15 05:02:55
Reactive Armour 2481 0 726? - 2015-01-15 05:02:55woot :-D
Still alot to learn and lots of head scratching, but at least some progress.
The worst part is that system/station names are wrong in 1%... And how to differ between "wrong written" and "new system/station" ....
As you grow your tool, you can "lock" systems, so as not to accept new station names. I would hope we could get some sort of database with a api (like edsc) for stations with with a confidence rating on station names. In my own "notes" I just not accept new stations in systems I've visited. Because those are confirmed by my own two eyes. For all other I just accept what stations come my way and housekeeping happens when I visit them.
- Status