Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

Status
Thread Closed: Not open for further replies.

wolverine2710

Tutorial & Guide Writer
Incase it helps anyone get a leg up, I've pushed a very small and quick vagrant+ansible project to github.com/theangryangel/elite-dangerous-eddn-elasticsearch-kibana-vagrant that will setup a provision a VM to collect data from EDDN, shove it into Elasticsearch, provide a Kibana 4 frontend and automatic culling of data older than 30 days.

Although I have put minimal instructions in the repo, consider it sold and seen with no support, I'm afraid.

Couple of caveats;
I'm a sysadmin, not a programmer. Code quality may vary.
I would *not* run this on any public facing without modifying the ansible playbook. There are serious issues, such as it pulling a module or two directly from source, a lack of log rotation for a couple of components.

I rarely visit the forums, but I just want to thank everyone involved. I've had some great fun with this the last evening or two, pulling in data and analysing it.

If the EDDN does continue, and grows past it's current in carnation, please get in touch - I may be able to provide some resources through work - depending on what's required :)

Commanders I need HELP. I said from the start that I wanted to have an ELK-stack on the EDDN server. As I'm new to ELK I thought it would be a good idea NOT to install it directly on the server but to experiment with it on my own windows machine. The above setup does exactly that. Problem is that ansible is NOT officially support for windows but I found this excellent article of how to set it up. Works like a charm. Doesn't take long to set it up. Note: Iirc for the ansible stuff it looks as if you must NOT use a path with a space in it.

Everything went smooth till it was time that ansible should do its thing. As in installing the ELK stack. The problem is NOT related to ansible but to SSH it seems, which borks ansible.

The error I get:
Code:
GATHERING FACTS ***************************************************************
fatal: [default] => private_key_file (C:/Data/vagrant/kibana/.vagrant/machines/default/virtualbox/private_key) is group-readable or world-readable and thus insecure - you will probably get an SSH failure

TASK: [Add wheezy-backports repo] *********************************************
FATAL: no hosts matched or all hosts have already failed -- aborting

The solution seems simple enough. Just use chmod -c 600 to remove the read flags for group and other (Linux). The problem is just that cygwin (babun actually) under windows does not do that. Also tried with gitbash - some problem.

Either I must find a way to remove the read flags for group and other OR somehow I must convince openssh that read permission for group and other is OK. I've been at it the whole day but can't get past that openssh error.

If someone has experience with vagrant and the ansible UNDER windows I would really appreciate some help. Like I said it boils down to openssh. Perhaps someone is willing to try to setup the ELK stack under windows. If not I can understand.

When all fails I'm gonna set it up directly on the EDDN server using the normal ELK stack instructions and use the json ELK template and collector.py from the_angry_angel setup. BUT I thought the ELK stack under windows would be great for other commanders as well. Fingers crossed.
 
Last edited:

wolverine2710

Tutorial & Guide Writer
Why not set it up in a VM. VirtualBox is easy to use and with Win8 pro you get HyperV that absolutely rocks!

You mean running a Linux VM (under windows) and then running another Linux VM (with the ELK stack) in that VM?
What I'm currently doing is running VirtualBox under windows and running Linux in it. Problem is setting the VM up - which reaquires vagrant (easy) and ansible to install it properly. That last part is giving my a headache. The openssh part.
 
Last edited:
You mean running a Linux VM (under windows) and then running another Linux VM (with the ELK stack) in that VM?
What I'm currently doing is running VirtualBox under windows and running Linux in it. Problem is setting the VM up - which reaquires vagrant (easy) and ansible to install it properly. That last part is giving my a headache. The openssh part.

Can you explain what you are trying to do exactly with ssh? I use putty as a shell to localhost when I'm on Windows/Cygwin, and have never used babun.
 

wolverine2710

Tutorial & Guide Writer
Can you explain what you are trying to do exactly with ssh? I use putty as a shell to localhost when I'm on Windows/Cygwin, and have never used babun.

I wasn't clear it seems. Normally I write to much, was trying to be concise this time. Post will be bigger this time.

Have a look at the description of the_angry_angels tool. It allows you to understand the beneath better.
I've installed virtualbox without any VM on it. Normally you either install Linux/windows/BSD on it from scratch or you download a virtualbox VM with for example ubuntu. There are better ways. One is vagrant - its independent of the VM software you use. You run vagrant with a vagrantfile. In it you state which ready made VM you want to use. When issuing the command "vagrant up" it downloads it automatically. You can also tell vagrant to afterwards run scripts to install additional software and/or configure the VM - with so called Provisioners. A provisioner can be the Linux shell or specialized frameworks. One such framework is the open source framework ansible. In both cases afaik vagrant uses ssh to contact to the running VM and it issues the commands to install extra software with for example yum/apt. Vagrant (for windows) itself works fine. Ansible is not officially supported for windows. But you can get it running under cygwin/babun - link in initial post. Both give you basically the complete collection of Linux tools under windows. Afaik in the case of anisble openssh from cygwin/babun is used to connect to the VM. With vagrant I'm not sure, probably also openssh but then compiled for windows and part of vagrant. In both cases the same ssh private-key file is used. It created under windows by vagrant. When you look at the file with ls -l you see -rw-r--r-- for the permissions. Problem is that openssh is very picky and it should be -rw-------. With chown you normally change that. cygwin/babun,gitbash says its changed but in reality its NOT changed and remains -rw-r--r--. I need to either be able to change the perms or find a way to tell openshh to back off, play nice and accept the fact that the privat_key file is readable by group/others. Insecure of course but have no other option....

I hope it a bit clearer now why I need ssh. NOT for logging into the VM BUT for the ansible framework to log in on the running VM to do its magic.

Note1: with "vagrant ssh" you autimatically login with ssh on the running VM. One could use putty for it as well but for that you have to do some extra things. Not tried that yet.
Note2: babun is very cool. Its basically cygwin but more user-friendly, especially the pact package manager. worth a try.

@graham_reeds. I only need one Linux VM in virtualbox. The VM in a VM came from your suggestion to use a VM. In my case that would mean a Linux VM inside a Linux VM ;-(
 
Last edited:
ZeroMQ only provides one delivery guarantee - that any message promoted out of it's stack will be a complete message.

First; thank you for taking the time to setup and run such a test.

Second; OMG.

I have a few more things to button up with QAA and this will be at the top of list within a week or so.

I started on a banking application a long time ago. One of the things I had to figure out how to do with it is keep a transactional record and send the client what's new/changed since their last update.

So my plan is the server stores all the updates. When you connect you will tell the server the date-time stamp of the last time you (dis)connected and it will catch you up. While you are connected you receive updates.
 
Last edited:
I hope it a bit clearer now why I need ssh. NOT for logging into the VM BUT for the ansible framework to log in on the running VM to do its magic.

Hmm, sounds very specific, and I am not familiar with that kind of environment.

I did find a stackoverflow reference to something like this, look at the 3rd answer about Windows 8.1: http://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error

Note that it seems to be localization-specific, so if your are in German, it would be something other than Users.


This seems to be related, but I am unsure if they are helpful since I'm not familiar with ansible and vagrant: https://github.com/ansible/ansible/issues/9836

HTH, otherwise idk :)
 

wolverine2710

Tutorial & Guide Writer
Hmm, sounds very specific, and I am not familiar with that kind of environment.

I did find a stackoverflow reference to something like this, look at the 3rd answer about Windows 8.1: http://stackoverflow.com/questions/9270734/ssh-permissions-are-too-open-error

Note that it seems to be localization-specific, so if your are in German, it would be something other than Users.


This seems to be related, but I am unsure if they are helpful since I'm not familiar with ansible and vagrant: https://github.com/ansible/ansible/issues/9836

HTH, otherwise idk :)

Your SIR are a devil and a gentleman.

The first link (and many similar links) I did find - but didn't solve my problem. The last link I didn't find. Not enough Googlefu powers. THAT however did the trick. After commenting out the check and the exception raised in connection.py (of Ansible) ansible started installing all kind of stuff. I still DO get an error but that is because of the tool pleaserun which seems to be needing a newer version of Ruby. From the pleaserun git: "Pleaserun is a tool to generate startup scripts for the wasteland of sorrow that is process launchers." Commenting that out as well for 3+ occasions made the ansible.yml script continue but almost at the end ansible.yml halted because Kibana isn't running. That task would be done using Pleaserun.

I'm going to see how I can get Pleaserun running (updating Ruby etc) and hopefully then the whole ansible script DOES run without errors. Luckily vagrant allows for things like: vagrant up (starting and running if needed the provion scripts), vagrant halt (stop machine) AND vagrant destroy. This throws away the whole VM. Later issuing a vagrant up installs the VM again and then runs the provision script again. Ideal for testing.

Again: Thank you so much for this information. After I get Pleaserun toilet trained I'm hopeful that the whole ELK stack will run inside the VM. When I get the whole thing running I will report back with instructions of how to get it working - for other commanders. I will create an issue on the bug-tracker of the tool and hopefully the repo gets updated with the new info.

Signing off: One happy camper!!!
 
Last edited:
54676715.jpg
 
In case you miss the message from eddb: The JSON dump url has changed. The old files will be deleted after two weeks. Read more here...
 

wolverine2710

Tutorial & Guide Writer
Status ELK stack using the_angry_angel setup

I have it working. Kibana after 10 minutes shows 560+ messages have been collected. I had to massage the lot (ansible.ymll wrt Pleasrun) a bit to get it working though. Will install a new VM with vagrant/ansible and massage it so that it WILL install correctly under windows in one go.

Next steps:
  • Watching Andreas ELK/kibina video and other Kibana info.
  • Get info from the archiving services by Askarr and Themroc to fill the NoSQl database ElasticSearch database.
  • Delving into Python to achieve the above
  • Installing the lot manually or perhaps with ansible on the EDDN server.

Forgot how much fun Linux, programming can be OUTSIDE office hours ;-)
 
Is there a list or table somewhere of german/french/whatever language Items and the English equivalent?
In EliteOCR there is a file commodities.json. There are all non-rare commodities in english, german and french. Please regard the french translation done by FD is wrong in some points. The commodities.json file has them as they are in game.
This link allows you to download only the source 1.2MB instead of the compiled 60MB
http://sourceforge.net/projects/eliteocr/files/0.5 source/EliteOCR.0.5.source.zip/download
 
In EliteOCR there is a file commodities.json. There are all non-rare commodities in english, german and french. Please regard the french translation done by FD is wrong in some points. The commodities.json file has them as they are in game.
This link allows you to download only the source 1.2MB instead of the compiled 60MB
http://sourceforge.net/projects/eliteocr/files/0.5 source/EliteOCR.0.5.source.zip/download


Thanks seebek! And so the german data that came across EDDN recently - was that from EliteOCR (I didn't capture the headers at the time)? Surely it would transmit to EDDN in English. It must have come from something else.
 
Thanks seebek! And so the german data that came across EDDN recently - was that from EliteOCR (I didn't capture the headers at the time)? Surely it would transmit to EDDN in English. It must have come from something else.
If it came from EliteOCR then only from version 0.3.8. After that EliteOCR automatically translates to english for EDDN.
 
Last edited:
Superb explanation. Thank you. I did some digging and came across this post from one of the contributors to libzmq:

I'm guessing that's Pieter, it reads like him.

I would conclude that the publisher being run for EDDN should indeed send keep-alive messages, and it's our job as subscribers to reconnect some time after that keep-alive period has elapsed.

Option 2 only works in infra situations where you know the subscriber is not behind a firewall/nat/etc.

From that message thread, it seems that there's really two problems - if the publisher is rebooted (or equivalent loss of connection), the subscriber has no direct way to know that it has to re-establish the connection - if the subscriber is rebooted, the publisher has no idea that the connection it's publishing to is dead. The latter means the publisher is sending traffic to a dead connection & is less of a big deal beyond network traffic. The former means subscribers lose data - I'd say that's quite a problem.

Bear in mind that the people on the list are generally working in more ideal situations, very few of them are servicing end-users, the first time EDDN is taken out by someone running an old version of ZMQ, the mailing list will point out it's the users' responsibility to keep the library up to date. Likewise with API versions.

The point here being that a secondary problem is queue-backlog at the server; although it's transmitting packets on individual sockets, data that has still to be sent on one or more connections is still buffered in ZeroMQ.

While we're on the subject, the other way a subscriber can lose data is if it fails to receive data quickly enough. If the High Water Mark of the queue for the publisher for that subscriber is reached, it will discard messages. Therefore the other thing a subscriber must do is process incoming messages as fast as possible. Do no long-blocking work at all that prevents you from calling recv() again as soon as possible (or equivalent Tornado ioloop callback).

Yep, that's why I do batching with poll for timeout and recv(NOBLOCK) for draining.
 
Elite History Server

For those looking for historical EDDN data, I have built a cache system and a new protocol to deal with the problem of sending commodities one at a time. This system is based on protocol buffer, which in testing reduced the bandwidth taken by the messages from about 240 bytes zlib compressed JSON to around 95 bytes uncompressed protocol buffers. In addition, this system supports batching. When you need data, you call the endpoint with the last Unix timestamp of data you receive and it will send you everything since then. The code for the server is located at https://github.com/huadianz/elite-eddn-history and includes the protocol buffer schemas you need to receive the data. In addition, it also contains an example program that shows you how to unpack the protocol buffer binary stream.

The current server I am running is located at http://elite.servers.henryzh.com/api/v0/update/<timestamp>
The API will send you up to the last 1000 missed updates with Unix timestamps in MS >= the timestamp in the system. If you need more you will need to query the server again.

As this project is open source I welcome any suggestions or if you have any code you want to commit, that would be great. As for already planned features:
  • Update the network using batched commodity updates
  • Current status server (query the status of a station in its most recent state)
  • Timeseries server with influxdb
  • Using commodity IDs instead of names, if we get those from Frontier at any point
 
Last edited:
Status
Thread Closed: Not open for further replies.
Top Bottom