Release EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

[wolverine2710]

I used Slopey's tool during early beta days, and thought it brilliant. However, I've been playing w/o trading tools in B3 and have learned a heckuva lot more about trading. Moreover, I'm enjoying trading much more - actual skill/thoughtful planning is required now. So currently I do not plan on using any trading tools come launch.

That does not negate the OP's question, however. My concern will be that all these tools rely on manual user input with no verification. Doesn't that open the back end/data up for tons of manipulation by those who find great routes? Say someone one finds they can buy a product for 1,000 at location A, and sell for 2,000 at B. What's to prevent them from overriding prices of A to 2,000 and keeping the profit all to themselves? Sure, there's some safety in numbers (i.e. maybe another commander will catch the impropriety and correct it), but isn't the system open to manipulation, gamesmanship - and of course untimely data?

Poisining the well is indeed a concern. Movie quote Armageddon: It happend before, it will happen again. Slopey in the beginning had pranksters which inserted Java cakes into the database for example. Then BPC was manual input based. When EDDN is filled by for example Slopey´s BPC and Thrudds website poisining could be detected and stopped there. Thrudds user have to create an account before they can upload data. Slopey is I believeve going the same way. When fraude is detected those users can be banned. EDDN will also be filled by TD of kfsone and with results coming from OCR - when that succeeds. How to stop them? Don´t know if authentication can be done with ZeroMQ - checking. If not data could be uploaded to EDDN with REST or using a REST proxy. For this you can implement authentication with for example OAuthication. Multiple other ways for data validity can be thought of. Does a certian IP within one minute upload data for 3 stations, unlikely he was able to fly there in that time --> ignore that data etc etc. Hopefully next year there will an FD API or XMLfile created by ED which has price info, so you have safety by numbers. We have to see what measuremens are needed.

 

[Snuble]

Maybe the various tools should have the option to send "kill data". Lets say Slopey discover that some data is poisoned, bans the user, he should also be able to send info about data beeing dirty and this should be forwared to the firehose allowing every other tool to delete that data.

 

[wolverine2710]

The reason I ask is that for getting data I like the ZeroMQ method of getting data pushed to me, however I dont want to have a process constantly running to put data to EDDN. Also I dont want to set up that system every time I submit data. It would be much more convenient for me to just POST that data every time I update my system. Also I think EMDN posted line by line, I would very much want to post all data for a station when I visit because I will bulk submit to my own system (this is where a constant connection could help), but also have the chance to update line by line (for example thrudds tool will update every field when change focus, here he would have to chose if want to actually post that data for every update, every line would get posted multiple times as user change data).

Is that a private (not publicly released) system your using? Which is of course fine. Wrt `running process constantly´. When you normally would create a POST at that moment you instead use ZeroMQ to push data to eddn - its a not process which needs to be running constantly. Don´t know how marketdump posted its data - that source has not been released. The way I see it, you send a JSON structure to EDDN. That JSON structure can have data for 1 commodity or for all commodities. Depends on what the author of a client decides - for example only commodities for which the prices changed. Hope this clarifies it a bit.

- - - - - Additional Content Posted / Auto Merge - - - - -

Maybe the various tools should have the option to send "kill data". Lets say Slopey discover that some data is poisoned, bans the user, he should also be able to send info about data beeing dirty and this should be forwared to the firehose allowing every other tool to delete that data.

Not sure how feasible that would be for a client but it would not be a problem for EDDN. It just receivea JSON structure with a kill data flag/field which is send to a client which has to deal with removing data.

 

[jamesremuscat]

I'm at work so probably shouldn't spend too long drafting a response on these forums but the upload of data for EMDR/EMDN was (and therefore EDDN will be) an HTTP POST to a gateway, which deals with pushing the data across the ZeroMQ system for you.

Uploader-users don't need to worry about ZeroMQ.

 

[wolverine2710]

I'm at work so probably shouldn't spend too long drafting a response on these forums but the upload of data for EMDR/EMDN was (and therefore EDDN will be) an HTTP POST to a gateway, which deals with pushing the data across the ZeroMQ system for you.

Uploader-users don't need to worry about ZeroMQ.

And the conclusion is, I was wrong about uploading to EDDN. I should have RTFM-ed better. I apologize for the confusion I created and time lost.
Lesson learned. Good that I was wrong as it makes lives better/easier for the authors of clients. Now where it that cave I shoudl crawl back into....

 

[codersparks]

I (like others) plan on creating a Captain's Log type application and would like to be able to share the information that I find with others using the crowd sourcing approach discussed here.

I know I am late to the table, but I would be keen to get involved in this project. At the moment I can only commit a small amount of time (probably a couple of hours a week), but that should increase in a few weeks.

I notice that it has all gone quiet on this thread, is it being discussed elsewhere (I have searched but can't find anything) or has it all stopped?

 

[Gaspode]

I'm also interested - is this live / being discussed elsewhere ?

G.

 

[jamesremuscat]

Still a work in progress - haven't had as much time to devote to it as I've needed, but a proof-of-concept test system should be being announced at some point this week. codersparks - what sort of data would you be looking to share via your Captain's Log?

 

[wolverine2710]

I've been PM-ed by an author that (s)he has a POC for OCR-ing the commodities market. That would be a great source of data for EDDN!!

@jamesremuscat. Haven't had time to look into EDDN. Somehow I miss some time/hours each day ;-(

 

[Ishvii]

Brilliant idea - best of luck with it.

 

[wolverine2710]

Brilliant idea - best of luck with it.

To be honest, the original idea came from Andreas who created EMDN and marketdump. He did shutdown both tools when FD changed their data access policy. Marketdump was scraping ED for prices. EMDN in itself was completely legal but sadly was shutdown as well. At that time the only supplier of data for EMDN was marketdump - which after the policy change did illegal stuff.

 

[wolverine2710]

I (like others) plan on creating a Captain's Log type application and would like to be able to share the information that I find with others using the crowd sourcing approach discussed here.

I know I am late to the table, but I would be keen to get involved in this project. At the moment I can only commit a small amount of time (probably a couple of hours a week), but that should increase in a few weeks.

I notice that it has all gone quiet on this thread, is it being discussed elsewhere (I have searched but can't find anything) or has it all stopped?

I was MIA for a week and will be MIA again starting Thursday till wednesday next week. This time a small holiday. Hence my thread didn't get the love from me it deserves ;-)

COOL. EDDN is meant for dynamic data like commodities market prices and other dynamic data. For static data there is TGC - The great collector. Its a web-api. This was created by TornSoul for the crowd sourcing coordinates project. It currently stores distances and coordinates but will be extended for I believe all other static data as well - for example distance from a star to a station/platform etc. Would your captain's log tool by a windows tool which can read the netlog logfiles to get the current system a commander is in. If so, please consider linking it to TGC. Its very important for the project to know which star systems are out there, so we can crowd source the distances for it.

 

[codersparks]

codersparks - what sort of data would you be looking to share via your Captain's Log?

My initial thoughts were to just track my trades so that I can see where I have been the most profitable, that soon evolved into tracking buy/sell prices at different stations aswell.

As part of my app I would like to subscribe to the EDDN and as I want to consume the data, I feel its only right to share my data. Therefore, I would be willing to contribute any data that I can to help others, be that system information, commodity prices whatever.

I know there are other tools out there but I am looking for a project to do in my spare time. If I can use some of that time on a community project all the better.

 

[codersparks]

codersparks - what sort of data would you be looking to share via your Captain's Log?

My initial thoughts were to just track my trades so that I can see where I have been the most profitable, that soon evolved into tracking buy/sell prices at different stations aswell.

As part of my app I would like to subscribe to the EDDN and as I want to consume the data, I feel its only right to share my data. Therefore, I would be willing to contribute any data that I can to help others, be that system information, commodity prices whatever.

I know there are other tools out there but I am looking for a project to do in my spare time. If I can use some of that time on a community project all the better.

- - - - - Additional Content Posted / Auto Merge - - - - -

codersparks - what sort of data would you be looking to share via your Captain's Log?

My initial thoughts were to just track my trades so that I can see where I have been the most profitable, that soon evolved into tracking buy/sell prices at different stations aswell.

As part of my app I would like to subscribe to the EDDN and as I want to consume the data, I feel its only right to share my data. Therefore, I would be willing to contribute any data that I can to help others, be that system information, commodity prices whatever.

I know there are other tools out there but I am looking for a project to do in my spare time. If I can use some of that time on a community project all the better.

- - - - - Additional Content Posted / Auto Merge - - - - -

I was MIA for a week and will be MIA again starting Thursday till wednesday next week. This time a small holiday. Hence my thread didn't get the love from me it deserves ;-)

Hope you have a good holiday - No rush, as I said above I only have a limited time at the moment anyway.

Would your captain's log tool by a windows tool which can read the netlog logfiles to get the current system a commander is in

Probably not, as it is webapp based, but I have no issue with using an existing script that will harvest that data and publishing it (or creating one myself)

 

[wolverine2710]

Probably not, as it is webapp based, but I have no issue with using an existing script that will harvest that data and publishing it (or creating one myself)

THAT would be very cool and useful. Atm we don't have such a tool ;-(

 

[Andargor]

I've been PM-ed by an author that (s)he has a POC for OCR-ing the commodities market. That would be a great source of data for EDDN!!

I'm tinkering myself with OCR using GDI+ and tesseract (so it's Windows-based, for now). No guarantee anything will come of it because of limited time, but I will keep you guys posted. If I get things working reliably, then I would probably ask someone to integrate with a client and/or make an installer.

I might set up my own firehose, just a simple node.js/socket.io reflector. This could feed other stuff. I don't want to get too much into detail as I want the basic OCR tech to work reliably.

 

[LukeP]

Probably futile attempt . but maybe with enough spotlight?

[Feature request] Station market CSV dump

Support?

 

[jamesremuscat]

I might set up my own firehose, just a simple node.js/socket.io reflector. This could feed other stuff. I don't want to get too much into detail as I want the basic OCR tech to work reliably.

Don't set up your own, just point your output towards the EDDN - announcement coming Real Soon, I promise (or PM me for insider info...)!

 

[Snuble]

Don't set up your own, just point your output towards the EDDN - announcement coming Real Soon, I promise (or PM me for insider info...)!

What would be very cool is if FD could point some data toward EDDN as well...

 

[codersparks]

THAT would be very cool and useful. Atm we don't have such a tool ;-(

Do we still need this tool?

We now have the spread sheet with the system details, but as that does not have all the systems in I am guessing yes (for route planning purposes).

Let me know if you think I am wrong