Page 1 of 58 1234511 ... LastLast
Results 1 to 15 of 857

Click here to go to the first staff post in this thread.
Thread: EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

  1. #1

    EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way.

    Announcement: Saturday the 23th of January 2016.
    Brilliant news:
    On the 5th of January 2016, Zac Antonaci -Head of Community Management of FD- wrote in the "Open Letter to Frontier Developments" thread that usage of the iPhone companion app api is LEGAL. Source. Which makes the usage of tools which use this api like EDMC, EDAPI and EDCE LEGAL. Other great news: On the 18th of January 2016, Brett C. -Community Assistant/Manager of FD- created the "Elite Dangerous->Gameplay and Features->Player Tools & API Development" subforum. Its an excellent place for checking out new and existing tools. The EDCodex thread is also there.

    Foreword (2016)

    This thread is primarily meant for programmers who implement EDDN in their programs (or tools of any kind).

    You as a user don't need to know anything about EDDN beside the fact that it is a information relay. It means everybody can send data to it and the data will be distributed to everybody who is connected to it. Currently it is distributing market data, station shipyard and station outfitting data.

    There are sender programs and receiver programs. Examples: All iphone companion app api based tools like EDMC, EDAPI and EDCE (which are LEGAL) can optionally sent data to EDDN. Depending on the program this can be market data, station outfitting and station shipyard data. EliteOCR by Seebek only sends market data. Eddb.io by themroc is a receiver. Maddavo's Market Share (MMS) by Maddavo is a receiver. "RegulatedNoise DJ-Version" *All in one* Trading Tool (includes OCR) by Duke is both sender and receiver. There are quite a few tools out there which allows you to share your market data or receive publicly shared data from others. We expect there to be more very soon. There are commanders who have created Value Added Services for EDDN. Example: Archiving services as EDDN does NOT store any data it distributes at all.

    If you decide to write a tool using EDDN in any way, we will all be more than happy to help you. If you don't then don't worry. You won't miss a thing.

    A very extensive list of tools using EDDN and tools for starting with EDDN wrt programming can be found on EDCodex. You can find it in the EDDN category.


    *Begin historical documents*

    Announcement: Tuesday the 7th of July 2015
    EDDN v0.5 succesfully installed. Schema V1 support dropped.

    See this post by jamesrecmusdat for detailed information.

    Related news
    Announcement: Tuesday the 30th of June 2015
    First of July a new EDDN version will be installed - without schema V1 support.
    See this post for detailed information.

    Announcement: Friday the 19th of June 2015
    Starting the 1st of July 2015 EDDN schema V1 will be discontinued.
    See
    this post for detailed information.

    Announcement: Friday the 5th of June 2015
    "OCR error free" or "Mobile API based" tools.


    This week "OCR error free" (OEF) or "Mobile API based" tools were released. Elite Dangerous Companion Emulator (EDCE), EDMarketConnector (EDMC) and EDAPI. These are all open source projects written in Python.


    • All tools (optionally) send (market prices) data to EDDN.
    • It has become clear that similar tools have been created. Privately used and not publicly released.
    • One tool supports all publicly released trading tools. Accomplished by creating a .bpc, .csv and a TradeDangerous .prices files. Those can be imported into respectively Slopey's BPC, for example Cmdr Thrudd's trading tool and Trade Dangerous by kfsone.


    Announcement: 15th of April 2015
    New features and a proposal for a new V2 schema to review.

    New features are mostly behind-the-scenes admin related, making the network components easier to deploy and administer, but there is also now a status page available at http://eddn-gateway.elite-markets.net/ showing message throughput of the network. Anthornet also has an EDDN-monitor: http://eddn.ed-td.space/

    AnthorNet and jamesremuscat have been collaborating on a v2 schema that lets you submit multiple commodities in one message. The draft is up on GitHub: https://github.com/jamesremuscat/EDD...2.0-draft.json. It's also installed in the gateway so software authors can move to testing using it if they wish - usual /test hack applies. The schema is subject to change until we declare it stable.



    Announcement: 15th of December 2014
    Announcing Elite Dangerous Data Network alpha test!

    Jamesrecmuscat has made an Alpha version of EDDN publicly available. GREAT work James. See spoiler for details.
    It's long since time I posted this...

    Announcing Elite Dangerous Data Network alpha test!

    For full details see https://github.com/jamesremuscat/EDDN/wiki - which not coincidentally is also the Git source code repository for the project.

    There's a sample Python application in the Git repo - see Client.py.

    This is half of what's needed - the other important thing is defining the format of the message. What's there currently is a starting point, but the questions for you all are:

    • What else needs to be in the market data message?
    • What other data would people be interested to see distributed over EDDN?



    I'll be continuing to read this thread - and since I'm not back in the office until the new year, I should even have time to fix things if/when they break!

    Atm there exist quite a few third party trading tools created by passionate, creative and talented commanders. Each tool has their own speciality, features and audience. They currently all have in common that they are based upon manual input of data, typed in by commanders. This is because of FD's new data access policy which was announced 5+ days before SB2. Source.

    Trading tools I know off - probably not complete. Its from the OP of this thread.

    List of third party trading tools. The tools are listed by commander name - in alphabetic order.

    • kfsone. TradeDangerous: power-user trade optimizer (TD), users unknown. Trading tool, route planner, manual input of data. Used EMDN in the past. Thread "TradeDangerous: power-user trade optimizer". Sourcode.
    • Maddavo. A website where a TD's TradeDangerous.prices can be uploaded, merged with the most current one on his website and returned to the user. Effectively a way to get the most up to date prices for TD. Users unknown.
    • Slopey. The BPC. Trading tool, route planner. Thread "Slopey's Market Tool/Best Profit Calculator (BPC)!". Slopey mentioned that in SB1 at peak times there were 500 users.Ised EMDN in the past.
    • Zebidie. Trading tool and route planner. Thrudds Elite Dangerous Tools Zebidie mentioned in his thread a few weeks back he has 2300 users.


    PRE SB2: Trading tools, automatized data scraping, EMDN, FD's new data policy and consequences.

    Before SB2 there was EMDN by Andreas which contained commodities market information. EMDN stands for Elite Market Dump Network and received its information from marketdump (MD), a separate command line tool made by Andreas. EMDN was used by for example Slopeys the BPC (which came with MD), kfsone's TD, IxForres web-api. MD got its data by accessing the ED program. When the commodities market in Ed was opened the prices were gathered and send to EMDN. After the data policy changed, Andreas did shutdown EMDN. Something he mentioned he would do if FD asked him to.

    This had consequences for the trading tools. Until then the mentioned tools had more or less up to date market information. At least for those systems which had profitable routes, station snot to far away AND were visited by commanders. Mentioned by a lot of commanders the tools caused them to go the outer rim of SB1 and not to stay in the core systems. More profit to be made because of the implemented market mechanisms in ED. After the shutdown of EMDN the BPC , IxForres web-api and to a lesser extended TD were affected. The BPC data became stale, the web-api was shutdown and TD again solely relied on manual entered prices.

    EMDN: technical point of view. Its proposed successor EDDN.

    The idea for EDDN came from EMDN by Andreas which itself is based upon EMDR (EVE Market Data Relay). In its most basic form EMDN is nothing more then a data stream. More specific a ZeroMQ publish/subscribe pattern with a bit of logic in the receiver to do sanity checks. Data was send to it by MD as a zlib compressed JSON structure. Note: ZeroMQ has support for multicast to reduce bandwidth. In normal English: Data send to EMDN will be send to everybody who has subscribed itself to it. Again this was used by Slopey to store market data in his database server, emdn-tap of TD, IxForres webapi.

    Nothing has to be stored in EMDN but CAN be. Andreas for a while has set up an ELK stack. E: Elasticsearch for deep search and data analytics. L: Logstash for centralized logging, log enrichment and parsing. The K in ELK stands for Kibana. For powerful and beautiful data visualizations. What can be done with this is explained by Andreas in a few threads and a video. Thread: "Economist Newsletter #1: Insight into the current market dynamics". Thread: "Economist Newsletter #2: The Eranin food crisis". Video: "[Video] Market data visualization with EMDN and ELK"

    EDDN is nothing more or less then EMDN with a different name. And EDDN is fed by data obtained in a legal way. In this case data send to it by trading tools - based on manually entered data.

    EDDN usage for Market data visualization ONLY.
    Note: Market data visualization with EDDN and ELK. Everybody has his/her preferred way of playing Elite. Wrt trading: Some use the ingame tools, others prefer trading tools. EDDN can also be used by traders as a pure analysis tool. See the previous spoiler tag.

    Tools created for EDDN and/or using it.
    20140120. Heavily WIP. I've have to reread the whole thread to find the gems hidden in it ;-)

    This list is NOT complete. Atm according to EDCodex there are 13 publicly released tools for EDDN.

    EDDN Legend:
    SUB: Subscriber of EDDN, connects to EDDN and retrieves data.
    PUB. Publisher. Sends data to EDDN.
    Value added service. This can for example be an archiving service where one can collect historical data.

    Name:
    Author: Askarr
    Description: Archive for EDDN messages.
    Website: N/A
    Source available: No
    Thread: N/A
    Compatible with Release: Yes
    EDDN support: SUB
    Value added service (VAS): Archiving
    Comments: Announcement here, with code sample of how to use it. More info about usage here. Note: You must first ask for an key before you can use it.
    Note: Saturday 20150425. Added to ED3PTT.

    Name: Elite Trade Analyzer
    Author: Pandemic
    Description: Trading tool.
    Website: N/A
    Source available: Yes. Open source license: not specified. Gitub repository.
    Thread: N/A
    Compatible with Release: Yes
    EDDN support: SUB
    Value added service: N/A
    Comments: This tool will analyze and find the best routes for trade data persuant to several parameters: Proximity to current location, distance between trade stations and current commodity prices and supplies
    Note: Saturday 20150425. Added to ED3PTT.


    Name: eddb - Elite: Dangerous Database
    Author: themroc
    Description: Routeplanner.
    Website: eddb
    Source available: No
    Thread: eddb - a site about systems, stations, commodities and trade routes in Elite: Dangerous.
    Compatible with Release: Yes
    EDDN support: SUB
    Value added service: Archiving
    Comments: Much more functionality is planned, like a REST-API.
    Note: Saturday 20150425. Its (now) in ED3PTT.


    Archiving services for EDDN - with an API

    See this post for an overview.

    Note: Those are archiving services with an API. There are also services which do store EDDN data but don't have an API. They offer stored EDDN messages for a certain period in the form of a download. Examples: eddb.io and Maddavo's Market Share.

    The current situation. We have multiple manually input based trading tools. They afaik do NOT have price info for every system/station/platform. If a station/platform has price information its far from up to date and/or incomplete. This situation will likely gets worse when the Galaxy opens up in Gamma. Manual entering of prices in the tools takes time and dedication. All those great tools have in common that they are basically data island.

    Now to the title of the thread. If all price information was shared between the different tools the end effect would be that each tool would have more reliable, less outdated data, making each tool more useful. One possibility would be that the tools have a web-api which would allow to get the latest prices and a web-api to upload prices to them - could be with authentication like OAuth. This has drawbacks, the biggest one I see is that for full interoperability each trading tool has to support the web-api of all other tools. Which becomes impractical real soon.

    Might I suggest another solution: EDDN. Basically when data is entered in a trading tool its submitted to EDDN. All trading tools subscribed to EDDN directly receive that information. The data islands disappear. In effect a trading tool becomes more valuable. EDDN can also be used to check data entered in each tool. Does this mean that a certain tool loses its identity? Does it mean a tool becomes obsolete? Does it mean a commander is forced to use a certain tool or change to another tool? I really don't think so. For example thrudd's tool has 2300 users who have to login and when they entered data they get to a higher level and can use more functionality of the tool. This doesn't change.

    Can EDDN become a reality and useful? It all depends on the authors of the trading tools. If they see merits in it, then an EDDN can be setup. If too many authors think its a bad idea or does not fit in the ideology of an author, EDDN will add little to the party AND the data island will continue to exist.

    The whole idea of this thread is to discuss if there is interest in it and discuss the technical site of it. The only thing I can do is to present the idea. For the rest its up to the authors . Let the discussion begin. Of course every commander can chip in. I will send a PM to the authors to make them aware of this thread.

    Note: Tool authors. Should I've made a error in describing your tool, number of users etc. Please let me know and I correct it. Should I have forgotten a tool and/or author, I apologize for that. Just send me a PM or post it in this thread.

    Please lets concentrate solely on the technical aspects. Lets not discuss whether or not there will be in game trading tools. And Please lets NOT turn this thread into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress - no matter how hard a particular site tries. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". I sincerely hope and trust you respect my wishes.

    Addendum/correction. I was the cause of spreading some misinformation about how to upload data to EDDN,my bad. Need better RTFM skills and/or more time. Luckily jamesremuscat (who´s is setting up EDDN corrected me): Snippet: "But the upload of data for EMDR/EMDN was (and therefore EDDN will be) an HTTP POST to a gateway, which deals with pushing the data across the ZeroMQ system for you. Uploader-users don't need to worry about ZeroMQ.

    Credits:
    Vivio Technologies for free hosting.
    After I created this OP I was contacted by commander Errantthought. He and his boss/manager both play ED. After talking to his boss he was able to offer me a free hosting solution (VPS Hosting) for the EDDN. I want to thank them both and their company for donating the resources to make EDDN possible.

    jamesremuscat. For reviving Andreas EMDN and creating the EDDN. Much appreciated.

  2. #2
    Although I am not a user of these tools, I can see a huge benefit in your trying to have all manual input resources leveraged for any of the tools people wish to make, so have a bump
    CMDR JUPITUS DARKSHADOW

  3. #3
    When the full game launches with 400 billion systems then these kind of tools are gonna become choked to death with the amount of data.

  4. #4
    Originally Posted by WelshSossy View Post (Source)
    When the full game launches with 400 billion systems then these kind of tools are gonna become choked to death with the amount of data.
    It's more down to the amount of players, most of those systems will never get touched.

  5. #5
    I wrote my OP last night. Didn't had time to PM the authors of the trading tools. Something I've just done.
    I've also been contacted by a commander who likes the idea and is very interested in setting up EDDN. It of course all depends on the TT authors. Without data from them EDDN is not viable. Hopefully the authors like the idea. Fingers crossed.
    I'm totally open for ideas, suggestions etc....

  6. #6
    Why would you do that, if I find a good place to buy something super cheap or sell really high I am not telling anyone.

  7. #7
    Originally Posted by Hawkmoon View Post (Source)
    Why would you do that, if I find a good place to buy something super cheap or sell really high I am not telling anyone.
    You seem to have missed the last part of the OP. This is also the reason I can't comment on your post. Feel free to send me a PM if you want to discuss your post further. For clarity I will post the last part of the OP again.

    Please lets concentrate solely on the technical aspects. Lets not discuss whether or not there will be in game trading tools. And Please lets NOT turn this thread into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress - no matter how hard a particular site tries. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". I sincerely hope and trust you respect my wishes.

  8. #8
    A shared REST API for ED data would be amazing. Create a GitHub repo and let's get started!

  9. #9
    I'm totally up for integrating my site with this.
    The scale of the data gathering task at hand is so large it's silly for us all to compete for the limited time of the pool of users.
    I've already implemented the star data from ESDC, moving the trade data to a central location would be a good idea.
    My main concern is time, I wanted to try to get my site updated and solid before gamma/preview so I could at least spend a few days flying about in the game as I'm hardly had time to play beta 3.

  10. #10
    I would be interested in participating. I am planning on writing my own web based "captain's log" app that I can use in the steam overlay. If we can get a common schema defined that would be a good place to start.

  11. #11
    Originally Posted by cowboy View Post (Source)
    A shared REST API for ED data would be amazing. Create a GitHub repo and let's get started!
    The functionality of EDDN is of course in the hands of the implementer. I think the ZeroMQ solution has proven itself. Its perfectly suited for trading tools like Slopey's and Thrudds because they have their own website and can continuously receive data and store it in their database. Remember the firehose (datastream) of EDDN just outputs directly what it receives. For a tools like Trade Dangerous which runs locally (command line) and has no website it would mean a commander would need to run emdn-tap (which updates the local sqlite database) continuously to be up to date. For that a web-api to get the latest prices would be great. The API could be made so that for example a starting point and a radius could be given to reduce the bandwidth. Perhaps in a later phase EDDN could become a true ELK-stack and the Elasticsearch database could be opened so that every commander which has Kibana installed then access the data for analysis purposes.

    Thrudd has said he is in. Have not heart from Slopey or kfsone. A network of one isn't a network ;-(

  12. #12
    Originally Posted by Kuf View Post (Source)
    I would be interested in participating. I am planning on writing my own web based "captain's log" app that I can use in the steam overlay. If we can get a common schema defined that would be a good place to start.
    Welcome on board commander.

    Not sure what you mean with steam overlay wrt your captains log?

    Concerning format. In andreas marketdump thread I believe he has posted the format of the json structure he was using in his marketdump thread. The code for EMDN is in github. Perhaps there we can find what we need. Andreas seems to be MIA. Another possibility is emdn-tap which is in the repo of TD - old version of TD. Aside from the pure market data I think its good to have at least a field for the uploader, Slopeys BPC, Thrudds tool, Trade Dangerous. That way we know what comes from what source. It ALSO opens up the possibility that once OCR-ing succeeds and that data is uploaded to EDDN that when a certain TT author doesn't like that (ideology) he could simply filter out that data from the firehose (datastream).

  13. #13
    I watched the "crowd-sourcing" thread with interest and was quite surprised that nobody had agreed on a common format to do this stuff (that would also work if Frontier ever released an API), and everyone was concentrating on CSV files (for a time) and trying to co-ordinate stuff.

    What they really need to do is sit on a skype-chat for 30 minutes and unify it all.

    Just my 2c



    (also, great work all!)

  14. #14
    Originally Posted by Zebidie View Post (Source)
    I'm totally up for integrating my site with this.
    The scale of the data gathering task at hand is so large it's silly for us all to compete for the limited time of the pool of users.
    I've already implemented the star data from ESDC, moving the trade data to a central location would be a good idea.
    My main concern is time, I wanted to try to get my site updated and solid before gamma/preview so I could at least spend a few days flying about in the game as I'm hardly had time to play beta 3.
    Welcome on board commander. PERFECT that you want to help to feed EDDN.
    Received your PM. Hoping I hear soon from Slopey and kfsone.

    Been there done that. Basically haven't played SB3 at all so far. To many threads I have to take care of, see my profile. Also saving myself for Gamma. I think its best you finish your work on your tool, play some ED and help to flesh out a JSON structure (if you have time) for EDDN. First EDDN has to be set up anyway. From what I've seen in the python code of Andreas the basic firehose (shows in command box console what is received) is 12+ lines of code. As you already have code to put manually entered data in your database the hard part is already done. Sending data EDDN is afaik also straightforward.

  15. #15
    I keep on wondering if it'd be possible to switch the EMDN back on, pushing data to it from the plethora of manual-entry tools there currently are. There doesn't seem much point reinventing the wheel (or in this case, the firehose!) - plus I've already written a site that uses data from it

Page 1 of 58 1234511 ... LastLast