Discussion External API Requirements Thread

[Cmdr Sharky]

Edit: Yes in hindsight I should have chosen a different name. EDDR (Elite Dangerous Data Relay) would have been a more fitting name then EDDN (Elite Dangerous Data Network) as EDDN does not store anything. Ah well live and learn ;-)

Networks don't store things either. Machines on the network store things.

 

[maz_r]

Can I add yet another vote for the UI data to be exported. As an inveterate tinkerer, some feed as to the status of throttle, speed, shields, landing gear, running lights, hardpoint deployment etc would provide the community with an entirely optional (i.e. gives no advantage to a player who does not use it) way to massively improve the immersive nature of the game for those that want to .

I appreciate that this probably isn't what you had in mind when you started this thread, since such a feed is entirely client based and does not need an API per se; for example, simply dump a short 'message' to a user defined COM port every time a UI element is updated... Then sit back and watch the weird and whacky ways that this info is transformed by a grateful (if minority) set of us. Simply can't wait to get my programmable lights to change to 'red alert' if attacked

 

[Artie]

....Yes I hope FD creates EDDN functionality into the core server and then it could stream market updates to clients. It even has a few advantages over EDDN....

I actually hope they don't, or it will be at least optional. For two reasons:
- you cannot request data you want to know in given moment directly, you must wait for them (and hope they will arrive at all)
- you must listen to stream non-stop and you may miss something when anything bad happens ("listener" restart, problems with connectivity, etc.)

 

[wolverine2710]

I actually hope they don't, or it will be at least optional. For two reasons:
(1) - you cannot request data you want to know in given moment directly, you must wait for them (and hope they will arrive at all)
(2) - you must listen to stream non-stop and you may miss something when anything bad happens ("listener" restart, problems with connectivity, etc.)

Added bullet points in this color. For starters I suggested making it optional - opt-in.

(1) Not sure how to interpret this. EDDN is a relay as would be FDDR. What it receives is send directly to the output of EDDN - called the firehose. YES, with EDDN you have to wait till the prices for a station is sent to it before you receive it. Most end users use a trading tool (TT) like the BPC, Thrudd, for them this is not a problem. Those TT's have their own database. Just startup the BPC/Thrudd and the price data will always be up to date. The same goes for the TD merging tool of maddavo. Import the prices with a TD subcommand and TD is up to date. Already other TT's are using EDDN's data. I hopefully explained there are additional services which WILL give you history prices data. EDDN/FDDR is NOT about requesting at will the latest prices for a station. BUT when enough commanders use EDDN/FDDR effectively you will more or less have the up to date prices for a certain station.
(2) See (1). You as an end-user do NOT have to listen to the firehose 24/7. Use a TT or one of value added services. Today themroc posted his cleaned eddb data which uses amongst other things uses EDDN. This could be used as a base by OCR tools to clean their own data. If he deems it needed/useful themroc could make it so that for example EliteOCR uploads a dirty .csv/.bpc to eddb and receives back a clean/corrected one.

Should I have misinterpreted your post, feel free to correct me and I will try to re answer your questions/remarks.

Note: I'm not saying FD should create FDDR, I was merely suggesting it - which is the prupose of this thread. Should they decide not to do it, EDDN will take care of gathering/distributing the data. It would just make life a bit easier and the data is guaranteed to be valid - no pranksters changing stuff. When EDDN is used, In that case its ALSO an opt-in because a commander has to install a tool which uploads the data received by the web-api - in whatever form this is implemented/supplied.

As another commander stated: EDDN is a case of "if you built it, they will come". The same will be true for FDDR. A few days back EDDN had 450 subscribers. Mostly commanders building their own tools using EDDN prices data. Some out of curiosity. Atm the BPC and Thrudd are neither uploading to EDDN nor using the EDDN data. TD (Maddavo's tool) and other currently smaller TT's are using the EDDN data.

 

[Artie]

ad 1] Yes, but it doesn't work too well for user data, for example (like rank status, current location, etc.). More precisely, it can be done by streaming data of all users when anything changes, but that's crazy overhead. Similar for market data - it works now, with the limited user numbers involved (which are sending data), but streaming markets of whole inhabited part of galaxy when anything changes here is overhead again - even when there will be just markets where is any real player (and we talk around 300k players now, I think). It is doable for sure, and someone may benefit from it (despite the fact that may flood given tool database with data that users will never use), but as I am not directly against it, it should be just an option (in case anything like that will be implemented).

ad 2] Yes, but some failure may affect tool itself. I see no reason why some tool should rely on some other tool to get Frontier's data, when Frontier's official source of data will be provided.

But, as this thread is just for API requests and this post is more a discussion-like, I am stopping it here. I said what I wanted to say to this topic.

 

[Astrospanner]

We’ll keep you informed as we progress and thanks in advance for your participation in this!

Michael

Thank you for asking for input!

I thought about this too much over the weekend; apologies if someone has already posted similar. What I've come up with is more "use cases" than API requirements; I hope it still helps.

The App I would like to see is a history log for a player (you can only see your own data!). It serves two purposes
1) log the route that the player has taken from system to system
2) log all data about a port (outpost, etc) when a player has docked

When I say all data, I mean everything that's accessable from the station menu and navigation tab. So you've got
- all the trade data; at the time when you docked
- prices and available components at outfitting
- available missions
- prices and available ships
- if there is a black market.
- distance from nav point

The key is that this is a snapshot.

The way it work (for the player) is this:
Whenever you dock, you have the option of sending all the station data to your companion app; for 10 credits.
This is a non-zero cost as you are just speeding something up that you could do by hand (for free). At this point, the data is sent to the companion app. It's the players responsibility to have the app running and available on eg local wifi. If the player doesn't spend the money, it doesn't get sent. This way, Frontier don't have to snapshot the entire milkyway often.

In addition, when a player dumps out of hyperspace, there's a shipboard command to send current location to the companion app (system name; any details already on the nav computer (ie star type, distance to stations, legal system)

In the app, you can then do queries like "where have I been with a black market?"
"where can I get a class 2 A rated missile launcher?"
"what was the route I followed to get here?"
Prices can be analyzed for trade patterns, but it's only with the data you've been and collected already.

I don't think this is particulary immersion or economy breaking as it just does what organized players are doing with their own spreadsheets. It also doesn't require Frontier to keep a complete log of everything for every player; the data is all kept by the player for the player.

Possible downside: a 3rd party app could take the data from different users and produce almost real time "good trade" maps.


I'd also like to suggest in station photography, saved to the app. So you can ask the station to give you a traffic camera-like
image when you've docked, or coming through the docking port. Again, only available at a (small) credit cost, after you've already docked. This gets saved to the companion app; players can then use these to show off to friends and get them involved. The cameras don't give any gameplay information away as it's not real time...


Finally, I'd like the companion app to help hud colour customization; have some stock star images (different types and hence colours) and allow the hud colour to be seen overlaid, to pick the one the player likes best and doesn't screw up visibility by a particular star type (for example)). This doesn't require any API though.

 

[wolverine2710]

ad 1] Yes, but it doesn't work too well for user data, for example (like rank status, current location, etc.). More precisely, it can be done by streaming data of all users when anything changes, but that's crazy overhead. Similar for market data - it works now, with the limited user numbers involved (which are sending data), but streaming markets of whole inhabited part of galaxy when anything changes here is overhead again - even when there will be just markets where is any real player (and we talk around 300k players now, I think). It is doable for sure, and someone may benefit from it (despite the fact that may flood given tool database with data that users will never use), but as I am not directly against it, it should be just an option (in case anything like that will be implemented).

ad 2] Yes, but some failure may affect tool itself. I see no reason why some tool should rely on some other tool to get Frontier's data, when Frontier's official source of data will be provided.

But, as this thread is just for API requests and this post is more a discussion-like, I am stopping it here. I said what I wanted to say to this topic.

We seem to misinterpret each other posts. EDDN is for generic dynamic data - can be used for other dynamic data. Can't imagine FD is gonna use FDDR for user stats. I suggested using EDDN as a collector for FDDR data to minimize band width usage at the FD servers/side. As a reference, EMDR handles 1 Million messages a day. BUT I agree this thread is about the API, not EDDN. Feel free to PM me or hop over to the EDDN thread. Others and myself are more then willing to answer all your question there ;-) Signing off.

 

[Serpentstar]

Added bullet points in this color. For starters I suggested making it optional - opt-in.

Although this has nothing to do with the current message thread, it is appreciated that some people don't use red to highlight something. I cannot actually see red on black or a difference between red and black text. Orange is perfectly fine. Thank you.

 

[Javert]

For me the most obvious information to be able to send to an API would be :

- Commodity and trade pricing information. What are the products, buy and sell prices and stock levels at stations that I have visited.
- What are the facilities available at different stations that I've visited.
- What are the actual ship upgrades available the last time I visited a station.
- What systems have I actually visited, and for which systems have I got complete cartographic data (as opposed to partial data).
- Statistical information about ship upgrades etc.

 

[Landy_Ed]

I posted this outside of this thread, hope a repost here is not considered bad behaviour but I only just spotted it :

Rather than having to look around and navigate menus while in flight, it would be really useful if there were another app, rather like the iOS app but simply cast into an unused monitor for those of us that have multiple monitors but can't span our displays, that could display some or all of all the info from these sub menus.

Rather than Frontier putting the effort into a full app, putting the data into the for 3rd parties to build something over would be cool. It need not be in-process to the game (in fact, it really shouldn't be), so some latency would be acceptable.

 

[Krafter]

Pretty much everything I want is covered in previous posts but anyway here is what I would like and what I would use it for.

- Access to data about the system I came from and the system I am in now.

The reason I want this is I like to keep travel logs. Sometimes I just write down the system name, other times I write down details like "Cool binary star system". If I had this information I could write an app that grabs the data and automatically generates a log entry for each system. Then I could fill in the details as I go.

This next one might seem a bit silly but.

- I would like to know the star type of the primary star in the system I just jumped to.

The reason for this is I have hue RGB lighting in my room and I would like to dynamically change the light of the room to match the color of the primary star in the system.

I have a lot of other ideas but most of them are covered in other requests.

 

[Andrew Sayers]

Note: I'm not saying FD should create FDDR, I was merely suggesting it - which is the prupose of this thread. Should they decide not to do it, EDDN will take care of gathering/distributing the data. It would just make life a bit easier and the data is guaranteed to be valid - no pranksters changing stuff. When EDDN is used, In that case its ALSO an opt-in because a commander has to install a tool which uploads the data received by the web-api - in whatever form this is implemented/supplied.

Pranksters are an important point actually, and not just for trade data. Would it help for Frontier to expose some kind of interface to publicly guarantee the accuracy of certain messages? For example, requesting the server publish given types of information (current trade data, ignore list, group members etc.) to api.elitedangerous.com/user/<user-id>/<random-number>.json, which the client can then pass on to EDDN.

 

[sweetie]

I'm creating this application which will be a voice interface for Oculus Rift Users:

https://www.youtube.com/watch?v=pZi_XZhFzyM

(1) I need a Port Name feed when entering a port.
(2) Docking Request Results with Pad Number that was approved for docking.
(3) A hook to send a system name to select for a jump point. In the voice activated part I have favorite trade routes. It would be nice to pass it the jump system name I need to go to and from the call it makes that system name the selected next jump system
(3b) A hook to send a port name to select in super cruise.
(4) When entering any part of the Starport services an xml file or json file is written locally I can access to grab that information I saw on the screen, from outfitting, commodities market, ships. The content that is dumped is only what I selected to in the services and view. This makes the user have to got to each section as they would in game.
(5) Some combat log file that is written with information about the person I shot at, system it took place and outcome... like your ship was blown up, cmdr billy bob was blown up. Ship stats could be included, shields dropped and ship resulted in 40% hull damage. Going to use this for a commanders log diary and would be nice if a lot of it could be auto generated from these files. think of it as a black box.
(6) When entering into a system some way to tell that system has been entered with X Y Z position (instead of having to hack some log file) The X Y Z allows the traveled data base to expand as the user goes through the game. Consider it as a Diary/GPS system that once you go there you know the system and its location and this will track ingame date and time the visit occurred.

 

[Kipper]

I would like to register a loud vote against JSON. It's a terribly inefficient and difficult to parse format which translate to wasted cycles dealing with it which matters when there's a lot of them. I read ridiculous things like omg it handles 200 updates a minute with a thousand end-points it's great!

The target capability is a thousand updates per second with ten's of thousands of end-points.

The constructive thing to do would be to suggest alternative formats which you think will do the job better, that way we can discuss the merits.

In any case I'm not entirely sure on your point of speed - how quickly you can serve and parse data depends on the platform that's serving it and the hardware/software that's parsing it.

Some data formats might be better than others but that's for discussion.

 

[RickDangerous]

Pranksters are an important point actually, and not just for trade data. Would it help for Frontier to expose some kind of interface to publicly guarantee the accuracy of certain messages? For example, requesting the server publish given types of information (current trade data, ignore list, group members etc.) to api.elitedangerous.com/user/<user-id>/<random-number>.json, which the client can then pass on to EDDN.

That looks like a query API. Query APIs are a very bad idea, as then apps rely on the SLA of FD for every single request to their system. That would be insane. Let's not do that.

 

[RedWizzard]

So we'd have an API to query for known system and station data which could include last-known-good commodity data and then a streaming service for market data. Part of the data you query for is to get the system-station ID which is like a stock-ticket but probably just a GUID perhaps modified to have a couple digits to reflect the galactic sector it's in. Then in the streaming service you use the fixed-size ID (not variable size strings) to send data and now you can easily do it with a binary format.

I would like to register a loud vote against JSON. It's a terribly inefficient and difficult to parse format which translate to wasted cycles dealing with it which matters when there's a lot of them. I read ridiculous things like omg it handles 200 updates a minute with a thousand end-points it's great!

Let me register a loud vote against binary formats. They're brittle, harder to use and debug, and any extra efficiency is rarely needed. Json and xml are standard options with ready made parsers available for almost every language. But xml is a fair bit more heavy-weight than json.

Your json performance example is meaningless. 200 updates a minute doing what? DB? Messages? HTTP serving? What hardware? What size json documents? Without a comparison of the same software serving a different data format you can't draw any conclusions about the performance. In all likelihood it was the software architecture to blame, not the choice of data format.

 

[RedWizzard]

That looks like a query API. Query APIs are a very bad idea, as then apps rely on the SLA of FD for every single request to their system. That would be insane. Let's not do that.

If FD's servers are down then there is not much use for the data anyway.

 

[Andrew Sayers]

That looks like a query API.

In hindsight that wasn't the most readable post I've ever written

Right now, the ED server sends a message to an ED client, the message is retrieved from the client with clever jiggery-pokery (OCR, manual entry etc.), then that data is sent to a third party (e.g. EDDN). So far we've talked about an API that removes the jiggery-pokery but still requires the third party to trust the client. If third-party tools are an important use case, it would be good for Frontier to find a way to verify the accuracy of messages.

One way to verify the accuracy of messages would be to copy them to a public URL for third parties to download. Say the server sends your client a message like { "hydrogen fuel": "100 credits" } - the client could request that message be copied to a publicly accessible URL, allowing third parties to verify the message was genuine.

I don't see a big SLA issue here - the implementation could be as simple as static content being written to a file and served through any old web server, or even served straight from an S3 bucket. And even if that went down, third parties could just accept unverified messages from clients until the server came back online. Then again, I'm more interested in fixing the trust problem than any specific implementation - for example I'd be just as happy to see a solution where the server PGP-signs messages so the client can keep being the man in the middle but loses the ability to mount an attack.

 

[RickDangerous]

If FD's servers are down then there is not much use for the data anyway.

First of all, just because an API server is down doesn't have to mean that the game servers are down. Separate things. Second, violating an SLA doesn't have to mean that the server is down, it could simply mean it takes let's say 10 seconds to respond. With a query API where each request goes directly from app to FD servers, there is no way to guarantee any kind of response time, because you are tying the scalability of one system (the app) to another system (FD API servers) which are under completely different controls. It's just not a good idea.

This is why I suggested a few pages back that the FD API should be based on Atom feeds that provide essentially events of changes. Apps consume these, at their own pace, and update their own database. When they get client requests they will have all data available locally, and do not have to make any requests to FD servers. If the FD server is down that just means the app data will be stale, as opposed to "not available at all".

This is simple to implement, you can use "slow" XML encodings, and removes the "one request to app is one request to FD API" problem.

 

[Slopey]

Say the server sends your client a message like { "hydrogen fuel": "100 credits" } - the client could request that message be copied to a publicly accessible URL, allowing third parties to verify the message was genuine.

Why bother? In my case, if there's a single url to retrieve the commander's current info (a la the iOS app currently), and it's hosted on an ED server, that's good enough for me.

If the data has been spoofed then it's not the end of the world - it's only a game after all and we're not dealing with any real-world or sensitive data. By virtue of the fact I've got it from a known url is all the authentication I need really. If someone wants to go to the effort to hijack that url, good on them. PGP signing and multiparty authentication just seems overkill for the price of Hydrogen Fuel at Lave in game.

- - - - - Additional Content Posted / Auto Merge - - - - -

Let me register a loud vote against binary formats. They're brittle, harder to use and debug, and any extra efficiency is rarely needed. Json and xml are standard options with ready made parsers available for almost every language. But xml is a fair bit more heavy-weight than json.

+1. Binary would just be a pita.

XML or JSON is definately the way to go - it doesn't matter which, most modern languages can parse those directly into a nice structure or be queried directly with linq etc. Also, given that there's already a text based API to feed the iOS app, there's no need to complicate matters.

All FD need to do on the market/commander data front is allow us to parse the iOS app feed, and drop the secondary email authentication for it (which is a pain).

Telemetry would come directly from the client in one of several methods as it's time critical.

With those two, that'd address 99% of the use cases above.