Discussion External API Requirements Thread

[GooDer]

Hello,

my needs are quite specific. I want to start clan / corporation and so I want to use API, to import clan members list to my web app.

Both XML and JSON API are fine for me.

Data that I will need (at least I can think of at this time):

1) Clan member list (with some member ID to fetch member info) with basic information
- CMDR name
- rank for combat, trade, exploration
- some rank in clan (if it will be) for rights

2) Clan member info
- system / station where is right now
- rank with federation, imperium, alliance
- bounty on his / her head
- some statistics (how much he / she earn in bounty, trade, exploration.....)

3) Clan member ship list
- list of ships
- where are ships docked
- upgrades on ships mayby?

Thats all I need for running clan that I can think of right now

Would be great if you can implement something like that.

 

[Dethblight]

Not sure what would be needed or how to go about this, as I know nothing about programming, but I would like to see voice attack be able to do things like "Target engine" Rather than "cycle sub target". Or "Find the closest white dwarf star".

 

[Kaela]

Not an API per se but I would like a file to read that I could programaticaly determine Switch States. Landing Gear, Cargo Scoop deployed etc. Silent Running, Ship Lights. I'm building a switch board and I want to use LEDs to indicate the state of of a switch.

Snap...

I have a Arduino clone used in a flight sim (x-plane) that knew when the gear was down, up or in transition, flap setting s etc. For the switch data to be available in a realtime api would be brilliant then I could complete my own flight console. I have the buttons worked out for gear, scoop, lights, but as the surrent system uses 'toggles' they are unable to 'know' the current state... Oh for such an API in realtime...

 

[Mooreel]

REST API For
Current ship status(damage, position of spaceship in space etc)
Receive and set game options
Read & Write/Display messages @chat/infopanels

 

[Shotgun2097]

join the dots colour by no,s

in creating these if we are not careful we will end up playing by numbers , i dont like the sound of this its like no thought for ones self just follow the dots this is generally anti game and pro hand holding join the dots

 

[James Redshift]

What about in game Keyboard Input? What if players wanted to use their tablet for in game chat or map searches?

 

[Jaeger]

Two categories

Trading data - all data related to your commanders trading environment. Prices from stations visited, trades made etc.. For obvious reasons.

Ship telemetry, data relating to your ships location, orientation, velocity and current target. Tie in with GlovePIE and implement features like "Cruise Control" .

 

[RickDangerous]

It's safe to assume that communication is one way - we're not going to allow direct interaction or injection of the live data.

Michael

I don't think there's a need for that, as it would be easy to aggregate FD data with any other sources, as a mashup.

 

[RickDangerous]

For a start, then:
Access patterns/considerations:

1. Long-half-life data:

Much of your data (System names, Item Names, etc) is likely to have a long half time. I'd like to see this data provisioned as static content in more than one format (but that's because I've done this before for an MMO).

Some folks will want XML and some will want JSON and some will want JS and some will want CSV. For the static portion of the content, you can refresh this periodically - probably at patch time, and by serving it as static content you can encourage API consumers to use HEAD requests.

This is why I suggested an Atom feed as the main API: the initial dump will be equivalent to a static dump, and any changes can be appended to that feed. With this 3rd parties can get either current state or historical overview (change history).

If the main API is Atom feed with XML data it is easy for 3rd parties to provide current state snapshots in any number of formats. For example, I am working on a project that takes current community data and puts it into plain files (XML/JSON/CSV, what have you) in a Git repo. Pull that for easy access to current state and various formats.

The reason to use XML is for namespacing, as Atom feeds are longlived and immutable. GZip compression fixes the bandwidth issue.

To achieve this, some form of "EDGUID" (Elite Dangerous Globally Unique ID) which you can only retrieve from in-game. Provide an SLA that for a given star, any name or position changes will not result in a change of EDGUID and you establish the all-important COA.

+1 on this, it would be VERY helpful if FD provided the authoritative GUID's that everyone else can latch onto.

Your API would then require users to refer to a System by it's EDGUID and it's name stripped of any punctuation or spaces:

/api/system/AAX-192-KYW-929-A08/luytensstar/

This is a query API. See my previous post (and the referenced blog post) for why I think this is a very bad idea. 3rd parties should provide query API's, and any SLA's that go with that.

You have a few tiers of tools in an ecology like this: collection (EliteOCR), distribution (EDDN), aggregation (e.g. data source mashups), and querying (Thrudds). I would prefer if FD only did collection and distribution, and let the community do the last two.

 

[Caduceus]

I'm interested in creating an app for ED roleplaying purposes that simulates a passenger/crewmate on the ship, mixed in with ED game lore. The kinds of things I'd like to use as app event cues:

In/Out Combat Status
Docked Status
Hull Health Percentage
Cargo Full/Partial/Empty
Cargo Type
Credit Balance (and Net Wealth)
Current Ship Type
Elite Ratings
Size/Type of Space Station
System Allegiance/Govt/Economy-Type
Ship Repair Events
Successful Exploration Scan Attempt
Number of Stations in the System
Number of Nearby Systems
Current System Name
Current System: New to Player or Previously Visited System Status

 

[James Redshift]

Perhaps once Player Characters are added, ship life support status and CMDR vitals would be appropriate.

[video=youtube;C2YZnTL596Q]https://www.youtube.com/watch?v=C2YZnTL596Q[/video]

 

[MagmaiKH]

There are a couple of key problems to solve relevant to ED.

Optimizing trade-routes is the canonical "Traveling Salesman" problem.

The next problem is Astro Navigation although I think improvements to this should be built into the game (cull the systems with a 80 Ly capsule between the current system and the destination and you can plot routes quickly).

The last problem is locating things you want to buy - like a Class A5 FSD.

Is a two-way API planned? Would it be possible for apps to tell ED which star-system to navigate to? (Right now the in-game system list is limited to 20 Ly even when your ship can jump further than 20 Ly so you have to keep going to the galaxy screen for long trips plotted with another tool.)

This also touches on the exploration game mechanic. I expected the galaxy to be persistent meaning once someone explored a system it would stay explored. If you coupled that with exhausting mining and other resources it would give us a reason to push outwards and explore the galaxy to forever find new planets & new resources. New stations would need to be built et. al. (huge demand for resources, their paying price goes up, people start hauling materials there to trade.)

For reading data Themroc has a good list. I would add a few more things. We might want to break the list into things you can just ask for, like star coordinates, and things the person has to be in the system or station for the information to be available (like the market data).

• Systems
  • Stars & Star Types (so we can determine if it's scoopable which is important for astro-navigation)
• Station
  • Other amenities; re-fueling, outfitting, re-arming, ship-yard.
  • What outfitting is sold.
  • What ships are sold
  • Commodity Name in English, French, & German

 

[codersparks]

I think all of the information that I would like to be exposed has been covered by others in the thread.

Just wanted to say thanks for looking into this, I am looking forwards to being able to develop against the API

 

[Kipper]

I haven't read the full thread, but for me a good start would be to expose all "public" data in the API - anything that I can find out regardless of who/where I am. So that includes details on everything (known) that's in the galaxy map, commodities galactic average prices, ship and outfitting stats and galnet feed.

Additionally some end points that perform useful calculations, like being able to measure distance between two known systems, or even plot a route based on a given maximum jump range.

These are things the API can offer without having to authenticate as a commander, so would make a good phase 1.

Technically, API queries should probably be performed against a clone of the game database so that high traffic doesn't risk harming the game experience, and rate limited so one user writing bad code doesn't clog up all other 3rd party services - developers should be encouraged to consume data responsibly and make their own provisions for caching frequently accessed data - individual developer accounts should have an API key that can be revoked as a last resort.

A series of http(s) endpoints offering JSON should be sufficient for both web and app developers alike.

 

[Andrew Sayers]

Here's an idea from a DDF discussion (link only accessible to people with DDF access - sorry!).

If you looked at a writeable client API some time in the future, would you consider letting app developers create new tabs in the ship menus? For example, Slopey's BPC would create fewer immersion issues if it was displayed in-game next to the cargo tab.

Obviously that's a big long-term idea, but it brings up a wider question - how (if at all) does this project relate to the plan to license COBRA? This is a great opportunity to show off the power of Frontier's technology and explain core concepts that might confuse people new to the system, after all.

 

[MagmaiKH]

Another idea, maybe too far out there to do but food-for-thought, is for FD to roll the EDDN functionality into the core server and then it could stream market updates to clients that connect kinda like the NYSE or LSE does.

(Note that they make heavy use of private networks with multi-casting enabled - which we cannot do over the Internet.)

So we'd have an API to query for known system and station data which could include last-known-good commodity data and then a streaming service for market data. Part of the data you query for is to get the system-station ID which is like a stock-ticket but probably just a GUID perhaps modified to have a couple digits to reflect the galactic sector it's in. Then in the streaming service you use the fixed-size ID (not variable size strings) to send data and now you can easily do it with a binary format.


I would like to register a loud vote against JSON. It's a terribly inefficient and difficult to parse format which translate to wasted cycles dealing with it which matters when there's a lot of them. I read ridiculous things like omg it handles 200 updates a minute with a thousand end-points it's great!

The target capability is a thousand updates per second with ten's of thousands of end-points.

 

[clavain]

Here's an idea from a DDF discussion (link only accessible to people with DDF access - sorry!).

If you looked at a writeable client API some time in the future, would you consider letting app developers create new tabs in the ship menus? For example, Slopey's BPC would create fewer immersion issues if it was displayed in-game next to the cargo tab.
...

Also suggested here, for those of us without deep pockets.

 

[Slopey]

Here's an idea from a DDF discussion (link only accessible to people with DDF access - sorry!).

If you looked at a writeable client API some time in the future, would you consider letting app developers create new tabs in the ship menus? For example, Slopey's BPC would create fewer immersion issues if it was displayed in-game next to the cargo tab.

Obviously that's a big long-term idea, but it brings up a wider question - how (if at all) does this project relate to the plan to license COBRA? This is a great opportunity to show off the power of Frontier's technology and explain core concepts that might confuse people new to the system, after all.

From my perspective, I wouldn't be interested in pushing information into the game on a UI tab - I'd see that as something FD should do. They should add a market data tab in there as part of the core game imho.

If I had to do it, all I'd suggest would be a feature which adds a form of HTML browser from a known file location. I'm not interested in DirectX calls etc, or turning my app into some sort of injected overlay.

But anyway, Michael has already said that the API will be outbound only, for obvious reasons.

- - - - - Additional Content Posted / Auto Merge - - - - -

Another idea, maybe too far out there to do but food-for-thought, is for FD to roll the EDDN functionality into the core server and then it could stream market updates to clients that connect kinda like the NYSE or LSE does.

(Note that they make heavy use of private networks with multi-casting enabled - which we cannot do over the Internet.)

So we'd have an API to query for known system and station data which could include last-known-good commodity data and then a streaming service for market data. Part of the data you query for is to get the system-station ID which is like a stock-ticket but probably just a GUID perhaps modified to have a couple digits to reflect the galactic sector it's in. Then in the streaming service you use the fixed-size ID (not variable size strings) to send data and now you can easily do it with a binary format.


I would like to register a loud vote against JSON. It's a terribly inefficient and difficult to parse format which translate to wasted cycles dealing with it which matters when there's a lot of them. I read ridiculous things like omg it handles 200 updates a minute with a thousand end-points it's great!

The target capability is a thousand updates per second with ten's of thousands of end-points.

There's no need for ED to actually have a feed along the lines of a ZeroMQ network. All they need to do is expose the data for the current commander at the current location - exactly as the iOS app backend already does but without the silly email verification.

The ZeroMQ network (EDDM or otherwise) can be left to the community, as not everyone will want to listen to it.

 

[Andrew Sayers]

If I had to do it, all I'd suggest would be a feature which adds a form of HTML browser from a known file location.

This is perhaps a more relevant point to the current discussion. If Frontier made their new Android/Windows app web-based (i.e. the app itself is just a full-screen browser that loads http://elitedangerous.com/webapp), it would make an excellent reference implementation. And releasing the app under a permissive license would make it relatively easy for me to e.g. use their CSS to ensure my own projects fit the game's style.

 

[wolverine2710]

Another idea, maybe too far out there to do but food-for-thought, is for FD to roll the EDDN functionality into the core server and then it could stream market updates to clients that connect kinda like the NYSE or LSE does.

(Note that they make heavy use of private networks with multi-casting enabled - which we cannot do over the Internet.)

So we'd have an API to query for known system and station data which could include last-known-good commodity data and then a streaming service for market data. Part of the data you query for is to get the system-station ID which is like a stock-ticket but probably just a GUID perhaps modified to have a couple digits to reflect the galactic sector it's in. Then in the streaming service you use the fixed-size ID (not variable size strings) to send data and now you can easily do it with a binary format.


I would like to register a loud vote against JSON. It's a terribly inefficient and difficult to parse format which translate to wasted cycles dealing with it which matters when there's a lot of them. I read ridiculous things like omg it handles 200 updates a minute with a thousand end-points it's great!

The target capability is a thousand updates per second with ten's of thousands of end-points.

Read it yesterday, decided to sleep over it for a night before adding a post. I TRULY like the idea of FD rolling out EDDN functionality. For Commanders who know me a bit (see profile) this does not come as a surprise ;-) Lets call the FD version FDDR (FD Data Relay). In this case they only have to create a ZeroMQ (ZMQ) PUB (publisher), which is VERY easy to create. It would have a few advantages.

• Prevents poisoning the well. With EDDN it relies on multiple sources. Commanders against EDDN could send bogus info to EDDN. Just like the BPC in the past was fed was stuff like Javacakes and bogus prices. This could ofc be handled by EDDN (login for example) but it would make EDDN a bit more complicated and less K.I.S.S.
• Less bandwidth. If multiple commanders enter a station and open the commodities market normally with EDDN both commanders (if they participate in EDDN) would upload that info to EDDN - which then directly distriubutes it. FD could decided to send the info only once the info IF the market hasn't changed when multiple commanders in the same station open the market.

I DO agree that it should be an opt-in feature. As in commanders have an option in ED to have FD send their commodities prices to FDDR, which can then be read by subscribed tools/commanders. Some commanders don't want to share their info because they fear it will hurt their discovered trade runs - which is of course fine.

EMDR handles approx 1 Million messages a day (almost 12 messages per second). Its very scalable, robust, redundant. EDDN (which is heavily based upon EVE's EMDR) and FDDR can use this setup. I haven't logged in to the EDDN server to see the current bandwidth usage. MagmaiKH wrote: "The target capability is a thousand updates per second with ten's of thousands of end-points". I don't think that is true. Let me try to explain.

I think most commanders are not going to subscribe to FDDR, just like they don't all connect to EDDN. Commanders use trading tools like the BPC, Thrudd. These tools connect to FDDR and store the info locally in a database. The BPC, Thrudd website accesses that database. BPC/Thrudd atm get their data from OCR-ed screens and manual input. Hoping they output/upload to EDDN soon ;-) For tools which run locally without a huge database (for older/historical data) like Trade Dangerous its a bit different. Normally they should connect 24/7 to FDDR/EDDN to stay up to date. EDDN is just a simple relay, it doesn't store info. The mission of EDDN is clear and a simple one, K.I.S.S. principle. It relies on value added services (VAS) to make it more useful. One such service is created by Maddavo which has a merge tool for TD. It connects to EDDN. TD commanders can get the latest up to date .prices file for TD from within TD by a certain sub command. When TD commanders upload their manual entered (or OCR-ed) prices to the merge tool they update the merge tool. With EDDN atm we have at least two commanders who connect to EDDN AND make the stored historical data available to other commanders - so it can be used in their tools. I'm sure more services will be created. EDDB is being created. I would be surprised if in the end more then 100,200 commanders would connect directly to FDDR. Others will probably use the various VA services out there. A tool I like and which is being created by a commander (who liked the idea) for EDDN is the equivalent of EVE's EMDR map - See the solar systems light up as market data arrives.

If FD is concerned about bandwidth usage they could limit access to FDDR. As in for example only EDDN (or a few more) can subscribe to it. The community then connects to EDDN and its up to EDDN to be scalable, redundant enough. The community would provide the needed bandwidth!!!. In the case of EDDN we were very lucky. Commander Errantthought (thanks a lot man) pitched the idea to his boss and we got free hosting services. Its described on the wiki page of EDDN - to prevent possibly violating forum rules here. See this high-level overview of EMDR to get an idea of how this is done in EMDR. The discussed announcers/relays are visualized in the tool EMDR monitor - EMDR relay/announcer monitor web app. It would cost FD very little bandwidth in this setup - limit access to FDDR to a few tools like EDDN.

TLDR; Yes I hope FD creates EDDN functionality into the core server and then it could stream market updates to clients. It even has a few advantages over EDDN....

Edit: Yes in hindsight I should have chosen a different name. EDDR (Elite Dangerous Data Relay) would have been a more fitting name then EDDN (Elite Dangerous Data Network) as EDDN does not store anything. Ah well live and learn ;-)

Edit2: ZMQ supports multicast. Wrt ZMQ performance, see here. Snippet of info: "As for bandwidth usage we are able to get to 1Gb/sec boundary for messages 128 bytes long. 2Gb/sec boundary is passed for messages 1024 bytes long and bandwidth of 2.5Gb/s is reached for messages 32kB long" AND "Throughput gets to the maximum of 2.8 million messages per second for messages 8 bytes long".

To make this absolutely totally clear: I'm NOT the one who implemented EDDN. That excellent work has been done by commander jamesremuscat. Thanks again for all your hard your work James!!! I'm just the guy who liked the original EMMD idea by commander Andreas (heavily based on EMDR) and tried to see it revived....