When will we see a proper API implementation, and not this writing to log files nonsense?

[robbyp]

Well, any game that is any good has 3rd party tools being developed for it. We at EDD are very happy with the lovely journal, the way its written, and the amount of data provided. We are very happy with Frontier and their support, which is spot on. Any change to it now would cause massive disruption to the people actually writing 3rd party tools.

Frontier also have valid reasons to limit certain data rates/types. They don't want you creating a autopilot!

 

[Ozric]

Promised?

Or just drop the whole "API" nonsense altogether instead of wasting development resources on it.

I completely agree with this.

Given that Frontier said that they didn't want people to use the API for the companion app and then people went ahead and used it anyway. Then they were told to stop using it, they refused and carried on developing. Then there was the disgraceful behaviour by some 3rd party developers last year...

To be honest I'd count yourselves lucky that Frontier even have put as much effort into the journal as they have!

 

[Lustan]

Shared memory is very specific, very big can of worms for network game like ED....Please let's not invent issue here. Load is very minimal.

Not a developer, then says something opens can of worms and follows that with "not invent issues". If you can't defend Frontier without conflicting yourself in your own messages you obviously are blindingly biased.Why don't you take a moment from the game and reflect on why there were kickstarter refunds, why people gamed rep grinds early on, why people flock to money grabs, and why Obsidian Ant's opinion, who was actually invited by Frontier Dev to their campus, matters more than yours. Reflect on other's people desired ways to play the game instead of believing yours in the only way. Or don't and keep up that stroke.See what people like you don't realize is WE are the people who drive Frontier to make the game better. We make the difference. People like you who "accept whatever is given" don't understand is you aren't doing Frontier any service by saying "Everything is fine". All you do is encourage this game to die a slow death by not adapting the game. By all means, encourage Frontier to be stagnant. That will ensure that the game will remain exactly as you like for the next couple years until Frontier finds it not viable to maintain as the rest of the player base quits.

 

[Ulaidian]

I think they should have a datapad app made, that will connect to your kindle/tablet whatever, and give you all the info you need from maps etc I play on Xbox, so API stuff is not available for us (I think).. but an app.. well it can be used by any account holder on any platform.

 

[Esvandiary]

Given that Frontier said that they didn't want people to use the API for the companion app and then people went ahead and used it anyway. Then they were told to stop using it, they refused and carried on developing. Then there was the disgraceful behaviour by some 3rd party developers last year...

There are numerous inaccuracies in those first few statements... but the situation is much better now, and the excellent journal and the new market/shipyard/status files are slowly replacing the CAPI, which I think is a happy thing for all involved!

Anyway, IMO the journal is a good solution for the type of data it writes, and the choice of files for the non-historical shipyard/market/status data is valid - especially from a standpoint of minimising the barrier to entry for new developers trying to get started.

 

[robbyp]

I don't think this thread is going to get anywhere is it?

Obv. Frontier, having devoted one of their best programmers to the journal API ;-), and spending so much time and money doing it, and working with the community, are not going to drop it or replace it.

Whatever one feels about us 3rd party developers, a lot of people use our tools. EDSM for instance has millions of page hits per month. All of them rely on each other to feed data to each other. EDD has many thousands of users feeding the data network, as does EDMC, Captains Log etc etc etc. Most people have used Corolis or ED Shipyard to look at their loadout? All of these tools are being supported by Frontier now very well.

So they obv. want it to continue.

 

[Artie]

Promised?
I completely agree with this.

Given that Frontier said that they didn't want people to use the API for the companion app and then people went ahead and used it anyway. Then they were told to stop using it, they refused and carried on developing. Then there was the disgraceful behaviour by some 3rd party developers last year...

To be honest I'd count yourselves lucky that Frontier even have put as much effort into the journal as they have!

Well, as one of the so-called disgraceful developers I can say just this: It is incredible pain to maintain tools for such data-rich game as Elite is without any support and without any way how to get the data. Does anybody remember days, where all the stuff like market prices, star system information, etc. were written by hand or via OCRed screenshots? It was frustrating, tedious "job" and I doubt anybody wanted to do that for a long time. I dare to say that without any Frontier support and without any way how to get data and other stuff 3rd party tools simply cease to exist.

You may say - I don't care, I don't need it. It's a valid point. But, there are many players that simply won't be playing Elite anymore without the 3rd party tools (their words). Not because they want to "cheat", but simply because Elite is too complicated for them otherwise (and it may be caused just by the core game principles, not by the lack of some features in the game). Which, obviously, means less players in the game with all its consequences.

From that point of view it seems very reasonable to me that the game developer will dedicate some resources to support 3rd party developers that cares about the game, about the community and about the players, which may help the game to grow and keep the interest in it. In the end, it may be just beneficiary for everybody, even if they are not using any of the 3rd party tools provided.

 

[sollisb]

There are numerous inaccuracies in those first few statements... but the situation is much better now, and the excellent journal and the new market/shipyard/status files are slowly replacing the CAPI, which I think is a happy thing for all involved!

Anyway, IMO the journal is a good solution for the type of data it writes, and the choice of files for the non-historical shipyard/market/status data is valid - especially from a standpoint of minimising the barrier to entry for new developers trying to get started.

What barrier? If you don't know what you're doing, then don't do it, and leave to the people that do. Simples.

 

[Ulaidian]

Do the millions )) of Xbox and Ps3.1 players have a 'journal' we can use?

 

[Esvandiary]

What barrier? If you don't know what you're doing, then don't do it, and leave to the people that do. Simples.

Uh, OK... God forbid anyone try to learn anything...

 

[sollisb]

... and also against the ToS, for pretty good reason.


Yes, quite a reasonable fear factor. Currently their position can be very clear and unambiguous: if you are scraping game memory you are breaking the ToS and could quite reasonably be punished for doing so. If FD were to explicitly allow external applications to read game memory:

• It's essentially game over from an anti-cheat perspective, as it becomes massively more difficult for them to detect bad actors reading or modifying game memory when anything is allowed to map it in the first place
• Support queries related to third-party apps would become nightmarish as apps try to read memory wrongly, or just memory that's no longer valid, or ...
• It puts additional restrictions on memory locations of objects (since presumably they have to be predictable now...?) which could impact performance and, again, anti-cheat mechanisms
• It massively increases the barrier to entry for new developers wanting to try it out since you need to know how to map memory in another process, and the worst case goes from "I failed to read some JSON" to "I accidentally took out the game client"

I've made other suggestions before such as a local UDP stream, but the current file-based implementation is significantly simpler from an implementation and support perspective - as well as being easier for the casual end user - and Just Works.

Woah there.. Reading memory is not against ToS. My machine, I can read whatever I like. Writing to memory of the game however is a completely different story. That of course would breach ToS.

Anti-Cheat, they already have that in the form of recording all player action at server side, so, anyone attempting to cheat will be found out quite quickly.

Nothing to support for 3rd party apps. Are FD supporting my third party apps? No! They publish the data or give access to it, and then their end is done. They don't need to 'supporting' and other 3rd party apps.

Additional Restrictions on memory. No.. If they do it right, the queries they right will already be pointing tot he right place.

It massively increases entry level for new developers... Why? I don't understand this; so because new developers might have problems using the mechanics, no-one else should have them?

UDP Stream?? Why would you go near UDP ??

Uh, OK... God forbid anyone try to learn anything...

Anyone can learn can whatever they want. They do it on their own dime. Having a real API would also spawn a community of developers, and from that would be spawned a standard query system.

 

[sollisb]

Do the millions )) of Xbox and Ps3.1 players have a 'journal' we can use?

Those are consoles, so the architecture of any 3rd party tools would have to be developed by 'console savvy' developers. I'm not one, so I can't speak too intelligently about it. Or was your question going somewhere else?

 

[sollisb]

I don't think this thread is going to get anywhere is it?

Obv. Frontier, having devoted one of their best programmers to the journal API ;-), and spending so much time and money doing it, and working with the community, are not going to drop it or replace it.

Whatever one feels about us 3rd party developers, a lot of people use our tools. EDSM for instance has millions of page hits per month. All of them rely on each other to feed data to each other. EDD has many thousands of users feeding the data network, as does EDMC, Captains Log etc etc etc. Most people have used Corolis or ED Shipyard to look at their loadout? All of these tools are being supported by Frontier now very well.

So they obv. want it to continue.

That's perfectly acceptable! The tools I'd developed would not interfere or replace those tools.

 

[Ulaidian]

No mate, just wondering if we could have the same stuff that you are asking for I like all the 3rd party stuff, loved it when I could look on my phone and see where my AC Black Flag things were etc. It can be done, but rather that have a fear of opening the box, perhaps another box should be made, and it used for the API (or app) info.

 

[DocPossible]

You're going to have to persist stuff to disk anyway and speaking as an aging dev myself, I love log files. Human readable ones at least. Other games make extensive use of them and there are vast numbers of tools out there that parse them. Text files can be read by any app, any OS, any language. Once you provide an API, you're forcing everyone to use your choice of 'P' - Java? C? Python? R? .NET? JSON?

The older I've got, the more I like simple stuff that's hard to break. Just don't go changing the format and everything will be lovely.

 

[Esvandiary]

Woah there.. Reading memory is not against ToS. My machine, I can read whatever I like. Writing to memory of the game however is a completely different story. That of course would breach ToS.

So if you're silent running and I read the memory containing your exact position and status so I can see exactly where you are... That's fine, is it? I think not...

Anti-Cheat, they already have that in the form of recording all player action at server side, so, anyone attempting to cheat will be found out quite quickly.

Incorrect. Moment-to-moment gameplay is entirely P2P, with authority shared between the clients in the instance.

Nothing to support for 3rd party apps. Are FD supporting my third party apps? No! They publish the data or give access to it, and then their end is done. They don't need to 'supporting' and other 3rd party apps.

... and when FD support are spending time investigating weird crashes they can't explain because a third-party app got something a bit wrong ...

Additional Restrictions on memory. No.. If they do it right, the queries they right will already be pointing tot he right place.

So you want them to just point us at the exact structures in memory that various realtime information is stored in for use by the game client? You do realise how much of a cheater's paradise you'd be creating here, right? Can you name a single other multiplayer game that does something remotely similar?

UDP Stream?? Why would you go near UDP ??

Low-overhead method of sending small pieces of data to other applications, available in every language/framework under the sun. Same machine so loss isn't really a problem, but doesn't have the connection overhead or sequencing requirements of TCP. You could go for OS-specific IPC like named pipes if you prefer.

Anyone can learn can whatever they want. They do it on their own dime. Having a real API would also spawn a community of developers, and from that would be spawned a standard query system.

There is a community of developers. They're currently using the mechanisms provided, which are adequate for a wide range of tools that many people make use of.

 

[Ozric]

There are numerous inaccuracies in those first few statements... but the situation is much better now, and the excellent journal and the new market/shipyard/status files are slowly replacing the CAPI, which I think is a happy thing for all involved!

Anyway, IMO the journal is a good solution for the type of data it writes, and the choice of files for the non-historical shipyard/market/status data is valid - especially from a standpoint of minimising the barrier to entry for new developers trying to get started.

Numerous, maybe just poorly written? People did start reverse engineering the companion API before Frontier had given permission didn't they? Frontier did pull the companion app and asked people not to use the API as it wouldnt be supported didn't they? (thier mistake was not pulling the API then) People carried on using it despite that didn't they?

Well, as one of the so-called disgraceful developers I can say just this: It is incredible pain to maintain tools for such data-rich game as Elite is without any support and without any way how to get the data. Does anybody remember days, where all the stuff like market prices, star system information, etc. were written by hand or via OCRed screenshots? It was frustrating, tedious "job" and I doubt anybody wanted to do that for a long time. I dare to say that without any Frontier support and without any way how to get data and other stuff 3rd party tools simply cease to exist.

You may say - I don't care, I don't need it. It's a valid point. But, there are many players that simply won't be playing Elite anymore without the 3rd party tools (their words). Not because they want to "cheat", but simply because Elite is too complicated for them otherwise (and it may be caused just by the core game principles, not by the lack of some features in the game). Which, obviously, means less players in the game with all its consequences.

From that point of view it seems very reasonable to me that the game developer will dedicate some resources to support 3rd party developers that cares about the game, about the community and about the players, which may help the game to grow and keep the interest in it. In the end, it may be just beneficiary for everybody, even if they are not using any of the 3rd party tools provided.

I think you're putting words into my mouth a bit here. I didn't say the developers were disgraceful, but the actions of some certainly were in my opinion. I do remember indeed what it was like having helped Thrudd and Finwen in Beta and at release of the game. I also continue to use EDDiscovery and by extension EDSM.

People want to and enjoy putting their time into writing 3rd party tools, I get and understand that and I don't have a problem with it when done within the bounds outlined by the company making the game, not going against their wishes. I think the journal is a great tool and it has been improved immeasurably over the last year, but it will never be enough for some people.

 

[Esvandiary]

Numerous? People did start reverse engineering the companion API before Frontier had given permission didn't they? Frontier did pull the companion app and asked people not to use it as it wouldnt be supported didn't they? People carried on using it despite that didn't they?

Yes, people did start reverse engineering the companion API without permission, which is somewhat unfortunate. They also asked Frontier at the same time whether use of the CAPI was something they considered acceptable. Frontier then said they were discussing it, and devs continued to ask over time without answer, before finally getting a reply 8 months later saying that use of it was permitted.
I wasn't really part of the developer community at that point, but I do believe that had Frontier ever officially stated its use was disallowed, that the developers would have respected such a request.

More to the point, though... when they removed the companion app from the store, why would they have left the companion API active at all, if not to enable the community to continue to access it?

I think you're putting words into my mouth a bit here. I didn't say the developers were disgraceful, but the actions of some certainly were in my opinion. I do remember indeed what it was like having helped Thrudd and Finwen in Beta and at release of the game. I also continue to use EDDiscovery and by extension EDSM.

Well, the developers of both of those tools were among those whose actions (I assume) you're referring to as disgraceful. You're of course welcome to hold that opinion though; it was never exactly going to be a universally popular course of action, and rightly so.

People want to and enjoy putting their time into writing 3rd party tools, I get and understand that and I don't have a problem with it when done within the bounds outlined by the company making the game, not going against their wishes. I think the journal is a great tool and it has been improved immeasurably over the last year, but it will never be enough for some people.

If you look back through this thread, you'll find that all of the established tool developers who have posted in it are generally supportive of the current state of affairs. The journal is excellent, and it continues to get better. In my experience the third-party developers are very respectful of Frontier's wishes regarding what they do and don't want us to do - this is much easier now that they communicate such things more regularly.
It's fair to say that Frontier have really turned a corner in the opinion of most third-party devs (and feel free to look out for a letter soon-ish from EDCD to that effect).

 

[sollisb]

So if you're silent running and I read the memory containing your exact position and status so I can see exactly where you are... That's fine, is it? I think not...


Incorrect. Moment-to-moment gameplay is entirely P2P, with authority shared between the clients in the instance.


... and when FD support are spending time investigating weird crashes they can't explain because a third-party app got something a bit wrong ...


So you want them to just point us at the exact structures in memory that various realtime information is stored in for use by the game client? You do realise how much of a cheater's paradise you'd be creating here, right? Can you name a single other multiplayer game that does something remotely similar?


Low-overhead method of sending small pieces of data to other applications, available in every language/framework under the sun. Same machine so loss isn't really a problem, but doesn't have the connection overhead or sequencing requirements of TCP. You could go for OS-specific IPC like named pipes if you prefer.


There is a community of developers. They're currently using the mechanisms provided, which are adequate for a wide range of tools that many people make use of.

Firstly there would be no need to write to the game, so there wouldn't be client crashes. If the published queries were published by FDev then they'd be de-facto-standard, and they could limit what was and wasn't available.
Cheater's Paradise? Like how? All queries would be related to the current CMDRs data. No data about any other CMDR would be available.
I built a 5 server system to run a Flight Sim 737 Cockpit, utilising 5 screen for data management, 250+ switches, and all handled in realtime. I don't think I'll have any 'sequencing issues'. I currently read the JSON logs and display results on my MacBookPro.
The current community of developers, while doing fantastic work, are limited to the data that is sent to them. I'm saying it [the available data] could be a whole lot better, leading to even more apps.

Here's an Example;
I already have a service running to activate when my log file changes, which sends the changed data to any listener services registered to my main EDService.

I land in a system and I want to display on one of my screen, all stations, available landing pads and all bodies in the system.
To do that, I have to extract the system I just landed in, send that to EDDB and then parse the results from EDDB, then send it to the listener to display it on screen.

Can you see how having easily accessible API would reduce all that fluff?

 

[GraXXoR]

Best way to do this?! Are you kidding me? Are we back in 1984? This is the worst way to do this! All the information the log files provide should be provided through shared memory.

Writing to log files for 3'rd party apps is not only lazy programming, but incurs unnecessary load on the computer. If they wanted to stick to wring to log files, then at least create a ram disk, and periodically mirror the files to the HDD.



Not sure where you're going with this. Everything has bugs. Even a simple "Hello Word" application can have bugs.

You're right. This is not 1984... My pocket PC could handle the load of writing to a log file...

I run EDDiscovery and there is literally NO LAG in events occurring and them being picked up by EDDiscovery.

Edit:
The Only One of the only reasons why an API is better than a log and ironically, the reason you didn't pick up is that a log is output only. An API could allow for input, too.

Anyway, log file load on SSD is laughable compared to downloading a 25Gb game file.... My log directory is less than 1Gb and I've played this game A LOT.