Discussion Commanders log manual and data sample

[wolverine2710]

Wednesday 20/07/2016, 6:53 PM Zac Antonaci created this thread and gave us FD's proposal for what they call a "Player Journal"/“Commanders log”. Which basically is a client-side API (CS-API). Thursday 21/07/2016, 11:48 AM the Elite Dangerous Community Developers (EDCD) responded very quickly and came with suggestions (post #66) to enhance the CS-API. The EDCD has their own Discord server. FDev programmer Howard Chalkley was assigned as a spokesman and its the man coding the stuff - possible with help from other FDev programmers. Howard has said that a new JSON format and stuff like FDev id's as suggested will be implemented - coming from EDCD and this thread. Along with other suggestions made here. We can look forward to a revised CS-API this week. What is completely unclear what exactly will be added.

Since Thursday a lot of new suggestions (some by tool authors) were made. We also see lots of duplicates. I think its NOT realistic to expect that FD is going through this thread to figure what exactly was suggested and to weed out the duplicates and group things together. I think that should be done by the Community. We should limit us at this point to the fact that one or more Line delimited JSON files will be created with events aka journal entries which reflect what a user does. The EDCD format looks solid, as in explain with a rationale why it should be done. And perhaps a column which (existing) tools should benefit from this.

I don't have the time to do this myself. So I'm hoping that one or more volunteers could do that. Perhaps using a googdle doc document for the creation/editing. Pesonally I think it should not be editable by everyone - to make sure pranksters don't ruin our work, as he been done in the past.

I think that when we have a good document with suggestions it would definitely increase the change that more community input would potentially be implemented by FD. I'm confident that we as a community can make this happen. The hard part for FD is creating the base. Adding extra events to that in most case is relatively speaking quite easy.

 

[hchalkley]

Another request: List of possible values (within reason?) Things like StationType, Faction names?
For parameter such as commander name, can we get the maximum number of characters that can be in a commander's name?

Digging out the full range of possible string values for multiple value types would take some time - and as for faction names, there are over 75000 in the database.

The maximum commander name string appears to be 20 characters.

 

[themroc]

Digging out the full range of possible string values for multiple value types would take some time - and as for faction names, there are over 75000 in the database.

The maximum commander name string appears to be 20 characters.

I believe we could bring a lot of great functionalities to our tools if we had up2date minor faction data. Currently all sites with this information are heavily outdated, since people can't possibly keep 60k+ stations updated.

 

[jgm]

Digging out the full range of possible string values for multiple value types would take some time - and as for faction names, there are over 75000 in the database.

The maximum commander name string appears to be 20 characters.

I don't think that obtaining the full list of faction names is particularly important or useful, given that they change periodically. The items that I think would be useful would be your names for ships, modules, station types, etc. Basically, whatever keys you write out in the journal. We can then set up translations for them so users see suitable values.

And one other request: would it be at all possible for one or two people to have access to the internal builds prior to you going to beta, or alternatively an understanding that if we do find issues in beta that they will be looked at as a matter of priority? The addition of the journal will be a springboard for increased functionality for existing tools and the creation of many others, but if there are serious issues with it and they aren't addressed it will probably be the death knell for many community efforts.

 

[Bill]

I just had a thought. Would it be possible to log the current music file or indicate the mood somehow?

It would make it possible to write a contextual jukebox using the user's preferred music files.

 

[Iskariot]

Greetings Commanders,

As you may have heard in the 3^rd party developers discord or at Lavecon we have a new journal feed being added into game for 2.2 which developers can use to create a player Journal or “Commanders log”. What we would like is to give you information and support ahead of time to ensure that you are able to understand and utilise the functionality from day 1 of release.

To do this we have two downloadable links. One has a manual, explaining how the journal works. The other is a small set of data to get you started.

More data can be compiled later but we felt it was best to get the first set of info out so people can begin understanding the functionality of the Commanders Log.

What’s more, the very amazing Howard Chalkley (who is far more intelligent than I) will be on hand in this thread to answer any questions that you have regarding the journal.

As always, post on this thread and feel free to get in touch directly with any questions, concerns or recommendations.

Thanks again!

===

Manual
http://hosting.zaonce.net/community/journal/Journal_Manual.doc

Data sample
http://hosting.zaonce.net/community/journal/Journal.160719102509.01.log

I really, really, really wish you guys would add a fully featured ingame CMDRs log accessible from the HUD.

 

[Esvandiary]

I just had a thought. Would it be possible to log the current music file or indicate the mood somehow?

It would make it possible to write a contextual jukebox using the user's preferred music files.

This gave me a good chuckle.

It was satire, right?

 

[Genar-Hofoen]

I really, really, really wish you guys would add a fully featured ingame CMDRs log accessible from the HUD.

*sigh*

Once again - this sort of comment is off topic for this thread.

This thread is here purely for app developers to discuss with FDEV the contents of the new Journal file. Not to have a gripe about <insert favourite missing feature here>.

Signal. Noise. There is a difference.

Besides which - think about all the innovation done by every single 3rd-party web site and application. The sheer amount of creativity going on is incredible, and amounts to thousands of man-hours of work which as you well know, FDEV cannot commit to as they're busy adding other features to the game.

So, if you have something valuable to add, like the information to be included in the new Journal file, or the format of the file, etc. then by all means please add to the collective wisdom in this thread. If you have some gripe - go create a thread elsewhere.

Thank you.

 

[Iskariot]

*sigh*

Once again - this sort of comment is off topic for this thread.

This thread is here purely for app developers to discuss with FDEV the contents of the new Journal file. Not to have a gripe about <insert favourite missing feature here>.

Signal. Noise. There is a difference.

Besides which - think about all the innovation done by every single 3rd-party web site and application. The sheer amount of creativity going on is incredible, and amounts to thousands of man-hours of work which as you well know, FDEV cannot commit to as they're busy adding other features to the game.

So, if you have something valuable to add, like the information to be included in the new Journal file, or the format of the file, etc. then by all means please add to the collective wisdom in this thread. If you have some gripe - go create a thread elsewhere.

Thank you.

Sorry I made you sigh.

 

[Purrcat259]

Digging out the full range of possible string values for multiple value types would take some time - and as for faction names, there are over 75000 in the database.

The maximum commander name string appears to be 20 characters.

Thanks for the feedback! In the mean time I got a sizeable list of systems / bodies / faction names from EDDB - I was also asking for the limits as I am writing a journal generator so that other devs can test/debug log parsing performance before 2.2.

Keep up the good work

 

[wolverine2710]

Thanks for the feedback! In the mean time I got a sizeable list of systems / bodies / faction names from EDDB - I was also asking for the limits as I am writing a journal generator so that other devs can test/debug log parsing performance before 2.2.

Keep up the good work

Brilliant idea ;-) Wrt to Json parsers. There can be a lot of difference between the default JSON parser/generator for a certain language and third party ones. Did some digging this weekend but haven't made a post about it. RapidJson (c++) seems to be pretty darn quick. There is also a python wrapper for it. Ofc from their own website and without knowing the hardware used and the JSON file (size) its a bit useless. Still they write the following: "A community member reported that RapidJSON in their system parses 50 million JSONs daily". Thats about 1,728 msec for one JSON. Not to shabby.

 

[wolverine2710]

JSON schema - good or bad for the Journal line-delimited Json structures??

Would a JSON schema for the JSON journal be beneficial or bad? Given the fact that the journal will contains lots of different JSON structures. What are your thoughts? I'm a Java dev and have used XML and JSON before but I'm certainly NO expert. Tbh haven't used JSON schema before. Perhaps there are some JSON experts amongst us...

Some links I found:

• https://en.wikipedia.org/wiki/JSON#Schema_and_metadata
• http://json-schema.org/documentation.html
• https://stackoverflow.com/questions...s-of-using-strongly-typed-data-interchange-fo

 

[Artie]

Please no further discussion about JSON format for the journal, I think it was solved already to the point it is usable and there is no need to open it again and clutter this thread. It will be better to have rather some "real data" output for the log in the given developer's time than to confuse dev with ever-changing ideas for the log format change.

 

[wolverine2710]

Please no further discussion about JSON format for the journal, I think it was solved already and there is no need to open it again and clutter this thread. It will be better to have rather some "real data" output for the log in the given developer's time than to confuse dev with ever-changing ideas for the log format change.

I could be mistaken but I followed this thread from day one and Afaik JSON schemas never came up. It was not my intention to clutter this thread with a JSON discussion. My apologies if you felt that way. I do post frequently here, that is certainly true.

 

[Genar-Hofoen]

Please no further discussion about JSON format for the journal, I think it was solved already to the point it is usable and there is no need to open it again and clutter this thread. It will be better to have rather some "real data" output for the log in the given developer's time than to confuse dev with ever-changing ideas for the log format change.

I concur.

As far as I am concerned the JSON format is settled - line-delimited JSON for the win. That's all we need to know. Let Mr Chalkley get on with the coding for the journal now.

As far as JSON parsers are concerned - that's up to the individual programmers and we tend to know our stuff on that already.

Regards

 

[Micha]

Query: Wouldn't it be slightly easier on the system (as in client PC) to send the data out via a local UDP broadcast instead of writing (and flushing) to a file on every event?

It would be trivial for the community to write a caching logger, which can dump (some of) the data to disk as required.

It seems to me that quite a bit of requested data will swamp the Journal (such as turning various ship equipment on/off or switching between screens in-game), assuming these requests will be implemented.

Just a thought.

Alternatively/also, could players (and/or apps) specify -which- events they would like logged? Again, some players may not want to use all of the functionality and have a premium of diskspace (eg, laptop + SSD).

 

[Genar-Hofoen]

Query: Wouldn't it be slightly easier on the system (as in client PC) to send the data out via a local UDP broadcast instead of writing (and flushing) to a file on every event?

It would be trivial for the community to write a caching logger, which can dump (some of) the data to disk as required.

It seems to me that quite a bit of requested data will swamp the Journal (such as turning various ship equipment on/off or switching between screens in-game), assuming these requests will be implemented.

Just a thought.

Alternatively/also, could players (and/or apps) specify -which- events they would like logged? Again, some players may not want to use all of the functionality and have a premium of diskspace (eg, laptop + SSD).

Yes that's not a bad idea at all - I was thinking of suggesting something similar last week - but got caught in the JSON file momentum.

One of the other advantages of a UDP broadcast would be that you could have clients listening on other, different PC's, Macs, Android phones, Raspberry Pi's, wherever!

I too believe that some less powerful systems might find all the writing/flushing of a journal file might prove to be too much of a burden.

Configuring which events you want to know about might also be a good idea. Why not have both? Or even a choice between keeping a journal file, UDP broadcast, both if your system can handle it.

I don't know if it's too late for the UDP broadcast idea though - I certainly hope not. What say you, H Chalkley?

Regards o7

 

[Andargor]

Yeah, we've been talking about the client serving an API via TCP or UDP (for low-latency telemetry, for example).

Actually, if Mr. Chalkley would just open a port, hook into those events he was talking about, and pump them out a port, the community can write JSON loggers for him...

All is needed is the format.

 

[wolverine2710]

I think both journal log and TCP/UDP broadcasts have their merits. Some data is possibly better suited for UDP/TCP like telemetry (low latency). Not that we have that much (real-time) telemetry in the proposal atm and parsing a line-delimited small JSON structure could be quite fast/efficient. Other data is best kept in a persistent journal. A huge advantage of the journal log (like the current netlogs) is the HISTORY aspect. Nothing is lost. Its logged even when commanders doesn't use third party tools at a certain moment. Multiple tools (like captains logs apps) can parse the journal logs and display the route and other stats for a cmdr. With only TCP/UDP that data would be lost - unless a cmdr is running a third party tool all the time to create a Json file. Wrt disk usage. If not done automatically by FD when a new log is created when the current one becomes to large 3P tools tools could comprress with for example gzip the data. 3P tools could be made to remove from the logs unneeded data for a cmdr etc etc. Atm its not clear what FD has planned for the future wrt the client-side API and which direction they want to go. More journal log data, more telemetry etc. I'm hoping for flexibility and a default solution by FD and I hope they will send the journal log data JSON ALSO to a UDP/TCP port. This would enable the community to perform their magic and come up with all kind of superb solutions. All we need is data and very very preferably in two formats - persistent logs and UDP/TCP port.

 

[Artie]

Yeah, we've been talking about the client serving an API via TCP or UDP (for low-latency telemetry, for example).

Actually, if Mr. Chalkley would just open a port, hook into those events he was talking about, and pump them out a port, the community can write JSON loggers for him...

All is needed is the format.

But it means that anybody who wants just a log will need to run a third party app to get it and it will do just the same as the game does. There is no real benefit from it (that app needs to write stuff onto disk anyway, so why it shouldn't be done by the game directly?). But, as Wolverine wrote above, making both (log file and UDP) makes perfect sense - in the log should be the current stuff that is useful for import to third party apps/sites, UDP can have extra "real-time" events (like current ship temperature, ammo left, map opened, current coords, whatever) which aren't very suitable for the written log, for the apps that somehow extends player's experience during the game.

Mixing needs for "the valuable data to import" and "the valuable data for realtime apps" into a single solution is a road to hell in my opinion.