Discussion Commanders log manual and data sample

Yeah, I don't know where the sweet spot between fine-grained data and bandwidth would lie. Perhaps the game should aggregate high frequency events such as multicannon/frag cannon/railgun/AFMU/damage and output an AmmoConsumed/DamageTaken event every 5 seconds (if changed)? For the other items, the frequency of occurrence is low enough not to care IMO.

For your uses, perhaps you should filter out events you don't care about client side before uploading to Inara?
Maybe another file for high-frequency data or UDP port as was mentioned above should be more suitable for this. The filter on the user-side may lead to complications in my opinion.
 

wolverine2710

Tutorial & Guide Writer
I agree on the timestamp in ISO, plus kept it in UTC time, it's the easiest to deal with all timezone available by players.
It is up to 3rd party tool to convert at will.

As for the complete JSON, making it update in real time, will only push some trouble on the client side, having to reparse the whole thing at each update will become hard at some point just to check for new events.
Having one JSON objet per line, seems more accommodating to real time, as we can only open the file to the desired line.


EDCD is currently writing a doc with all these things, so we should soon release it for review.


Sounds like a plan ;-)

Not directly related to the JSON structure. Hopefully it will also include a recommendation to have the existing companion API info also exposed to JSON files. As in ships and outfitting data and of course market data. As also mentioned earlier by a little red devil ;-) Unfortunately I've to work today and can't help. That pesky Java software enigineering I have for a living always gets in the way.........
 
Last edited:
Sounds like a plan ;-)

Not directly related to the JSON structure. Hopefully it will also include a recommendation to have the existing companion API info also exposed to JSON files. As in ships and outfitting data and of course market data. As also mentioned earlier by a little red devil ;-) Unfortunately I've to work today and can't help. That pesky Java software enigineering I have for a living always gets in the way.........

It is in the doc ;)
 
This is amazing. I however request the addition of the following events...

Hardpoint Deploy/Retract
Landing Gear Deploy/Retract
Cargo Scoop Deploy/Retract
Lights Toggle On/Off
Silent Running On/Off
Auto Rotation On/Off
Flight Assist On/Off
Orbit Lines On/Off
Turret Mode (SRV) On/Off
Handbrake (SRV) On/Off
Report Crimes On/Off
Beacon Wing/Off
Navigation Lock Enable/Disable (with name).


I'd really like to see the deployables (Landing Gear, Hardpoints, Cargo Scoop, Silent Running, Auto Rotation, Flight Assist and Lights) on this list as you could do some really cool things with voice control and advanced commands. So I hope those make the cut. :)

Thanks for including what we have though!
 
I would prefer if the format used "self-contained" stanzas, where each logged event is its own object. Consistency and order can be assured through timestamps (see below) and serial numbers that are unique to an event within a given "session". That way, parsers don't need to keep state but can just read a line, parse it, and add the result to their own internal model store. They also don't need to keep parser state for the entire log document around all the time, and it might also make restarting tools or re-opening log files easier.

Please use international format for datetime stamps (ISO 8601) and they should be saved in UTC or game time. The tool/interface using it should be responsible for converting to local time.

E.g. 2012-04-23T18:25:43.511Z
That cannot be stated often enough. Standards, use them. ISO formats also support millisecond resolution which may be nice to have at some point, so I would suggest just dumping at that kind of resolution.

Something like:

Code:
{ "session": { "start_time":"2016-07-19T10:25:33.224Z", "part":1, "gameversion":"2.2 (Trunk)", "build":"r113955", "id":"eac44261" }}
{ "event": { "session":"eac44261", "time":"2016-07-19T10:25:35.163Z", "type":"LoadGame", "Commander":"HRC1", "Ship":"Asp" }}
{ "event": { "session":"eac44261", "time":"2016-07-19T10:25:35.822Z", "type":"Rank", "Combat":0, "Trade":0, "Explore":1, "Empire":0, "Federation":0, "CQC":0 }}

would be a bit more verbose, but would IMO make for better integrity. It would also deflate pretty well.
 
Last edited:
I write this post on behalf of EDCD (Elite Dangerous Community Developers). We are a group of developers that create and maintain 3rd party tools and sites for Elite: Dangerous. If you want to get in touch, join our Discord Server.

First of all - big thanks for the journal! We all agree that this is a big step forward and we are happy that the game and our tools can grow closer together :D. We analyzed the manual very thoroughly and came up with a bunch of ideas which we would love to see implemented.


Entry
PriorityRequestRationale
allHighFlush the journal file after each new entry (or set the unitbuf flag if using C++ iostream).Ensure that the journal entries aren't delayed by I/O buffering in the client from being made visible to apps.
allHighOne JSON object per-entry/line, rather than one per-file. Timestamps shouldn’t be used as keys.In the current proposal the file contents aren't well formed until the file is closed, and not even then if some timestamps are duplicates (invalid JSON).
allHighUse ISO 8601 format for timestamps, in UTC.
E.g.: 2012-04-23T18:25:43.511Z
Plain localtime is a pain to deal with, and needs special handling over midnight.
ISO 8601 is both human readable and unambiguously machine-parsable, doesn't need special handling over midnight, and is understood by JSON parsers.
newHighJournal should list user’s current state at startup, on ship swap, on resurrection and on shutdown of:
- current ship type
- current ship loadout (including crafting state of modules)
- cargo
- materials
- credits, debts and assets
- types and locations of other ships
Journal currently (mostly) lists only events/diffs. This makes it difficult for apps to reliably track the current state of the user over an extended period, and impossible to do so in the face of a discontinuity - e.g. the user plays on a different machine, intervention by FDev Support, loss of network connection (?), etc.
Also, for some use-cases just the overall status at the end of play is of interest.
allMediumUnique ship identifier/ID for all ship records (ShipyardBuy, Sell, Swap, LoadGame, etc.)It is not possible to pair and synchronize ship data with data stored in 3rd party apps automatically without this information.
2.2 HeadingMediumIndication of whether the game is beta or production.Apps may wish to store beta data separately to production, and/or discard data after beta ends.
3.5 RankMediumInclude major faction reputation.
Include Powerplay rank.
Include CQC prestige.
Include CQC sub-rank (numerical).
4.1 DockedMediumInclude additional station parameters like: wealth, population, economies, government, allegiance, controlling faction including its influence/state, services, list of illegal goods, and if it is a controlling/main station of the systemThis kind of information is currently entered manually by users on community sites/apps. Which is unreliable, incomplete and tiresome.
(NB: Controlling faction is included in the sample log but not in the manual).
4.2 FSDJump
4.4 Location
MediumInclude additional star system parameters like: government, allegiance, security, permit and controlling faction. Eventually, also list of all factions in the system and their influences/states. Similar to station information, this is currently entered manually by users on community sites/apps.
(NB: Controlling faction is included in the sample log but not in the manual).
4.4 LocationHighInclude station name if docked.No way for the app to know whether and where the user is docked otherwise.
4.5 Liftoff
4.7 Touchdown
MediumInclude local body coordinates (lat/long) where the ship is touching down at or taking off fromThis enables better tracking of ship’s location, rather than only knowing which body
6.1 ScanMediumInclude basic info from the local map i.e.:
- Distance from arrival point
- Earth / Solar mass
- Radius / Solar radius
- Gravity (for planets/moons)
- Number of rings
Write this entry even if the user doesn’t have a detailed surface scanner (but in which case omit the Materials property).
This basic info is relevant to the user (or an app on behalf of the user) in deciding whether it's worthwhile (or safe!) to plan to go prospecting on a body. Also, this kind of information is currently entered manually by users on community sites/apps.
6.1 ScanLowAlso include all info available from the local map, i.e.:
- Atmosphere gasses and percentages
- Composition and percentages
- Orbital period
- etc
This kind of information is currently entered manually by users on community sites/apps. Which is unreliable, incomplete and tiresome.
6.1 ScanHighDiscrete values for discrete statesFrom the description sample it looks like we have to parse the text for specific properties like atmosphere. We highly recommend to use separate discrete properties with numeric values for unique states.
3.3 LoadGameHighInclude game mode being enteredUseful for e.g. PowerPlay groups, to have estimates of which modes their members are going into. Also it allows apps to filter out data from Training/CQC.
4.1 DockedLowInclude starsystemThe app can use “Location” and “FSDJump” entries to keep track of the user’s starsystem, but safer to list the starsystem here too.
2.2 HeadingLowInclude client version number.Allows apps in the future to process historical data differently depending on which version of the client wrote the data. Which shouldn’t be necessary, but might be handy to allow for.
(NB: "gameversion" and “build” properties are included in the sample log but not in the manual).
new


_____________
Nice to haveJournal should duplicate all functionality of the Companion API. Notably:
- Commodity market listing of prices, demand and stock
- Outfitters listing
- Shipyard listing
Making the Companion API redundant would mean that the user no longer has to supply their Frontier login details to 3rd-party apps.

Sorry for the huge list but after all we are developers, right? [big grin]



Alot - EDTS
Anthor - EDSM, ED-TD
Artie - Inara
Biobob - EDCodex, ED-Board
Finwen, Robby - EDDiscovery
Genar_Hofoen - Captain's Log
OtisB - EDMC
themroc - EDDB
... and and the other EDCD developers who couldn't make it :)
 
Last edited:
A few more things I'd like to see. Events prefixed with an asterisk are new.



  • Startup
    • Heading
    • Please rename it 'Header' - 'Heading' sounds like it is talking about where the ship is pointing and
    • Use ISO 8601 timezone offsets rather than 'GMT Daylight Time' or similar (is that a Windows thing?) Timezone abbreviations are non-canonical and your platform is certainly able to output date/times in standard formats. In fact, why bother outputting a localdatetime and gmt_time keys, when you could just output localtime with a utc offset, and a named timezone, if it's that important to know which timezone the player was in.
  • Travel
    • FSDJump
      • Add a 'FuelUsed' key giving the number of tonnes of fuel used in a jump
      • Add a 'SystemState' key giving the current state (War, Boom, etc) of the system entered
      • Add a 'SystemSecurityLevel' key
    • Location
      • Add a 'SystemState' key giving the current state (War, Boom, etc) of the system entered
      • Add a 'SystemSecurityLevel' key
    • *DockingPermissionRequested: StationName
    • *DockingPermissionDenied: StationName, Reason
    • *DockingPermissionGranted: StationName, PadNumber
    • *DockingPermissionExpired: StationName
  • Combat
    • Interdicted
      • Name of interdictor
    • Interdiction
      • Name of interdictee
    • WarrantScanPerformed
      • Name
      • Bounty level
    • CargoScanPerformed
      • Name
      • CargoItems:[Type,Count]
    • ScanReceived
      • NameOfScanner
      • ScanType
 
Apologize for the question, but what exactly this post mean?? What we can use these info for? I dont understand......
 
Apologize for the question, but what exactly this post mean?? What we can use these info for? I don't understand......

This is about FDEV providing information about your game to other programs made by Elite: Dangerous players.

Elite Dangerous (The game) will write a file to your drive with detailed information about what you do in the game while you play. The file is a log of what you do in game.
Other programs made by players can then read that file and use the information to store in a database for future look up or present the information to you (the player) in a 3rd party program while you play.
And probably much much more.
 

Code:
{ "session": { "start_time":"2016-07-19T10:25:33.224Z", "part":1, "gameversion":"2.2 (Trunk)", "build":"r113955", "id":"eac44261" }}
{ "event": { "session":"eac44261", "time":"2016-07-19T10:25:35.163Z", "type":"LoadGame", "Commander":"HRC1", "Ship":"Asp" }}
{ "event": { "session":"eac44261", "time":"2016-07-19T10:25:35.822Z", "type":"Rank", "Combat":0, "Trade":0, "Explore":1, "Empire":0, "Federation":0, "CQC":0 }}

Good example! Agreed on the timestamp being ISO8601 UTC!


I feel like I should expand on my previous suggestion, why I think "complete file as JSON" is problematic even if it does have benefits.

Having the complete log file as JSON certainly makes it trivially simple to read and parse the log as complete entity. As sort of an archived end-of-session report, the current proposal certainly would be great.

For listeners interested in (almost) real-time consumption of events, the current format would be cumbersome. Due to the possibly high frequency of events, having a real-time listener read and parse the complete file on each event is, in my opinion, a huge waste of processing power and memory. The triviality of loading the import thus becomes a cost in terms of hardware and implementation complexity. Only partially reading newly added event data using a custom parser that can deal with the possibly incomplete (and even syntactically incorrect) JSON increases the complexity for third-party-developers and they would gain nothing from the "complete file as JSON" approach.

I also feel the "complete file as JSON" goes against the nature of the data, since it's event/message based. In a way, the log file becomes a message queue buffer. From this perspective, JSON object per event/per line offers much simpler and more appropriate structure. This way one can read any one line and parse from JSON without special implementations. This makes it trivially simple to listen to the file and extract the newest event from it. It does however become slightly more complicated when loading and parsing the file in full. In this case though, in my opinion, the complexity for third-party-developers interested in full reports is much much less, than compared to the complexity of extracting single events in real-time for the "complete file" approach.

In my opinion, both approaches are equally readable for programmers.

I don't know how complex the current proposed log format is to develop for Frontier, but in my experience (Python, PHP, Java, and a few others), the line-based approach would be entirely implementable using standard logging libraries with one custom log formatter, whereas the complete-file-approach probably uses the same "load, decode, add, encode, write" algorithm with all side-effects as described above.

Some quotes I find relevant :)


  • "KISS - Keep It Simple Stupid"
  • "Simple things should be simple, complex things should be possible." - Alan Kay
  • "Everything should be made as simple as possible, but not simpler." - Albert Einstein
 
Last edited:
Just to reiterate themroc's post above: thank you very much for adding this, it'll be a great help to application developers of many different stripes.

I don't know how complex the current proposed log format is to develop for Frontier, but in my experience (Python, PHP, Java, and a few others), the line-based approach would be entirely implementable using standard logging libraries with one custom log formatter, whereas the complete-file-approach probably uses the same "load, decode, add, encode, write" algorithm with all side as described above.
At a guess, they aren't rewriting the whole file every time, just appending as they go - so it wouldn't be valid JSON until the play session ended and the ending tags were appended.
... which now I think about it, presumably leaves open the possibility of invalid files if the game crashes.

I'm strongly in favour of the line-based approach, simply because IMO the overhead developers would have reading in the whole file at a time is minimal - instead of "parse one string of JSON, iterate over the elements" it's "iterate over and parse n strings of JSON"; not much effective difference.
This way the file could be "valid" at all times, as soon as each line is written.
 
Last edited:
This is amazing. I however request the addition of the following events...

Hardpoint Deploy/Retract
Landing Gear Deploy/Retract
Cargo Scoop Deploy/Retract
Lights Toggle On/Off
Silent Running On/Off
Auto Rotation On/Off
Flight Assist On/Off
Orbit Lines On/Off
Turret Mode (SRV) On/Off
Handbrake (SRV) On/Off
Report Crimes On/Off
Beacon Wing/Off
Navigation Lock Enable/Disable (with name).


I'd really like to see the deployables (Landing Gear, Hardpoints, Cargo Scoop, Silent Running, Auto Rotation, Flight Assist and Lights) on this list as you could do some really cool things with voice control and advanced commands. So I hope those make the cut. :)

Thanks for including what we have though!

This would be amazing, then we could link Voice Attack with AutoHotkey and a custom DLL to do .. well, plenty of cool stuff!
 
Nice... So when are we getting an in game UI for this?

Checked my computer logs last night to figure out what system I had a bounty in so I could go pay it off. [haha]
 
I wouldn't say that, because TCE is an in-game UI and would make use of nearly all provided data. ;)

Sorry, I assumed an FD-created in-game "flight log" UI; community-created overlays etc could of course make use of this data (as could things like Voice Attack). My mistake!
 
← Occasionally guilty of being a bit snarky.

I'm glad that you guys have fun messing around with this stuff, but I'd much prefer more player gaming experience solutions provided natively in the game itself.

I will more than likely never run third party executables for this game, but do make use of web apps occasionally.

Cheers.
 
I will more than likely never run third party executables for this game, but do make use of web apps occasionally.
And the web apps get their data from other players running third-party executables. If you use the web apps you should consider contributing.
 
The web apps I use don't use user data, at least not directly. But yeah, I agree it's nice to support them. [up]
 
The web apps I use don't use user data, at least not directly. But yeah, I agree it's nice to support them. [up]
If you use any of the trading and outfitting sites like eddb, Thrudd's, ED-TD, Inara, ETN, Roguey's etc or EDSM for route plotting then yes they do rely on data submitted by users running 3rd party apps like EDMC.
 
Last edited:
Top Bottom