Missing API / journal entries

sathlei · Aug 19, 2022

Elite Dangerous has an extensive ecosystem of external user-developed tools/services. Many late game aspects are virtually impossible without them.
The current set of Player Journal (v3.1) events limits possible third-party tools functionality and inhibits growth of many organic player-led in-game activities (such as player economy).

Therefore, I'd like to initiate/re-start discussion on the most needed Elite API / Player Journal additions with these two proposals:

Fleet Carries
- commodity trades (individual buys and sells; commodity, quantity, price, user)
stations
- mission board - although payments are user specific, missions or mission types should be identical

DrinkyBird · Aug 19, 2022

All that FC information is provded by https://companion.orerve.net/fleetcarrier isn't it?

sathlei · Aug 19, 2022

Thanks for a reminder about cAPI

I haven't used it before and it seems, the extended cAPI journal information should allow to monitor Fleet Carrier trades (excluding trade party - but this is minor issue).

fd-api/FrontierDevelopments-CAPI-endpoints.md at main · Athanasius/fd-api

Code dealing with Frontier Developments' API(s) for Elite Dangerous - fd-api/FrontierDevelopments-CAPI-endpoints.md at main · Athanasius/fd-api

github.com

Do you know any service or tool that provides such functionality based on cAPI information?

sathlei · Aug 24, 2022

It took me a moment to figure out how to work with cAPI (which uses OAuth 2.0 authorization with the Proof Key for Code Exchange (PKCE) extension), but the initial effects are promising for FC trade monitoring.

Although no individual trades are recorded, it seems possible to reconstruct them based on changes in FC market and cargo states.

A nice surprise was finding that individual additions to the FC cargo are stored. This allows to distinguish between bought-for-sale and mined cargo.

Example:

1. total FC cargo:

Code:

     commodity item_price total_qty total_value
1:     Tritium  48165.146      3698   178114711
2:      Silver   3486.776      5100    17782560
3:        Gold   4049.870       661     2676964
4: Alexandrite 217277.000        12     2607324
5:  Musgravite   9927.000       112     1111824
6:   Palladium      0.000        49           0
7:     Painite      0.000         4           0

2. mined FC cargo:

Code:

   commodity total_qty
1: Palladium        49
2:      Gold        44
3:    Silver        12
4:   Painite         4

3. aggregate FC market/cargo snapshot:

Code:

     commodity total_qty mined_qty total_value item_price FC_sell_price FC_sell_margin FC_expected_profit
1:      Silver      5100        12    17782560   3486.776         32062       28575.22          145733640
2:  Musgravite       112         0     1111824   9927.000        609931      600004.00           67200448
3:        Gold       661        44     2676964   4049.870         39981       35931.13           23750477
4: Alexandrite        12         0     2607324 217277.000        548864      331587.00            3979044
5:   Palladium        49        49           0      0.000         55032       55032.00            2696568
6:     Painite         4         4           0      0.000        269051      269051.00            1076204
7:     Tritium      3698         0   178114711  48165.146            NA             NA                 NA

I'm at the beginning of exploring cAPI for monitoring FC trades, and I cannot definitely say the available information is sufficient for the purpose. Having individual FC trades returned by cAPI would greatly simplify the task, but I'm aware of the challange of handling a huge number of such trades.

On the other hand, as noted before, individual FC cargo additions are stored. This suggests some extension of the cAPI FC market data may be possible.

eikenb · Sep 19, 2022

+1 to adding the mission board information to the API. That'd be super useful.

varonica · Sep 20, 2022

eikenb said:
+1 to adding the mission board information to the API. That'd be super useful.

Mission board information changes based on many things, time, system state, even the pilots reputation with the controlling factions, I don't see how it would be possible to provide that and keep it updated as it changes so often, it would be....impossible....

sathlei · Sep 23, 2022

It took me some time to create a tool for monitoring the most of fleet carrier related trades based on the information available though cAPI and Player Journal.

cAPI presents only a snapshot of fleet carrier's cargo and ongoing trades at a given moment. No information about the changes between snapshots (i.e. trades) is available. One can infer trades based on the changes in the fleet carrier cargo, FC market offers and FC trade balance.

Some transactions may get lost for low data refresh and pulling frequency. Assigning trade time and often star system is impossible.

FC:

Code:

    monitor_id           timestamp               system_name        commodity total_qty_diff  item_price  buy_price sell_price
38:         39 2022-09-14 15:22:51                   HR 8514           osmium             56   4429.7297      0.000         NA

actual trade:

Code:

    monitor_id           timestamp        system_name      commodity total_qty_diff item_price buy_price sell_price
10:         NA 2022-09-02 10:12:05            LHS 157         osmium           -389         NA      4004     361584

Fleet carriers are not only (primarily?) used as trading venues but also for storage and transport of commodities for selling at the space stations. Such "external" (to the fleet carrier) trades are partly recorded in the Player Journal. For the above mentioned low data refresh issue, the major challenge was matching cargo transfers between FC and ship with the subsequent station sales.

FC cargo changes:

Code:

    monitor_id           timestamp        system_name             commodity total_qty_diff item_price  buy_price sell_price
 1:          4 2022-08-24 19:14:22          Apotenile             palladium            -49      0.000         NA      55032
 2:          4 2022-08-24 19:14:22          Apotenile                silver          -1511   3486.776         NA      32062
 3:          5 2022-08-24 20:35:07          Apotenile                silver           -784   3483.314         NA      32062
 4:          6 2022-08-24 22:19:32          Apotenile                silver           -784   3480.048         NA      32062
 5:          7 2022-08-25 01:36:25          Apotenile                silver           -784   3474.248         NA      32062
 6:          9 2022-08-25 03:20:14          Apotenile                silver          -1237   3461.095         NA      32062

actual trades:

Code:

    monitor_id           timestamp        system_name             commodity total_qty_diff item_price  buy_price sell_price
 1:         NA 2022-08-24 18:51:02          Apotenile         silver           -727         NA      3495      49319
 2:         NA 2022-08-24 18:51:02          Apotenile      palladium            -57         NA      3654      59475
 3:         NA 2022-08-24 19:08:31          Apotenile         silver           -784         NA      3495      49319
 4:         NA 2022-08-24 20:06:50          Apotenile         silver           -784         NA      3495      49319
 5:         NA 2022-08-24 21:49:25          Apotenile         silver           -784         NA      3495      49319
 6:         NA 2022-08-24 22:40:41          Apotenile         silver           -784         NA      3495      49319
 7:         NA 2022-08-25 02:02:54          Apotenile         silver           -784         NA      3495      49319
 8:         NA 2022-08-25 02:15:53          Apotenile         silver           -453         NA      3402      49319

All recent FC-related trades reconstructed:

Code:

    monitor_id           timestamp        system_name      commodity qty buy_price sell_price  value    pnl
 1:         NA 2022-08-24 18:51:02          Apotenile         silver 727  3495.000      49319  35.85  33.31
 2:         NA 2022-08-24 18:51:02          Apotenile      palladium  57  3654.000      59475   3.39   3.18
 3:         NA 2022-08-24 19:08:31          Apotenile         silver 784  3495.000      49319  38.67  35.93
 4:         NA 2022-08-24 20:06:50          Apotenile         silver 784  3495.000      49319  38.67  35.93
 5:         NA 2022-08-24 21:49:25          Apotenile         silver 784  3495.000      49319  38.67  35.93
 6:         NA 2022-08-24 22:40:41          Apotenile         silver 784  3495.000      49319  38.67  35.93
 7:         NA 2022-08-25 02:02:54          Apotenile         silver 784  3495.000      49319  38.67  35.93
 8:         NA 2022-08-25 02:15:53          Apotenile         silver 453  3402.000      49319  22.34  20.80
 9:         19 2022-08-27 15:53:52 Synuefai EB-R c7-5           gold 120  4102.718      39981   4.80   4.31
10:         NA 2022-08-28 17:31:21 Synuefai EB-R c7-5 xihecompanions  27  3945.000      22116   0.60   0.49
11:         NA 2022-09-02 10:12:05            LHS 157         osmium 389  4004.000     361584 140.66 139.10
12:         NA 2022-09-11 14:55:44              Dread       platinum 673   766.000     245107 164.96 164.44

Here, only a single trade (9) was executed through FC market. All the others were completed at space stations.
Transaction values and PnL (profit&loss) in millions.
Low osmium and platinum "purchase" prices result from mining.

To summarize, even that fleet carrier trades reconstruction is mostly possible based on the data available in the Player Journal and through cAPI, recording and making available fleet carrier trades would be extremely helpful.

Currently, all player trades are recorded in the Player Journal. For fleet carriers' persistence, such information should be accessible through cAPI.

sathlei · Sep 24, 2022

varonica said:
Mission board information changes based on many things, time, system state, even the pilots reputation with the controlling factions, I don't see how it would be possible to provide that and keep it updated as it changes so often, it would be....impossible....

It's true missions depend on various factors and additionally change at each server refresh. But why player-specific mission board information cannot be saved when player accesses the board, in similar way to saving market information?

sathlei · Sep 27, 2022

I'd like to add one more candidate for API extension:

extending "Bounty" events already present in Player Journal with mission information

Currently, it is not possible to match bounties with the accepted missions (like massacre missions). Therefore, tracking mission progress is not possible based on the available information outside the game. I've checked information availability for massacre missions, but most problably this affects other combat missions.

Code:

{ "timestamp":"2022-09-26T09:51:32Z", "event":"Bounty", "Rewards":[ { "Faction":"LHS 3804 Nobles", "Reward":12652 } ], "Target":"empire_eagle", "Target_Localised":"Imperial Eagle", "TotalReward":12652, "VictimFaction":"LP 932-12 Society" }

LHS 3804 Nobles is the largest faction in LP 932-12 system, where bounty was earned. This bounty is connected with a massacre mission taken from some other faction in Gliese 868. This record does not allow to assign it to the mission.

Situation is better with trading/delivery missions, since appropriate information is present in cargo-related records:

Code:

{ "timestamp":"2020-11-01T14:55:54Z", "event":"CargoDepot", "MissionID":646439237, "UpdateType":"Collect", "CargoType":"PerformanceEnhancers", "CargoType_Localised":"Performance Enhancers", "Count":2, "StartMarketID":128937008, "EndMarketID":128932533, "ItemsCollected":2, "ItemsDelivered":0, "TotalItemsToDeliver":2, "Progress":1.000000 }

sathlei · Oct 3, 2022

Seems, no kills are recorded in Player Journal during fight in Combat Zone. Just targeting messages, music events and tons of NPC and other users' messages (skipped below).

Another missing information to be included in API.

Code:

{ "timestamp":"2022-09-30T09:30:17Z", "event":"ShipTargeted", "TargetLocked":true, "Ship":"cobramkiii", "Ship_Localised":"Cobra Mk III", "ScanStage":3, "PilotName":"$ShipName_Military_Independent;", "PilotName_Localised":"System Defence Force", "PilotRank":"Master", "ShieldHealth":0.000000, "HullHealth":0.000000, "Faction":"Workers of Popontia Free", "LegalStatus":"Lawless" }

{ "timestamp":"2022-09-30T09:30:18Z", "event":"ShipTargeted", "TargetLocked":false }

{ "timestamp":"2022-09-30T09:30:19Z", "event":"Music", "MusicTrack":"Combat_Dogfight" }

{ "timestamp":"2022-09-30T09:30:21Z", "event":"Music", "MusicTrack":"Combat_CapitalShip" }

{ "timestamp":"2022-09-30T09:30:25Z", "event":"ShipTargeted", "TargetLocked":true, "Ship":"typex_3", "Ship_Localised":"Alliance Challenger", "ScanStage":3, "PilotName":"$ShipName_Military_Independent;", "PilotName_Localised":"System Defence Force", "PilotRank":"Competent", "ShieldHealth":93.154129, "HullHealth":100.000000, "Faction":"Workers of Popontia Free", "LegalStatus":"Lawless" }

{ "timestamp":"2022-09-30T09:30:40Z", "event":"ShipTargeted", "TargetLocked":false }

{ "timestamp":"2022-09-30T09:30:40Z", "event":"Music", "MusicTrack":"Combat_Dogfight" }

sathlei · Oct 6, 2022

Another missing API data:

FC Universal Cartographics transactions

Going back for a moment to the FC-related data available through cAPI, I would like to stress that the current mechanism is quite cumbersome and results in misleading informations.

The reason for that is the server refrtesh cycle and the scope of data made available. Basically, a snapshot of the FC martket & cargo available through cAPI is refreshed every 10-15 minutes. What happens between the data refrershes get lost. Provided your FC jumps in the refresh window, transactions performed in one system will be recognized in the another. Multiple transactions are bundled, and any price changes are lost.

I will try to show later what kind of insight can be gained from the available data, as a taste of what could be achieved when better data is available.