The alpha version for this 64-bit migration is now available - https://github.com/EDCD/EDMarketConnector/releases/tag/Release/5.5.1-alpha0
See the message above for what needs testing.
See the message above for what needs testing.
- Status
If we're just a casual user that doesn't do testing and the like, should we just leave ours alone for now?
Yes, this is tagged '-alpha0' for a reason.
Major heads-up for forthcoming very necessary update to EDMC
First of all, if you've not heard the big news about Update 14, go read.
TL;DR - There'll be an EDMC update in the next few days that all users should ensure they have installed before Update 14 is deployed next week so as to ensure their data can actually be useful to EDDN Listeners.
Now, what does this mean for EDMC? For the application itself no changes should be necessary. For third-party plugins ... well if you keep track of anything per Cmdr then you'll probably want to start paying attention to
Any EDDN Listener is going to need to know if a message's data pertains to the Legacy or the Live galaxy. Many are probably going to choose to only process data for the Live galaxy going forwards. That means all EDDN Senders need to provider information to allow the Listeners to discern which galaxy a message is from. EDDN recently added new header fields "gameversion" and "gamebuild" for just this scenario.
Adding those fields naively to EDMC would have been easy, but there's a wrinkle when it comes to "failed on first attempt to send" and "delayed due to the setting to not send until docked" messages. Those are currently stored in a flat file,
So, at the very least that code would have needed changing to also store gameversion/build. I decided to just go the whole hog and start storing the full, ready to send over the network, message instead. And if I was going to have to convert to a new format to cater for that I decided I might as well change it from a flat file (inefficient, only written to disk periodically, so you can lose messages if you crash) to an sqlite3 database. At startup if a replay.jsonl file is found all of its entries will be converted into the sqlite3 database (
Going forwards all EDDN messages from EDMC after version 5.5.0 (the current release) will include "gameversion" and "gamebuild" as specified in the EDDN documentation. Note what that says about the values for CAPI-sourced data, or where the version/build couldn't be determined for any reason.
NB: So as to not dump the need for a different update on third-party plugin developers this next version of EDMC will still be 32-bit. We'll handle the move to 64-bit in a later 6.0.0 release once the dust from this has settled, and third-party plugin developers have had time to react to the 5.5.1-alpha0 pre-release which is 64-bit.
I'm still waiting to hear from EDSM and Inara.cz developers to see if any changes will be necessary when sending data to their APIs.
First of all, if you've not heard the big news about Update 14, go read.
TL;DR - There'll be an EDMC update in the next few days that all users should ensure they have installed before Update 14 is deployed next week so as to ensure their data can actually be useful to EDDN Listeners.
Now, what does this mean for EDMC? For the application itself no changes should be necessary. For third-party plugins ... well if you keep track of anything per Cmdr then you'll probably want to start paying attention to
gameversion (we track it as state['GameVersion'] passed to journal_entry()) so as to tell if the data is for the Cmdr's Legacy or Live profile.Any EDDN Listener is going to need to know if a message's data pertains to the Legacy or the Live galaxy. Many are probably going to choose to only process data for the Live galaxy going forwards. That means all EDDN Senders need to provider information to allow the Listeners to discern which galaxy a message is from. EDDN recently added new header fields "gameversion" and "gamebuild" for just this scenario.
Adding those fields naively to EDMC would have been easy, but there's a wrinkle when it comes to "failed on first attempt to send" and "delayed due to the setting to not send until docked" messages. Those are currently stored in a flat file,
replay.jsonl in the configured 'Output' file location. Unfortunately the code that writes messages to that file only stores the Commander name, schemaref and the "message". Missing from this is the EDDN "header", which is where these new gameversion/build fields go (and they'd not have been present/set in current messages anyway).So, at the very least that code would have needed changing to also store gameversion/build. I decided to just go the whole hog and start storing the full, ready to send over the network, message instead. And if I was going to have to convert to a new format to cater for that I decided I might as well change it from a flat file (inefficient, only written to disk periodically, so you can lose messages if you crash) to an sqlite3 database. At startup if a replay.jsonl file is found all of its entries will be converted into the sqlite3 database (
eddn_queue-v1.db in the same folder), and then the file deleted. All of these 'legacy' messages will have " (legacy replay)" appended to the EDDN header->softwareName, and will have empty strings for both "gameversion" and "gamebuild" (because there's no way to be sure of what version they came from... no I'm not going to start figuring out the date/time ranges for each just for this).Going forwards all EDDN messages from EDMC after version 5.5.0 (the current release) will include "gameversion" and "gamebuild" as specified in the EDDN documentation. Note what that says about the values for CAPI-sourced data, or where the version/build couldn't be determined for any reason.
NB: So as to not dump the need for a different update on third-party plugin developers this next version of EDMC will still be 32-bit. We'll handle the move to 64-bit in a later 6.0.0 release once the dust from this has settled, and third-party plugin developers have had time to react to the 5.5.1-alpha0 pre-release which is 64-bit.
I'm still waiting to hear from EDSM and Inara.cz developers to see if any changes will be necessary when sending data to their APIs.
This is all too complicated for me. I use EDMC mainly just to populate EDSM. I will only be using the Live version of 4.0. What do I need to do to update EDMC, and when?
Just accept the update, it'll, likely be called 5.6.0, when it becomes available sometime in the next few days (I'm hoping to get it out on Friday latest so that the weekend players pick it up).
As you're obviously paying attention to this forum thread (at least now), I'll definitely post a reply (and edit the OP) when the release is available, at which point you can manually "Help" > "Check for Updates..." to kick it off. It would eventually be offered automatically anyway (assuming you've not disabled that), the program checks for an update if it's been at least 8 hours since it last did so.
Just install the new version when it's released, and all will be taken care of
Oh, can't I just run the update from within the opened app? That's what I normally do. It should uninstall the old one and install the new one itself, I hope?Just install the new version when it's released, and all will be taken care of
To be clear, I mostly wanted to give users a heads up that they definitely want to install this forthcoming update if they care about their data being useful to the third-party sites.
Also, if you've still been playing on 3.8 by choice because you couldn't get used to the 4.0 UI now might be the time to reconsider that.
Also, if you've still been playing on 3.8 by choice because you couldn't get used to the 4.0 UI now might be the time to reconsider that.
It's business as usual. As @Athan said, just make sure you install the new version when it shows up. No need to search for it on your own.
Small tweak to the plan. As the underlying changes to the EDMC's eddn code are larger than "start sending gameversion" would suggest I'm waiting on the other developer to complete a code review of the changes. So, I'll put out a beta today, with the full release hopefully tomororow, Sunday latest.
I have just made a PRE-release of updated ED Market Connector to aid third-party sites it communicates with when Update 14 goes live:
github.com
Please test if able. As this is a pre-release it will not be offered automatically.
If no (new in this release) bugs come to light then I'll put out the full release 5.6.0 tomorrow around 12:00 UTC.
Note in particular the consequence of Inara.cz's decision to only have Live galaxy data on its API. That means Legacy commanders will no longer be able to update their account via EDMC.
Release Release/5.6.0-beta1 · EDCD/EDMarketConnector
Pre-Release 5.6.0-beta1 We now test against, and package with, Python 3.10.8. The code for sending data to EDDN has been reworked. This changes the 'replay log' from utilising an internal arra...
github.com
Please test if able. As this is a pre-release it will not be offered automatically.
If no (new in this release) bugs come to light then I'll put out the full release 5.6.0 tomorrow around 12:00 UTC.
Note in particular the consequence of Inara.cz's decision to only have Live galaxy data on its API. That means Legacy commanders will no longer be able to update their account via EDMC.
I need to adjust the EDSM plugin code, so 5.6.0 release will be a little later today now.
Release 5.6.0
Tha major reason for this release is to address the Live versus Legacy galaxy split coming in Update 14 of the game.See the section "Update 14 and the Galaxy Split" below for how this might impact you.
Changes
- We now test against, and package with, Python 3.10.8.
- The code for sending data to EDDN has been reworked. This changes the 'replay log' from utilising an internal array, backed by a flat file (replay.jsonl), to an sqlite3 database.
As a result:- Any messages stored in the old
replay.jsonlare converted at startup, if that file is present, and then the file removed. - All new messages are stored in this new sqlite3 queue before any attempt is made to send them. An immediate attempt is then made to send any message not affected by "Delay sending until docked".
- Sending of queued messages will be attempted every 5 minutes, unless "Delay sending until docked" is active and the Cmdr is not docked in their own ship. This is in case a message failed to send due to an issue communicating with the EDDN Gateway.
- When you dock in your own ship an immediate attempt to send all queued messages will be initiated.
- When processing queued messages the same 0.4-second inter-message delay as with the old code has been implemented. This serves to not suddenly flood the EDDN Gateway. If any message fails to send for Gateway reasons, i.e. not a bad message, then this processing is abandoned to wait for the next invocation.
Currently, the feedback of "Sending data to EDDN..." in the UI status line has been removed.
If you do not have "Delay sending until docked" active, then the only messages that will be at all delayed will be where there was a communication problem with the EDDN Gateway, or it otherwise indicated a problem other than 'your message is bad'. - Any messages stored in the old
- As a result of this EDDN rework this application now sends appropriate
gameversionandgamebuildstrings in EDDN message headers.
The rework was necessary in order to enable this, in case of any queued or delayed messages which did not contain this information in the legacyreplay.jsonlformat. - For EDSM there is a very unlikely set of circumstances that could, in theory lead to some events not being sent. This is so as to safeguard against sending a batch with a gameversion/build claimed that does not match for all of the events in that batch.
It would take a combination of "communications with EDSM are slow", more events (the ones that would be lost), a game client crash, and starting a new game client before the 'more events' are sent.
Update 14 and the Galaxy Split
Due to the galaxy split announced by Frontier there are some changes to the major third-party websites and tools.- Inara has chosen to only accept Live galaxy data on its API.
This application will not even process Journal data for Inara after 2022-11-29T09:00:00+00:00 unless thegameversionindicates a Live client. This explicitly checks that the game's version is semantically equal to or greater than '4.0.0'.
If a Live client is not detected, then there is an INFO level logging message "Inara only accepts Live galaxy data", which is also set as the main UI status line. This message will repeat, at most, every 5 minutes.
If you continue to play in the Legacy galaxy only then you probably want to just disable the Inara plugin with the checkbox on Settings > Inara. - All batches of events sent to EDSM will be tagged with a
gameversion, in a similar manner to the EDDN header.
Ref: EDSM api-journal-v1 - All EDDN messages will now have appropriate
gameversionandgamebuildfields in the header as per EDDN/docs/Developers.md.
As a result of this you can expect third-party sites to choose to filter data based on that.
Look for announcements by individual sites/tools as to what they have chosen to do.
Known Bugs
In testing if it had been broken at all due to 5.5.0 -> 5.6.0 changes it has come to light thatEDMC.EXE -n, to send data to EDDN, was already broken in 5.5.0.In addition, there is now some extra 'INFO' logging output which will be produced by any invocation of
EDMC.EXE. This might break third-party use of it, e.g. Trade Computer Extension Mk.II.This will be fixed as soon as the dust settles from Update 14, with emphasis being on ensuring the GUI
EDMarketConnector.exe functions properly.Notes for EDDN Listeners
- Where EDMC sourced data from the Journal files it will set
gameversionandgamebuildas per their values inFileheaderorLoadGame, whichever was more recent (there are some events that occur between these). - If any message was already delayed such that it did not have the EDDN header recorded, then the
gameversionandgamebuildwill be empty strings. In order to indicate this the softwareName will have(legacy replay)appended to it, e.g.E:D Market Connector Connector [Windows] (legacy replay). In general this indicates that the message was queued up using a version of EDMC prior to this one. If you're only interested in Live galaxy data then you might want to ignore such messages. - Where EDMC sourced data from a CAPI endpoint, the resulting EDDN message will have a
gameversionofCAPI-<endpoint>set, e.g.CAPI-market. At this time it is not 100% certain which galaxy this data will be for, so all listeners are advised to ignore/queue such data until this is clarified.
gamebuildwill be an empty string for all CAPI-sourced data.
Plugin Developers
- There is a new flag in
statepassed to plugins,IsDocked. See PLUGINS.md for details.
VirusTotal
Pre-emptive upload to VirusTotal for EDMarketConnector_win.5.6.0.msi.
Last edited:
I just applied this update, and now my updates aren't showing up on Inara. Don't know if this is a problem with me, EDMC, EDDN, or Inara! I've mentioned this on Inara's forum as well. EDDB is not updated either.
I just visited and uploaded data for all these stations:
I just visited and uploaded data for all these stations:
Wyndham Dock | MCC 549︎ 
| M | 1,567 Ls | 0 Ly | 36,712 | 207 Cr | 266 days ago |
Pippin Dock | MCC 549︎
| M | 2,006 Ls | 0 Ly | 44,965 | 207 Cr | 91 days ago |
Biggle Dock | MCC 549︎
| L | 1,233 Ls | 0 Ly | 25,555 | 207 Cr | 81 days ago |
It appears you did not, however, I did. (My messages are from 2022-11-29 00:58). Are you in Odyssey or Horizons 4.0? Is there anything in the debug log?
As forbiddenlake says, there's not much I can do outside of wild guesses unless you open up a proper bug report with EDMarketConnector logs.
Okay I've done that, thanks for the link! I am a software developer myself so please let me know if there's anything else I can do.
Whilst the servers are down, just dropping in to say "thank you" - EDMC tool is invaluable, and it obviously takes a not insignificant amount of time/ effort to maintain
- Status