Release EDDiscovery 18.X is now out! New modules panel upgrade and Engineering

[Eahlstan]

You're killing the processes by force. This can damage the DB, so I guess your DB is now out of order.

Please go to %localappdata%\EDDiscovery (assuming you're using the installed version with no modifications) and rename EDDUser.sqlite and EDDSystem.sqlite to start from scratch and see if that fixes your problems.

 

[robbyp]

Yep, killing a program what uses a DB will end up killing the DB in the end.

 

[Alkibiades]

I did a clean reinstall of edd and it built the db from scratch. Now its significantly faster than before. Capi never logs in but it does not open a website (nor does market connector which I banned into the task bar) and I close it myself

 

[Bravada Cadelanne]

I did a clean reinstall of edd and it built the db from scratch. Now its significantly faster than before. Capi never logs in but it does not open a website (nor does market connector which I banned into the task bar) and I close it myself

Both EDDiscovery and EDMC ask the OS to open the login URL, so if the default https: handler is messed up in Windows, then that can cause the website not to open.

 

[Iron Body]

My db gets out of date often, because I dont use it every time. So EDDisovery then tries to download large amounts of star data from EDSM.
But this is quite slow because of bandwidth restrictions in EDSM I think. And meanwhile EDdiscovery grinds to a halt. So I have to kill the process, which corrupts the db etc.
This is what it looks like anyway, and I can understand that.

To get round this I have severed communication with EDSM and use MC to update EDSM instead.
But I would like to keep the local star db reasonably up to date. And I periodically download the EDSM data dumps for other projects.

So I wondered if there was any way to recreate the EDdiscovery star dbs by importing data from an uncompressed EDSM daily data dump ?

 

[Bravada Cadelanne]

My db gets out of date often, because I dont use it every time. So EDDisovery then tries to download large amounts of star data from EDSM.
But this is quite slow because of bandwidth restrictions in EDSM I think. And meanwhile EDdiscovery grinds to a halt. So I have to kill the process, which corrupts the db etc.
This is what it looks like anyway, and I can understand that.

To get round this I have severed communication with EDSM and use MC to update EDSM instead.
But I would like to keep the local star db reasonably up to date. And I periodically download the EDSM data dumps for other projects.

So I wondered if there was any way to recreate the EDdiscovery star dbs by importing data from an uncompressed EDSM daily data dump ?

EDDiscovery rebuilds the database from a full dump if it hasn't been updated for 14 days, as otherwise it would require quite a few Systems API requests to fetch the updates (for 14 days, it'd be 56 requests each with a window of 6 hours, and each request taking somewhere between 10 and 30 seconds to process).

It might be appropriate to extend the no-update rebuild time to e.g. 3 months - the 14 day update window was set a few years ago, when EDSM had about a quarter as many systems. I think EDSM is also on faster hardware than it was back then, so the API requests return faster.

The 7-day systems dump will only be useful if used within 7 days of the previous update.

 

[Eahlstan]

[2022-02-03T11:45:36.132Z] Last record time 02/02/2022 19:07:00
[2022-02-03T11:45:41.452Z] .. Updated 10901 to 03.02.2022 01:06:56

Let's say that's about 6 seconds. It would take 4320 seconds (72 minutes) to reach the current API limit of 720 (if you don't do anything else that queries the API).
So, you'll never reach the hourly limit, thus the 14 days limit could be completely removed

 

[Bravada Cadelanne]

Let's say that's about 6 seconds. It would take 4320 seconds (72 minutes) to reach the current API limit of 720 (if you don't do anything else that queries the API).
So, you'll never reach the hourly limit, thus the 14 days limit could be completely removed

Apologies, it's not 14 days, but 28 days: <https://github.com/EDDiscovery/EDDiscovery/blob/master/EDDiscovery/EDDiscoveryControllerSync.cs#L66>

        const int ForcedFullDownloadDays = 28;      // beyond this time, we force a full download

Before 9.0.4 (January 2018), it was 7 days since the last update or 28 days since the last full sync.

Perhaps we should either make that threshold configurable, so the user can choose between database I/O (adding systems to an existing systems table requires a lot of seeking) versus internet download (2.4GB for the full download vs about 100MB over 112 requests for 28 days of catch-up requests), or increase it to e.g. 90 or 180 days (360 or 720 requests)

 

[Angelpichu]

Is there a way for EDDiscovery to auto download player journals not found in the local journal folder if they find them on the Frontier API, for players who play across multiple computers?

 

[Bravada Cadelanne]

Is there a way for EDDiscovery to auto download player journals not found in the local journal folder if they find them on the Frontier API, for players who play across multiple computers?

The Companion API journal endpoint stores journal entries by date, not by session, and does not include any Fileheader or Shutdown messages, so is not an alternative to copying journal files between computers (e.g. by uploading them to Google Drive, OneDrive, Dropbox, etc).

 

[Angelpichu]

The Companion API journal endpoint stores journal entries by date, not by session, and does not include any Fileheader or Shutdown messages, so is not an alternative to copying journal files between computers (e.g. by uploading them to Google Drive, OneDrive, Dropbox, etc).

Ah, dang. Does EDDiscovery pick up when journal files are updated by programs other than Elite Dangerous, like Resilio Sync? The idea would be on the primary computer, EDDiscovery runs, while journal files are synced and pushed to the primary computer.

 

[Bravada Cadelanne]

Ah, dang. Does EDDiscovery pick up when journal files are updated by programs other than Elite Dangerous, like Resilio Sync? The idea would be on the primary computer, EDDiscovery runs, while journal files are synced and pushed to the primary computer.

As long as the file is either:

• written sequentially (much like robocopy would do);
• renamed into place from a temporary file (much like a Chrome download or a rsync); or
• locked for exclusive access during writing (much like a copy/paste would do)

then EDDiscovery should see the new file(s) and read them if they are in a directory it is configured to watch (e.g. by setting that directory as the journal directory for a [possibly dummy] commander)

 

[Bravada Cadelanne]

Perhaps we should either make that threshold configurable, so the user can choose between database I/O (adding systems to an existing systems table requires a lot of seeking) versus internet download (2.4GB for the full download vs about 100MB over 112 requests for 28 days of catch-up requests), or increase it to e.g. 90 or 180 days (360 or 720 requests)

Apologies, I completely forgot that EDSM limits its output to 1000 systems per request, so:

• The 28 day catchup would currently be 823 requests, not 112 requests
• The 90 day catchup would currently be 2463 requests, not 360 requests
• The 180 day catchup would currently be 4779 requests, not 720 requests

 

[Eahlstan]

Apologies, I completely forgot that EDSM limits its output to 1000 systems per request, so:

• The 28 day catchup would currently be 823 requests, not 112 requests
• The 90 day catchup would currently be 2463 requests, not 360 requests
• The 180 day catchup would currently be 4779 requests, not 720 requests

Are you sure?
I just started EDD and it took 2 request (according to my .hlog and the visible log) to get 11010 systems.

Checking for updated EDSM systems (may take a few moments).
Downloading systems from UTC 10.02.2022 21:50:43 to 11.02.2022 03:50:43
Downloading systems from UTC 11.02.2022 03:50:37 to 11.02.2022 09:50:37
EDSM systems update complete with 11010 systems

 

[Para Handy]

What does this mean?:

 

[Eahlstan]

Don't you speak archaic english?

lastest - Wiktionary, the free dictionary

en.wiktionary.org

 

[robbyp]

That's been there many years. Shows how much attention you paying!

 

[Bravada Cadelanne]

Is it supposed to be Lastest, or Latest?
If it's the latter, is it a roundabout way of saying "Number by date ascending", where the default is "Number by date descending"?

 

[Eahlstan]

It could also be called "newest" as most people would know that word.
I'm all for keeping EDD in archaic english.

 

[Para Handy]

Is there any way to alter the "high value" threshold on the web server scan panel? It changes OK on the main EDD scan panel but the web server panel seems to be stuck with the default 50,000 value (IIRC).

Loving the webserver by the way.