Release Trade Computer Extension Mk.II

eventure

eventure
Here comes my first status update:
After some testing and optimization, I was unfortunately unable to process the data in a reasonable time frame. Approximately 30k stations took approximately 30 minutes, which is not acceptable.

The problem is not the number of stations, but how the data is listed in the Json. In particular, the outfitting and trading data are identified as text, and not with the ID number.
Searching through a text takes considerably longer than comparing a number. This significantly delays processing and therefore leads to a long update period.

Therefore, I will now look at the data from Spansh and have it calculated. What I have seen so far when skimming through the data is that it is complete with IDs and is not mixed up but sorted by star system.
This could lead to more structured processing, especially if the data needs to be supplemented while in the game. So I'm more optimistic about this data template, although it's more data (around 8.6GB vs around 5.8GB), it's probably easier to process. I'll let you know when the first results are available.
 

Atroposchaos

A
I have a question and suggestions for TCE:

Couldn't TCE run an EDSM API check when entering a new system/docked and accessing that system's EDSM market/outfitting/station data?
API: https://www.edsm.net/en/api-system-v1

Instead of trying to download and process the whole EDSM database at a time, it just actively checks your current system/maybe other systems around that one with x LY and, if possible, updates the TCE local database if newer.
It would slowly process the systems around you as you fly around for other trades.
This would also remove your issue with processing speed, as EDSM's API would do that for you when you use it to search.

That can be combined with the data from Spansh to give a comprehensive list of trade routes.

Also, you could do the trick with the EDSM by processing the complete database beforehand.
e.g. let it process when docked or, If the user allows, slowly process it over time as they play.
Then, instead of reprocessing it all, use the API to update it as needed (live data x LY radius the user can set around the current star system).

If required, do a full re-process later on or do a differencing of the previous database and update the differences only.
 
Last edited:

Stumpii

Stumpii
Atroposchaos said:
Couldn't TCE run an EDSM API check when entering a new system/docked and accessing that system's EDSM market/outfitting/station data?
API: https://www.edsm.net/en/api-system-v1

Instead of trying to download and process the whole EDSM database at a time, it just actively checks your current system/maybe other systems around that one with x LY and, if possible, updates the TCE local database if newer.
It would slowly process the systems around you as you fly around for other trades.
This would also remove your issue with processing speed, as EDSM's API would do that for you when you use it to search.
Click to expand...
Just tried myself and the API works great. The problem however is the number of markets and the sheer quantity of data. As an example, if you do a search on Spansh for Sol, there are nearly 6000 markets within a 30LY sphere. So if you jump 30LY to the edge of the old sphere, that is ~4500 (ball park) new markets available. Of course some markets can be chopped because of landing size, etc. but it is probably too many for every user to be spamming EDSM for the data on 1000s of markets after every jump. Still amazes me how many systems and stations there are in the bubble and how much data Inara and the like must process.

I think the best method is still to parse the daily download file for the markets as TCE does now. It is possible to process the json line by line because it is an array of markets, instead of reading the entire array into memory. This limits the memory usage (for reading) to 1 market. Can then discard or process the market before stepping on to the next.
 

Atroposchaos

A
Stumpii said:
Just tried myself and the API works great. The problem however is the number of markets and the sheer quantity of data. As an example, if you do a search on Spansh for Sol, there are nearly 6000 markets within a 30LY sphere. So if you jump 30LY to the edge of the old sphere, that is ~4500 (ball park) new markets available. Of course some markets can be chopped because of landing size, etc. but it is probably too many for every user to be spamming EDSM for the data on 1000s of markets after every jump. Still amazes me how many systems and stations there are in the bubble and how much data Inara and the like must process.

I think the best method is still to parse the daily download file for the markets as TCE does now. It is possible to process the json line by line because it is an array of markets, instead of reading the entire array into memory. This limits the memory usage (for reading) to 1 market. Can then discard or process the market before stepping on to the next.
Click to expand...
I think the panel you're looking for is the Shopping list in the top right on the top bar in TCE; if you put your mission material in there, it will find a station and alert you when you are in the system (Assuming I know how that feature works that is).

I agree that processing the daily for Spanch(Unknown time) if faster than EDSM(30+ Min) is still the best option overall; however, it does not have to be the only option used as I assume neither database contains all stations/markets.

Alternatively:
It could only check the current system markets when docked; it would help people find those tremendous 0 LY 12-20k+ local trade routes dotted about the place (I have made lots from stumbling on these types of trades as they are fast to do).

Even then, It does not have to check all 6000 Markets in that 30LY sphere when combined with the data from Spansh.
It would only have to check the ones Spanch was unaware of or had out-of-date info on to fill in the gaps in Spanch with EDSM.

From the API, you can check a specific station if required, which would greatly cut down on the checks required by the API.
Get information about the market in a station.

HTTP Request: GET https://www.edsm.net/api-system-v1/stations/market
ParameterDefaultDescription
marketId*NULLThe game marketId, if used no other parameters are needed.
 
Last edited:

eventure

eventure
Hi again,
After checking all the options, I came to the following conclusion: Processing the data from the existing files in this form makes no sense because the amount of data is simply too large.

That's why I came up with the idea of creating a tray app that downloads this amount of data when the PC starts up and could then reduce it to the bare minimum in the background to save resources. It creates the necessary files, which TCE can then process as usual.

Since the EDDB files back then were not optimized for TCE, the tray app will now do this to further shorten the processing time or to update all data in the background, if desired.
A few optimizations still need to be made so that TCE synchronizes and coordinates with the tray app.

I have already started creating the app, but it will take a while to release it as a beta version.

What do you think of such a supporting app for TCE?

Atroposchaos said:
I have a question and suggestions for TCE:

Couldn't TCE run an EDSM API check when entering a new system/docked and accessing that system's EDSM market/outfitting/station data?
API: https://www.edsm.net/en/api-system-v1

Instead of trying to download and process the whole EDSM database at a time, it just actively checks your current system/maybe other systems around that one with x LY and, if possible, updates the TCE local database if newer.
It would slowly process the systems around you as you fly around for other trades.
This would also remove your issue with processing speed, as EDSM's API would do that for you when you use it to search.

That can be combined with the data from Spansh to give a comprehensive list of trade routes.

Also, you could do the trick with the EDSM by processing the complete database beforehand.
e.g. let it process when docked or, If the user allows, slowly process it over time as they play.
Then, instead of reprocessing it all, use the API to update it as needed (live data x LY radius the user can set around the current star system).

If required, do a full re-process later on or do a differencing of the previous database and update the differences only.
Click to expand...
As Stumpil stated, there will be too many queries for the server. But the idea is not bad ta all.
Thanks for your suggestion!
Danieboy said:
Hi,
Just started to get Server is Down Status is the connector app , is it down?

thanks,
Click to expand...
Please use a previously released version of EDMC. Checkout this post and the follow ups: https://forums.frontier.co.uk/threads/trade-computer-extension-mk-ii.223056/post-10302962
 

Atroposchaos

A
eventure said:
Hi again,
After checking all the options, I came to the following conclusion: Processing the data from the existing files in this form makes no sense because the amount of data is simply too large.

That's why I came up with the idea of creating a tray app that downloads this amount of data when the PC starts up and could then reduce it to the bare minimum in the background to save resources. It creates the necessary files, which TCE can then process as usual.

Since the EDDB files back then were not optimized for TCE, the tray app will now do this to further shorten the processing time or to update all data in the background, if desired.
A few optimizations still need to be made so that TCE synchronizes and coordinates with the tray app.

I have already started creating the app, but it will take a while to release it as a beta version.

What do you think of such a supporting app for TCE?


As Stumpil stated, there will be too many queries for the server. But the idea is not bad ta all.
Thanks for your suggestion!

Please use a previously released version of EDMC. Checkout this post and the follow ups: https://forums.frontier.co.uk/threads/trade-computer-extension-mk-ii.223056/post-10302962
Click to expand...
Preprocessing all the data beforehand sounds like a good plan its got my vote.
I never turn my computer off so for me, that's a win.
Whenever I start up ED and TCE, it should already be up to date :)

What database will you be using for the TCE Database tray app: Spanch/EDSM or both?

I would recommend keeping the previous day's source DB (spanch/EDSM or both) and then differencing it against the new days nightly.
This can be done by a simple data class equal check, e.g. P = N if true, ignore as no change from the last day's DB; if not, then process into the TCE format.
It should make it much faster to process subsequent updates after the first full database check is done.
 

eventure

eventure
Atroposchaos said:
What database will you be using for the TCE Database tray app: Spanch/EDSM or both?
Click to expand...
The spansh database is better suited for this because it contains all reference IDs for each data record. this makes them easier to read and process.
If a record is updated, it must be newer. However, the data from the download is deleted after processing; when unpacked it takes up almost 9GB of space, too much for my taste and having the data twice is beyond the scope. A timestamp will be created after the update so that the database is not downloaded again, as before.
 

Danieboy

D
Any Ideas ?????

1707851690089.png
 

Attachments

  • 1707851683706.png
    1707851683706.png
    23.2 KB · Views: 33

Stumpii

Stumpii
eventure said:
The spansh database is better suited for this because it contains all reference IDs for each data record. this makes them easier to read and process.
If a record is updated, it must be newer. However, the data from the download is deleted after processing; when unpacked it takes up almost 9GB of space, too much for my taste and having the data twice is beyond the scope. A timestamp will be created after the update so that the database is not downloaded again, as before.
Click to expand...
It just seems so inefficient for every user to be downloading a GB of data every day of which most of the data is just thrown out. Atroposchaos does make a point about determining the delta in the data. If a dedicated computer could download the file from Spansh and process it into a better format and only include the changes required, those delta files could be placed on a server and all the clients could download these changes and incorporate them. The delta may not even be an entire station or market, but just pieces of the data required (like a patch, i.e. station:1234;commodity:12,65,0,32,55 or station:1234;state:boom). Would need to run some tests to see how much of the data is changing each day.
 

eventure

eventure
Stumpii said:
It just seems so inefficient for every user to be downloading a GB of data every day of which most of the data is just thrown out. Atroposchaos does make a point about determining the delta in the data. If a dedicated computer could download the file from Spansh and process it into a better format and only include the changes required, those delta files could be placed on a server and all the clients could download these changes and incorporate them. The delta may not even be an entire station or market, but just pieces of the data required (like a patch, i.e. station:1234;commodity:12,65,0,32,55 or station:1234;state:boom). Would need to run some tests to see how much of the data is changing each day.
Click to expand...
I know, but you're forgetting about the players who don't play ED every day. It is simply not possible for me to keep each player's own update patch for their databases on a server. Remember that TCE can update its databases as you play and if you enter new regions you will need the new region data and therefore all data must be present.

Another important point is that my programming skills are not sufficient. I have no idea about php scripts and cron jobs and I don't have the time to acquire these skills to accomplish this. Sorry, but then it will be easier for me to load all the data and process it locally as before.
 

Stumpii

Stumpii
eventure said:
I know, but you're forgetting about the players who don't play ED every day. It is simply not possible for me to keep each player's own update patch for their databases on a server. Remember that TCE can update its databases as you play and if you enter new regions you will need the new region data and therefore all data must be present.

Another important point is that my programming skills are not sufficient. I have no idea about php scripts and cron jobs and I don't have the time to acquire these skills to accomplish this. Sorry, but then it will be easier for me to load all the data and process it locally as before.
Click to expand...
I probably wasn't clear. The server would keep the patch file of the differences between todays and yesterdays Spansh data (one patch per day), not each players data. Each player would then download todays patch and apply it to their own database. But yes, would be better for a server to do this on a timely manner and I also have no knowledge of such things. It could run a home computer, downloading the Spansh file and just uploading (SFTP) the patch to a server, but probably not a good idea to run on a home PC. I am interested in how much actual data changes each day, so I may try looking into that.
 

eventure

eventure
Hello everyone,
I'm close to finalizing the TCE-Relay tray app to preprocess crowdsourced data from nightly Spansh/EDSM dumps and then update the TCE databases. The app is currently updating commodity prices, station and star system data. Faction and points of interest data will be available at a later date. So I am looking for some beta tester before launching the new 1.9.3 version to the public.

If you are interested in testing the new tray app with TCE, please contact me here or via PM. You will then be sent a link with additional information at the start of the closed beta.
 

eventure

eventure
Version [1.9.3.0] released
Changelog:
  • Added TCE-Relay as a tray application, preparing crowdsourced data from Spansh and EDSM nightly dumps to update TCE's databases as like EDDB-Relay did it before.
  • Added to TCE-Relay an autostart with windows function, doing all the stuff required in the background to download, extract and transform the nightly dumps into TCE's data schema.
  • Added extended support of markets from 65k to 215k, if you want most markets of the game in your database.
  • Fixed an issue with the shopping list showing elements and outfitting locations outside the selected trade range.

Notes:
Either download the version above, install and transfer your old database, or use the auto-update function of the launcher.
Click the 'new version' notice at the bottom of the launcher.

If you find an error, please report and use the Report a Bug button at the launcher and don't forget to copy the error details and ErrorReportFile.7z archive.
I'm adding the VirusTotal link in case you get a warning from your AV app. In this case, please update your AV app.
VirusTotal: https://www.virustotal.com/gui/file/16cec320ceee0be97e56cb55825d3ac82715ea7f08286ec2ba877b3252731ee2
 

eventure

eventure
Version [1.9.3.0] Hotfix released
Changelog:
  • Fixed an issue with the Language database.
  • Fixed an issue with the 'Same as Game Language' button at the Settings tab of the launcher.
  • Fixed an issue with the 'Force EDSM Update' button at the Cartography panel.
Notes:
Extract the files into the TCE installation folder and overwrite the existing files. The hotfix is now included in the release version.

If you find an error, please report and use the Report a Bug button at the launcher and don't forget to copy the error details and ErrorReportFile.7z archive.
 
Last edited:

Danieboy

D
getting .Net errors all the time now. :-

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.IndexOutOfRangeException: Index was outside the bounds of the array.
at System.Array.InternalGetReference(Void* elemRef, Int32 rank, Int32* pIndices)
at System.Array.GetValue(Int32 index1, Int32 index2)
at Microsoft.VisualBasic.CompilerServices.Symbols.Container.GetArrayValue(Object[] Indices)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.InternalLateIndexGet(Object Instance, Object[] Arguments, String[] ArgumentNames, Boolean ReportErrors, ResolutionFailure& Failure, Boolean[] CopyBack)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.ObjectLateInvokeDefault(Object Instance, Object[] Arguments, String[] ArgumentNames, Boolean ReportErrors, Boolean[] CopyBack)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateIndexGet(Object Instance, Object[] Arguments, String[] ArgumentNames)
at Trade_Computer_Extension.Panel_Trade_Overview.Show_POS_StationData(Int32 marketID) in F:\Trade Computer Extension\Trade Computer Extension\Panel_Trade_Overview.vb:line 560
at Trade_Computer_Extension.Panel_Trade_Overview.Show_POS_Data() in F:\Trade Computer Extension\Trade Computer Extension\Panel_Trade_Overview.vb:line 330
at Trade_Computer_Extension.Panel_Trade_Overview.Update_Panel() in F:\Trade Computer Extension\Trade Computer Extension\Panel_Trade_Overview.vb:line 219
at System.Windows.Forms.Control.OnVisibleChanged(EventArgs e)
at System.Windows.Forms.Form.OnVisibleChanged(EventArgs e)
at System.Windows.Forms.Control.WmShowWindow(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.Form.WmShowWindow(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9181.0 built by: NET481REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/mscorlib.dll
----------------------------------------
Trade Computer Extension MK.II
Assembly Version: 1.9.3.0
Win32 Version: 1.9.3.0
CodeBase: file:///E:/SteamLibrary/steamapps/common/Elite%20Dangerous/TCE/Trade%20Computer%20Extension%20MK.II.exe
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 14.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9220.0 built by: NET481REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9220.0 built by: NET481REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9181.0 built by: NET481REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.Runtime.Remoting
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9214.0 built by: NET481REL1LAST_B
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Remoting/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
System.Xml.Linq
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml.Linq/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.Linq.dll
----------------------------------------
System.Data
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9214.0 built by: NET481REL1LAST_B
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_64/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll
----------------------------------------
System.Numerics
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll
----------------------------------------
Newtonsoft.Json
Assembly Version: 10.0.0.0
Win32 Version: 10.0.3.21018
CodeBase: file:///E:/SteamLibrary/steamapps/common/Elite%20Dangerous/TCE/Newtonsoft.Json.DLL
----------------------------------------
System.Runtime.Serialization
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Serialization/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Serialization.dll
----------------------------------------
Snippets
Assembly Version: 0.0.0.0
Win32 Version: 4.8.9220.0 built by: NET481REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Anonymously Hosted DynamicMethods Assembly
Assembly Version: 0.0.0.0
Win32 Version: 4.8.9181.0 built by: NET481REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_64/mscorlib/v4.0_4.0.0.0__b77a5c561934e089/mscorlib.dll
----------------------------------------
System.Data.SQLite
Assembly Version: 1.0.117.0
Win32 Version: 1.0.117.0
CodeBase: file:///E:/SteamLibrary/steamapps/common/Elite%20Dangerous/TCE/System.Data.SQLite.DLL
----------------------------------------
System.Transactions
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_64/System.Transactions/v4.0_4.0.0.0__b77a5c561934e089/System.Transactions.dll
----------------------------------------
System.EnterpriseServices
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_64/System.EnterpriseServices/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.EnterpriseServices.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0 built by: NET481REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
System.Windows.Forms.DataVisualization
Assembly Version: 4.0.0.0
Win32 Version: 4.8.9037.0
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms.DataVisualization/v4.0_4.0.0.0__31bf3856ad364e35/System.Windows.Forms.DataVisualization.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
 
You must log in or register to reply here.
Top Bottom