Release Introducing EDColonise.net, a colonisation candidate finder

GroobTM

G
Current Version V2.01

ED Colonise is a tool to help CMDRs find colonisation targets within 1000 ly of Sol.

The tool is available at EDColonise.net.

I have improved the tool's performance and incorporated some of the community's recommendations. A short change log can be found below.

Unfortunately, the tool still suffers from inaccuracies due to insufficient data being sent over EDDN. Remember to contribute data to the network and verify the tool's suggestions using in-game map mode. If you keep finding unavailable systems, jump to a deeper page of the results or use a more specific search.

What's next?
I know Frontier added new journal events relating to colonisation and claims. Once EDDN sends these events, I will update this tool to use them, which will improve its accuracy.

Besides this, the tool will likely not undergo significant changes until the summer, as I will be busy with real life until then. Please leave any feature recommendations or bug reports below.

V2.01 (15/04/2025)
  • Added a blacklist for invalid systems that were missed by the filter.
    • Added all Thargoid Spire sites to the blacklist.
V2 (12/04/2025)
  • Upgraded and re-wrote the database. Query times should be more in the 5-7 seconds range instead of 10-15.
  • Changed "Landable Bodies" to "Disembarkable Bodies" to exclude bodies you could land on but not build on.
  • Added a link to the closest Trailblazer Megaship and an option to sort by closest Megaship.
  • Fixed the "Distance to Sol" sort option by sorting by ascending distance, not descending distance.
  • Added option to select how many results are shown per page (5, 10, 20, 50).
  • Added the option to jump to a specific page of the results.
I have created a tool to help CMDRs locate and rank currently available colonisation candidates within 1000 ly of Sol.

This tool is available at EDColonise.net.

There are some issues that I am currently aware of:
  1. The tool relies on information from EDDN to track claims and colonisation, so sometimes, the tool doesn't match the game.
  2. The tool currently takes a while to get search results. This is a result of the current database server being crap. I have already created a v2 of the database that significantly improves search times, but I need to find a server to host it. (If anyone would be willing to help me with this, it would be greatly appreciated.)

If you have any questions/feedback/recommendations, feel free to share. :)
 
Last edited:

GroobTM

G
The tool might not be available immediately as I am adding a link to this post.
It's now live.
 
Last edited:

GroobTM

G
Macros Black said:
The site looks really good, CMDR :)

A little gold plating suggestion:
Would be awesome if you could show the nearest well stocked stations to supplies for:
  • Refinery
  • Industrial
  • High Tech
  • Agricultural
Maybe call Inara.cz, e.g. https://inara.cz/elite/commodities/?formbrief=1&pi1=1&pa1[]=10487&ps1=HIP+50874&pi10=3&pi11=0&pi3=2&pi9=2000&pi4=0&pi14=0&pi5=720&pi12=0&pi7=500&pi8=1&pi13=1
Click to expand...
One thing I'm wary of with doing something like this is that other people have already created better tools for tracking this kind of information (such as DaftMav's Colonisation construction spreadsheet), and I don't know what I could bring to the table that they haven't already done better. That being said it could be valuable to show the distance to the closest trailblazer megaship.
 

tsh847

T
Just lost two hours of my life checking a system find with this, "Include claimed systems" unchecked. Guess what I found there? A colonization ship arrived a week before I did. So... gold platings can wait. Sorry to bother.
 

GroobTM

G
tsh847 said:
Just lost two hours of my life checking a system find with this, "Include claimed systems" unchecked. Guess what I found there? A colonization ship arrived a week before I did. So... gold platings can wait. Sorry to bother.
Click to expand...
Currently because there is no claim event in the game journal, I have to check for system claims indirectly via the docking journal event. This combined with the fact that not everyone sends their data over EDDN means there will always be some inaccuracy with the tool.
I would always recommend using the colonisation map mode on the galaxy map to verify that a system is unclaimed.

I am currently working on the next version of the tool incorporating some features that were suggested by users. One of which will be the ability to jump to specific pages in the results to skip over the already claimed systems that the tool has not yet picked up on yet.

Sorry that the tool led to you wasting your time.
 

markleaf

M
Maybe a suggestion for you to implement, can you add planets you can disembark on? It seems if you can land on it and the temperature is too high you can't walk on it, if you can't walk on it you can't build on it it seems. As well if you could add a "heatmap" of distance distributions of the planet that would help with finding a system as well. There's some systems where they have a lot of landable bodies but half of them end up being at like 500,000 ly out.
 

GroobTM

G
markleaf said:
Maybe a suggestion for you to implement, can you add planets you can disembark on? It seems if you can land on it and the temperature is too high you can't walk on it, if you can't walk on it you can't build on it it seems. As well if you could add a "heatmap" of distance distributions of the planet that would help with finding a system as well. There's some systems where they have a lot of landable bodies but half of them end up being at like 500,000 ly out.
Click to expand...
I didn't realise there were restrictions on what you could disembark on, but looking at the wiki, there are a few restrictions that I will look into implementing.

The heatmap idea is a little more complicated, and part of the reason I provide the Spansh link for each result is so that people can investigate that information themselves if they wish. It is something that could be implemented, but might have to wait for a v3 version of the tool when university isn't consuming all my time.

Aside:
Interestingly, when I looked at the Wiki, I saw that you can disembark on hot planets if you land on the cold/dark side. If you can build on the dark side and the planet is not tidally locked, eventually, your settlement will be a death trap.
 

markleaf

M
GroobTM said:
If you can build on the dark side and the planet is not tidally locked, eventually, your settlement will be a death trap.
Click to expand...
From what I've seen it just ends up you not being able to build on the planet at all, there are no build slots. I don't know if anyone seen something different.

For the heatmap it can just be something simple like:

0-1000 Ls: 5 (50%)
1000-10,000 LS: 3 (30%)
10,000-50,000 LS: 1 (10%)
50,000+ LS: 1 (10%)

Just a basic range to see the general layout. Should just be grabbing the number of bodies in the range and dividing by the number of bodies.
 

nemolomen

nemolomen
GroobTM said:
...
Aside:
Interestingly, when I looked at the Wiki, I saw that you can disembark on hot planets if you land on the cold/dark side. If you can build on the dark side and the planet is not tidally locked, eventually, your settlement will be a death trap.
Click to expand...
Oh no! Think of the NPCs...
 

ArcherS

A
One option: get the Stations update from spansh after the Thursday AM maintenance. That file is generated more frequently, so you're not waiting 22 hours to get an update with systems that finished colonization from the tick. E.g., the colony whose primary port I completed a couple days ago is correctly listed in Inara, but EDColonise doesn't show any colonization targets around the system. (HIP 53340)
 

GroobTM

G
ArcherS said:
One option: get the Stations update from spansh after the Thursday AM maintenance. That file is generated more frequently, so you're not waiting 22 hours to get an update with systems that finished colonization from the tick. E.g., the colony whose primary port I completed a couple days ago is correctly listed in Inara, but EDColonise doesn't show any colonization targets around the system. (HIP 53340)
Click to expand...
You raise a good point. One issue though is that downloading the file after the tick would not necessarily contain updated information as the information from Spansh is updated by players who won't have played for long enough to update the Spansh database.

The solution to this would be to grab this data from EDDN live when it's updated.

The only issue that could arise is that computing which systems are in range of a colonised system can be a resource intensive task and so doing this throughout the day could negatively impact performance in other places (which is why it's normally run at 5 a.m.). I'll look into it as it would be a good improvement.
 

GroobTM

G
markleaf said:
From what I've seen it just ends up you not being able to build on the planet at all, there are no build slots. I don't know if anyone seen something different.

For the heatmap it can just be something simple like:

0-1000 Ls: 5 (50%)
1000-10,000 LS: 3 (30%)
10,000-50,000 LS: 1 (10%)
50,000+ LS: 1 (10%)

Just a basic range to see the general layout. Should just be grabbing the number of bodies in the range and dividing by the number of bodies.
Click to expand...
Just to clarify, are you suggesting an actual graphic? Because if so, a distribution would be easier to read than a heat map.
 

Destranix

D
What is your database layout and how are you creating the database from dumps?

I am currently writing a C++ tool for streaming a dump directly into a database (low working memory usage), actually aiming to create a tool like yours eventually. I finished the stream part, next is parsing the system data to a custom struct and writing stuff to SQLite database.

So I have to plan about how to smartly store data in the database with spatial sorting to allow efficient searches for nearest colonisation root and such. Also I have to think about how to e.g. recognize systems that are valid to start colonisation from.
I can plan that myself, but would be not smart to do the work someone else already did.

Also, depending on how you create/update your database, I might be able to offer a more efficient tool.
 

GroobTM

G
Destranix said:
What is your database layout and how are you creating the database from dumps?

I am currently writing a C++ tool for streaming a dump directly into a database (low working memory usage), actually aiming to create a tool like yours eventually. I finished the stream part, next is parsing the system data to a custom struct and writing stuff to SQLite database.

So I have to plan about how to smartly store data in the database with spatial sorting to allow efficient searches for nearest colonisation root and such. Also I have to think about how to e.g. recognize systems that are valid to start colonisation from.
I can plan that myself, but would be not smart to do the work someone else already did.

Also, depending on how you create/update your database, I might be able to offer a more efficient tool.
Click to expand...

​

The Database​

The database structure can be broken down into two groups:

The first group is the systems' data, which contains tables for systems, coordinates, stations, bodies, rings, hotspots, and factions. Some extra tables store types like body, reserve, and hotspot types.

The second group comprises functional tables containing precomputed search values and a staging table.
The precomputed search tables are:
  • A nearby systems table that contains couples of colonised systems and the uncolonised systems within their range.
  • A system summaries table that includes total body value, number of disembarkable planets, distance to Sol, etc. (Any metric that I want to sort the results by goes in this table.) This table only included uncolonised systems that are in the nearby systems table.
  • A table of distances between uncolonised systems (in the nearby systems table) and the closest trailblazer megaship.

The staging table tracks new inserts and updates from the last data dump import so the program knows what to add to the precomputed tables.

The Program​

I convert my deserialised objects into composite SQL datatypes, which allows me to insert an array of 2000 systems per batch into an SQL function that then inserts the array into the main tables.

The function inserts new dependencies and handles conflicts with existing records (whether that's updating conflicting records or ignoring conflicts).

There are five composite datatypes and functions for systems, stations, bodies, rings, and hotspots. In one batch of 2000 systems, there may be 200 stations and 20,000 bodies.

Triggers catch new system inserts and updates that change a system's colonised status and add them to the staging table.


Once the new data has been imported into the database, the program updates the nearby systems table by constructing a K-D tree of the coordinates of colonised systems and another of uncolonised systems. Then, the staged systems are taken from the staging table and used to probe the trees (colonised systems probe the uncolonised tree and visa-versa), finding all nodes (systems) within the colonisation range of the staged system.

Once all the systems in range have been found, the couples are added to the nearby systems table. If a system is updated from uncolonised to colonised, the old records from the nearby system table will also be deleted, as well as any linked records in the bodies, rings, and hotspots tables since they're no longer useful.

Updating the remaining precomputed values tables involves just some standard count queries and a bit of maths. However, I do truncate the table before updating it in case new information has been added to existing systems.


Database Choice​

When I started this project, I also used SQLite because of its ease of setup. However, I quickly found that I didn't like its reduced feature set (at least for this project), and I wasn't sure how it handles concurrency.
My advice is to consider whether it's the right choice for the program you want to make, but my experience with it is too limited to give you sage advice.

If you want to see the actual code or my DDL and DML, send me a DM, and I might be able to show you the repo (although it's a bit of a mess).
 

Destranix

D
Thanks for that detailed information! It gave me a few ideas I haven't had in mind.
GroobTM said:
My advice is to consider whether it's the right choice for the program you want to make, but my experience with it is too limited to give you sage advice.
Click to expand...
I guess I won't need that much database features. Actually all I need is a transaction store and SQL JOIN, everything else is just bonus.
 
You must log in or register to reply here.
Back
Top Bottom