Discussion What is the most efficient way to crowdsource the 3D system coordinates

[TunaMage]

Hi RW. I take it your frontend app is not public?
Regarding a new frontend, I agree with everything you are saying. I think the suggested reference stars (using whatever method that works) is important for the frontend - where the original ref systems create an unclear target location.

I also think that feedback to the user entering the data is important, which is where I struggled a bit with TGC as there is zero feedback.

Once data has been accepted on the front end there should at minimum be "here's the calculated coords of your new system", and a nice to have would be a small graphic of your new system relative to sol, with a compass pointer to the galactic centre. Its a very quick and robust way for a user to eyeball that the system they entered is more or less where they think they are.

But also, as you say, maybe worth waiting for the FD API first...

 

[Smacker]

Thanks, just spotted this myself. There were a couple of copies of Harermid with misspelt names. I've just updated the github repo.

Also here is a list of case corrections:

https://gist.github.com/SamMackrill/d08034c98bb047d79c3b

 

[Finwen]

I wanted a more general solution (since any set of reference systems is going to fail for some cases) so I've added a suggestion function that should make it easier to locate distant systems. Once it has at least 3 distances it'll recommend up to 5 systems (the ones with the shortest names) that are the most effective at distinguishing between the candidates that have been identified so far. You can click on a name to add it to the table of distances. It seems to work pretty well for the systems I've tested it on. Let me know how it works for you - I'm not as far out as you are so I haven't been able to test it on any really tricky systems yet.

I am at the edge of populated worlds now. But i am going out for a little trip so i will test your new solution. Feels like a good idea

 

[Finwen]

I wanted a more general solution (since any set of reference systems is going to fail for some cases) so I've added a suggestion function that should make it easier to locate distant systems. Once it has at least 3 distances it'll recommend up to 5 systems (the ones with the shortest names) that are the most effective at distinguishing between the candidates that have been identified so far. You can click on a name to add it to the table of distances. It seems to work pretty well for the systems I've tested it on. Let me know how it works for you - I'm not as far out as you are so I haven't been able to test it on any really tricky systems yet.

I found a bug in your new sysinfo.html page.

1. Load go to sysinfo.html
2. Enter S171 8 and press GO
3. It shows 7 distances and 5 potential candidates..
4. Enter S171 9 and press GO
5 It shows coordinates for nr S171 9 but Distances are still from the OLD system S171 8.
6 If you do a reload on sysinfo.html and enters S171 9 then it shows the 18 distances for S171 9

This confused me allot before i did understand whats going on

My PM to you about different trilatation in new version might be due to this bug....

 

[RedWizzard]

Hi RW. I take it your frontend app is not public?

You can get it here: https://github.com/SteveHodge/ed-systems. It's html based but it's all clientside scripting so you can just open local copies of the html files. I haven't got around to getting it hosted anywhere. Entry.html is the page for entering new systems. Sysinfo.html lets you see/check what data is recorded for a system (the functionality of this and entry.html have drifted together so I'll probably merge them at some point). systemsvis.html is a simple 3D display of all the systems we have.

I also think that feedback to the user entering the data is important, which is where I struggled a bit with TGC as there is zero feedback.

Once data has been accepted on the front end there should at minimum be "here's the calculated coords of your new system", and a nice to have would be a small graphic of your new system relative to sol, with a compass pointer to the galactic centre. Its a very quick and robust way for a user to eyeball that the system they entered is more or less where they think they are.

But also, as you say, maybe worth waiting for the FD API first...

A 3D display is an interesting idea. Currently I show a list of the (known) nearby systems so you can check that against the nav panel but a 3D view of the nearby systems might be more intuitive.

The TGC/EDSC webpage for entering data is rather lacking in feedback. The API itself actually provides pretty good detail on what's been done: whether or not the system has been added or updated, what's been done with each distances, results of trilateration etc. But I still think it's better to check everything before it's submitted, at least with the current API. A new API could provide a "dry run" option where the server processes the data and provides feedback but doesn't actually record anything.

 

[RedWizzard]

Also here is a list of case corrections:

https://gist.github.com/SamMackrill/d08034c98bb047d79c3b

Thanks for that. I've applied most of them. There are a few that I think StarChart has wrong:
"XX chi ...": "chi" doesn't have a capital in game for all the one's I've looked at. Weirdly for all the other similar names I've looked at, such as "24 Iota Librae", the greek letter name is capitalised.
"M Hydrae", "P Eridani", "R Doradus": ED is inconsistent for these names with a single letter first part. These ones are capitalised in game but "g Lupi" and "k Persei" are not.
"Cephei Sector KM-W C1-24": names with this form have the letter at the start of the final bit lower case ("... c1-24"). I'm correctling these automatically.
"Ic 2602 Sector...", "R Cra Sector...", "Lbn 623 Sector...": these are "IC 2602 Sector", "R CrA Sector", and "LBN 623 Sector" in game.

 

[RedWizzard]

I found a bug in your new sysinfo.html page.

1. Load go to sysinfo.html
2. Enter S171 8 and press GO
3. It shows 7 distances and 5 potential candidates..
4. Enter S171 9 and press GO
5 It shows coordinates for nr S171 9 but Distances are still from the OLD system S171 8.
6 If you do a reload on sysinfo.html and enters S171 9 then it shows the 18 distances for S171 9

This confused me allot before i did understand whats going on

My PM to you about different trilatation in new version might be due to this bug....

Oops, made a silly copy and paste error. Thanks for finding this. It's fixed now.

 

[Finwen]

You can get it here: https://github.com/SteveHodge/ed-systems. It's html based but it's all clientside scripting so you can just open local copies of the html files. I haven't got around to getting it hosted anywhere. Entry.html is the page for entering new systems. Sysinfo.html lets you see/check what data is recorded for a system (the functionality of this and entry.html have drifted together so I'll probably merge them at some point). systemsvis.html is a simple 3D display of all the systems we have.

I host a copy here if someone wants to test it. http://robert.astronet.se/Elite/ed-systems/

 

[wolverine2710]

You can get it here: https://github.com/SteveHodge/ed-systems. It's html based but it's all clientside scripting so you can just open local copies of the html files. I haven't got around to getting it hosted anywhere. Entry.html is the page for entering new systems. Sysinfo.html lets you see/check what data is recorded for a system (the functionality of this and entry.html have drifted together so I'll probably merge them at some point). systemsvis.html is a simple 3D display of all the systems we have.

A part of your tool is being hosted by Maddavo. Its the distances entry part. You can find it here. Not sure about which version Maddavo is hoting. Kfsone of TD has support for TGC built into TD. With it you can submit distances to the TGC. HAven't tested it so don't know if its shows a good list of reference stars.

 

[TunaMage]

A 3D display is an interesting idea. Currently I show a list of the (known) nearby systems so you can check that against the nav panel but a 3D view of the nearby systems might be more intuitive.

Yep, personally I find a visual display very handy, and adds a bit of life. Below is a screenshot of how my excel based route planner displays a route from Lave to Sol. Very simple but you get the idea you're travelling 'down and to the left' which is good to have in the back of your mind when actually travelling.
(on the side, excel is nasty time wise to calculate these sort of algorithms, so I'll be changing to something else when I get the time )

Spoiler

 

[RedWizzard]

A part of your tool is being hosted by Maddavo. Its the distances entry part. You can find it here. Not sure about which version Maddavo is hoting. Kfsone of TD has support for TGC built into TD. With it you can submit distances to the TGC. HAven't tested it so don't know if its shows a good list of reference stars.

Ah, cool. I should probably start putting a version number in there somewhere.

 

[Phyl]

Hi Guys,

Just wanted to jump in here as I've just come across this thread. It's an interesting project and I'd like to get involved. I bring 15 years of professional experience as a developer with a focus on information management, business app development and 8 years experience developing graphical and other creative apps. I hope i have some useful experience and resources I can offer.

I ended up here as I've been considering a concept for an explorer tool and as a starting point have been looking for a reliable source of system coordinate data. I've found a number of potential sources, but they all seem pretty inconsistent in terms of data volumes, quality and also the co-ordinates systems being used. My initial vision is for a tool incorporating a 3D visualisation of the explored parts of the galaxy, allowing explorer to plot their journey history as well as journeys and locations of others, tag systems with screenshots and other information, and to generally promote an explorer community.

I haven't been able to scan through much of this thread as it's so large and I lack the context to fully understand many of the posts. To cut to the chase, is there a single data source that you are all using and contributing to? I've seen there are a few different APIs - are these independent or are they based on the same core datasets?

I'm also interested how are you collecting data? Are you relying on manual input, scraping log files or using CV or something similar?

I'm keen to know who are the key-players in this and what level of coordination is there.

Cheers
Phil

 

[maddavo]

A part of your tool is being hosted by Maddavo. Its the distances entry part. You can find it here. Not sure about which version Maddavo is hoting. Kfsone of TD has support for TGC built into TD. With it you can submit distances to the TGC. HAven't tested it so don't know if its shows a good list of reference stars.

Ha, funny. I saw the git update and just updated it today.

 

[maddavo]

Hi Guys,

Just wanted to jump in here as I've just come across this thread. It's an interesting project and I'd like to get involved. I bring 15 years of professional experience as a developer with a focus on information management, business app development and 8 years experience developing graphical and other creative apps. I hope i have some useful experience and resources I can offer.

I ended up here as I've been considering a concept for an explorer tool and as a starting point have been looking for a reliable source of system coordinate data. I've found a number of potential sources, but they all seem pretty inconsistent in terms of data volumes, quality and also the co-ordinates systems being used. My initial vision is for a tool incorporating a 3D visualisation of the explored parts of the galaxy, allowing explorer to plot their journey history as well as journeys and locations of others, tag systems with screenshots and other information, and to generally promote an explorer community.

I haven't been able to scan through much of this thread as it's so large and I lack the context to fully understand many of the posts. To cut to the chase, is there a single data source that you are all using and contributing to? I've seen there are a few different APIs - are these independent or are they based on the same core datasets?

I'm also interested how are you collecting data? Are you relying on manual input, scraping log files or using CV or something similar?

I'm keen to know who are the key-players in this and what level of coordination is there.

Cheers
Phil

Hi Phil, welcome! Here are things as I understand them.

In short, we use www.edstarcoordinator.com as the reference for system coordinates. It has an API mechanism for submitting distances between systems for trilaterating (multilaterating?) coordinates and querying the system database. There are various tools and databases that exchange data with edstarcoordinator.

As a bit of history, through several beta and gamma versions of ED, other clever ppl in this thread came up with methods to calculate system coordinates accurately. In ED all coordinates are on a 1/32 Ly 3D grid. There were issues of rounding/truncating, squaring etc etc to come up with accurate algorithms and then at one point the accuracy of the distances being captured changed by a factor of 10 (lost a decimal place). But essentially, since release there has been a pretty accurate way to get system coordinates and map the galaxy.

We were assisted along the way by Michael Brookes (ED Executive Producer) who kindly supplied to us spreadsheets of known system coordinates. For Gamma/Release version of ED, Michael gave us a spreadsheet with around 19k systems. This was basically a list of all systems with a station in it (although I think we now have one or maybe more that have added stations since the list was provided). Mapping efforts in the last three months have expanded the database to around 22k systems.

In this thread, it was mutually agreed that the coordinate system would be SOL-centric. This is how the coordinates are presented in-game in the Galaxy Map. Michael's list was not SOL-centric so the coordinates were converted. Therefore, the coordinates stored and provided by www.edstarcoordinator.com are in reference to SOL as (0,0,0) .

Cheers,
Maddavo

 

[Phyl]

Hi Maddavo,

Thanks for the info - very interesting. I will take a look at the API on edstarcoordinator and go from there.

I am assuming that in-game the majority of the galaxy is procedural and is locally generated as there are some interesting looking files in the StellarForge folders. Does anyone know any more about how the galaxy is actually generated?

Cheers
Phil

 

[Andargor]

I am assuming that in-game the majority of the galaxy is procedural and is locally generated as there are some interesting looking files in the StellarForge folders. Does anyone know any more about how the galaxy is actually generated?

Nothing official, other than stray comments from Michael about exploration data being separate and server-side, and hard to represent on the galmap.

We've have a couple of debates here on how the galaxy is generated and how the client accesses that info, so this is very speculative on my part:

- The galaxy seems to have all star types and positions pre-generated server-side (some kind of global seed + some static positions based on actual astronomical data)

- The client accesses this information when you browse the galaxy map. It seems to retrieve the star types and positions from the server depending on the area you are looking at, and your zoom level. If you go straight to the core with a search in the galmap, you will notice a delay while it fetches the data. Ergo, the client doesn't contain these star positions (and that would take up too much space, anyway)

- Some exploration data from your save game seems to be used and is layered on the map (e.g. when you hover over a system, and see if you have the system data)

- System data is a completely different "layer", and it's all on the server. It seems randomly generated when you enter a system, or is already statically determined in advance (e.g. Sol)

 

[Phyl]

Again, interesting, thanks.

Has anyone looked at the verbose log files? There are some coordinates captured in there associated with bodies in a system, but I'm not clear at the moment if these are entirely local coordinates within a system or are galactic. I've not managed to figure out a relationship between these and the other coordinate systems, but I was going to do some testing to try and get a better understanding. If anyone has done that it might save me a job... If it is possible to parse coords from log files, it would make things a lot easier

Here are some examples of relevant lines from the logs:

{13:05:38} System:130(Bhutas) Body:62 Pos866.272,2998.89,-1924.16) cruising
{13:09:12} System:130(Bhutas) Body:61 Pos6529.04,18800,7114.46)
{13:22:04} System:4(Neganhot) Body:0 Pos-9.58462e+007,-8.62e+009,-1.30846e+010) cruising
{13:42:38} System:11(Reorte) Body:0 Pos-1.16271e+010,9.57255e+009,-5.2836e+009) cruising

I'm hoping (probably in vain) that it will turn out that these coordinates can somehow be translated into galactic coordinates. The scale of the difference between bodies in different systems makes me think this could be the case...

From what I've seen of the logs (which is a bit limited at the moment) the coordinates of body 0 tend to remain constant, where as a the coordinates of the other bodies change - which would make sense if they are orbiting body 0. This also makes me think these do refer to coordinates of the body and not of the ship. Even it this turns out to a be futile path, having access to body numbers for systems could be useful in some contexts.

 

[Andargor]

{13:05:38} System:130(Bhutas) Body:62 Pos866.272,2998.89,-1924.16) cruising

I had had the same interrogations, and someone indicated that they were the local coordinates of your ship in normal space, unfortunately. Don't waste your time on them.

 

[graham.reeds]

I would be highly surprised if the server generates the coordinates and sends it across. The delay is due to it crunching the numbers to get the data. If I were doing this the galaxy would be 102400 light years across. Now recursively subdivide that until you get to the lowest level and run the algorithm to generate mass from the density wave map. Several maps could be used to determine types (HII regions, etc). The local area is generated from real data Sol, Bernard Star, et al; while some is hand generated by FDev, Lave, etc. Outside that is the procedurally generated systems.
Server side calculations don't scale.

 

[RedWizzard]

I would be highly surprised if the server generates the coordinates and sends it across. The delay is due to it crunching the numbers to get the data. If I were doing this the galaxy would be 102400 light years across. Now recursively subdivide that until you get to the lowest level and run the algorithm to generate mass from the density wave map. Several maps could be used to determine types (HII regions, etc). The local area is generated from real data Sol, Bernard Star, et al; while some is hand generated by FDev, Lave, etc. Outside that is the procedurally generated systems.
Server side calculations don't scale.

Yes, I agree and I believe Michael said or implied as much in one of his posts (IIRC he said that any change to the position or structure of systems required a client patch). I think you're right that about the basic technique: systems in the smallest subdivided volume (a sector) are generated using the coordinates and the output of a density function and then the resulting systems are combined with a list of hand-generated and real life systems. This is essentially the same technique that the previous games used and I have yet to see anything to suggest they're not doing the same with ED.

Michael said there were about 144,000 real life objects in ED.