Discussion What is the most efficient way to crowdsource the 3D system coordinates

[Phyl]

I think you're right that about the basic technique: systems in the smallest subdivided volume (a sector) are generated using the coordinates and the output of a density function and then the resulting systems are combined with a list of hand-generated and real life systems.

I think this is spot on. The folder Frontier_Developments\Products\FORC-FDEV-D-1003\Win32\Shared\StellarForge\Galaxies\MilkyWay contains an overrides file, which just from the name sounds like it is providing the hand-generated and real life systems. The rest of the StellaForge seems to contain a mix of static databases and generative processes. I assume these are all run locally but use a common seed and server timestamp to ensure everyone's galaxies are the same.

Interestingly within here there is a galaxies (plural) folder - which only contains one galaxy - the Milky Way. Makes me wonder if other galaxies may be supportable for the future... Would need a pretty big FSD to jump to Andromeda!

I feel I'm reaching a bit of a dead-end with co-ordinate identification - hoping the future API might bring something in that regard but the crowd source model isn't really going to be suitable for exploration where systems are being visited for the first time. Nobody is going to bother filling in every system they jump into, and it doesn't seem there is an alternative at the moment.

- - - - - Additional Content Posted / Auto Merge - - - - -

I had had the same interrogations, and someone indicated that they were the local coordinates of your ship in normal space, unfortunately. Don't waste your time on them.

I did... a bit... And came to the same conclusions!

 

[Smacker]

Thanks for that. I've applied most of them. There are a few that I think StarChart has wrong:
"XX chi ...": "chi" doesn't have a capital in game for all the one's I've looked at. Weirdly for all the other similar names I've looked at, such as "24 Iota Librae", the greek letter name is capitalised.
"M Hydrae", "P Eridani", "R Doradus": ED is inconsistent for these names with a single letter first part. These ones are capitalised in game but "g Lupi" and "k Persei" are not.
"Cephei Sector KM-W C1-24": names with this form have the letter at the start of the final bit lower case ("... c1-24"). I'm correctling these automatically.
"Ic 2602 Sector...", "R Cra Sector...", "Lbn 623 Sector...": these are "IC 2602 Sector", "R CrA Sector", and "LBN 623 Sector" in game.

Thanks, I have corrected things at my end. Still a few left your end (checked in game):

System names differ by case RedWizzard vs Starchart
19 LAMBDA BOOTIS vs 19 Lambda Bootis
ADARA vs Adara
BATARA KALA vs Batara Kala
ED ASICH vs Ed Asich
EK DRACONIS vs EK Draconis
FN BOOTIS vs FN Bootis
LIR vs Lir

 

[RedWizzard]

I am at the edge of populated worlds now. But i am going out for a little trip so i will test your new solution. Feels like a good idea

I've made some improvements: it'll now provide suggestions even after successful trilateration. These suggestions are to systems that it expects will extend the margin of the chosen candidate (i.e. the number of extra distances that the best candidate location matches over the next best candidate). Also any possible suggestions already in the table are indicated. My process now is to enter distances for Sirius, Privir, and Arug (the first 3 defaults) and then enter two more distances using suggestions. Often there are suggestions for the 4th distance among the other default systems, and suggestions for the 5th distance are also often already in the table from previous entries. It all seems to work very well now, at least for systems as far out as I am (about 700 Ly). In fact it works too well: some of the systems I'm submitting TGC is not able to trilaterate even though my entry page is happy to locate them (those cases will get fixed when I next run a cleanup pass over the TGC data).

- - - - - Additional Content Posted / Auto Merge - - - - -

Thanks, I have corrected things at my end. Still a few left your end (checked in game):

System names differ by case RedWizzard vs Starchart
19 LAMBDA BOOTIS vs 19 Lambda Bootis
ADARA vs Adara
BATARA KALA vs Batara Kala
ED ASICH vs Ed Asich
EK DRACONIS vs EK Draconis
FN BOOTIS vs FN Bootis
LIR vs Lir

Thanks, I'm away for the next couple of days but I'll get these fixed when I get back.

 

[Acaelus Thorn]

I'm sorry to butt into the middle of this, but I'm relatively new to the game and didn't want to read this long thread to get the answer. I tried clicking the spoiler on the first post of the thread that claimed it would show how to obtain the remaining coordinates, but it just said todo. I am considering a long term exploration trip and would like to provide as much information as I can to help support the crowd sourcing effort, but I do not know how to obtain the coordinates once I am in system. How can I do this?

 

[Finwen]

Welcome to help out.

I am mortly using the calculation made vy RedWizzard,

I have a cloned html page here: http://robert.astronet.se/Elite/ed-systems/entry.html

Fil in the name of the cirrent system.
And then go to galaxy map. Search for the others and endter distance with 2 decimals.

Afte you have entered 5 or more distances the page will enable Submit to TGC button after it have a solution for the cooredinates.

 

[Inhumierer]

I think you still need to delete this one:
{
"id": 24586,
"name": "Haremid",
"coord": [
-82.40625,
-18.875,
53.09375
],
"cr": 2,
"commandercreate": "Inhumierer",
"createdate": "2015-02-17 18:58:46",
"commanderupdate": "Inhumierer",
"updatedate": "2015-02-17 19:01:10"
},

Yes sir, guilty here ;-)
I noticed I had the wrong system name right after submitting it, but I don't know of a way to remove them again.

This brings me to another point, pls correct me if I am wrong:

EDSC was created by Tornsoul.
There is no sign of life from Tornsoul since December something.
EDSC API is getting slower and slower. My guess is that it's approaching critical mass in a few weeks and will rapidly become slower to unusable.
The DB at EDSC contais some errors (besides my own ones) - some spelling errors, some typos (which will be easy and non critical to correct), and actually 172 pairs of systems with different coordinates given by commanders. These are obvious wrong.
And I bet there are several systems where the trilateration is unsuccessful because there is a typo in 1 or 2 of 6 or 10 distances. Right now for the Auriga site (will go to beta test soon) I have done a complete import from EDSC, and will do export to EDSC and more error hunting this week.

What (I think) we need sooner than later: Have at least one, preferrably a few more, databases like EDSC with APIs to exchange data, including the EDDN data propagation, and not use EDSC as a reference anymore.
Edit: This is the (static) list of problems & unmatched systems: http://the-temple.de/sysmaintenance.php.htm
Edit 2: One more error popped up right now - LTT 7548 (-35.6875, -12.21875, 77.4375)Potriti (-39.625, -6.59375, 62.59375), distance should be 16.35 ly. Got it correct by anonymous CMDR, and again as 16 ly by anonymous.

 

[rmb4253]

Edit: This is the (static) list of problems & unmatched systems: http://the-temple.de/sysmaintenance.php.htm
Edit 2: One more error popped up right now - LTT 7548 (-35.6875, -12.21875, 77.4375)Potriti (-39.625, -6.59375, 62.59375), distance should be 16.35 ly. Got it correct by anonymous CMDR, and again as 16 ly by anonymous.

I noticed a couple of entries in your list with my name on them! What's the answer? Do I need to go back and resubmit the distances to EDSC again? I'm quite happy to do that and some of the other mistakes on the way if it will help.

 

[Inhumierer]

I noticed a couple of entries in your list with my name on them! What's the answer? Do I need to go back and resubmit the distances to EDSC again? I'm quite happy to do that and some of the other mistakes on the way if it will help.

This is data extracted from EDSC, so you (or someone else) have submitted distances using this commader name to EDSC. Since there is no authentication anyone can pick any name (s)he wants. BTW what is your name?

For crowdsourcing the coordinates it would be good if a few commanders - not neccessarily you, doesn't matter in fact - would go there and check the distances again. Since this list contains ONLY pairs of systems with different distances, there MUST be an error in it. But no need to start jumping there and burning fuel like mad, I will come up with a live list and some webinterface around it in short time, hopefully this week. There you will enter the system you are actually in, and the page will show you a list of wrong distances in the databases, ordered by distance to your location. You may then verify, if you want. And there will be a list of "suspicious" distances, which either look odd or prevent triangulation, so they are probably wrong.
I'll ping when it is usable and I have tested it myself with at least a few distances.

Edit: To clarify the contents of this table a bit, first line says: in EDSC, system G 139-50 is located at (-33.84375, 22.96875, 54.25), system ALRAI SECTOR GR-V B2-3 is located at (-25.90625, 35.46875, 65.34375). This would be a distance of 18.5 ly. CMDR Ibizan (or someone claiming to be) submitted 18.6 ly as distance between those systems, and then he again submitted a distance of 18.5 ly, probably a typo.
Next line is similar, but the difference is so small, this may be an error by getting the distance out of the navigation panel, not the GM.

 

[rmb4253]

Ah, I see. Well, I took a trip to the first place on your list last night and submitted several distances from the Alrai Sector one. Definitely 18.5ly to G 139-50!

Your proposed list of dodgy distances sounds an excellent idea and I would definitely use it. I'm not a programmer, so I have no idea how you guys do this stuff, but borrowing an idea from Hastarin's Elite Assistant (which announces if a system I have just entered is not known to EDSC) perhaps you could use the info from the net log to check your list and prompt the user to update EDSC if it is needed?

Good luck with it anyway and thanks.

CMDR Ibizan

 

[RedWizzard]

This brings me to another point, pls correct me if I am wrong:

EDSC was created by Tornsoul.
There is no sign of life from Tornsoul since December something.
EDSC API is getting slower and slower. My guess is that it's approaching critical mass in a few weeks and will rapidly become slower to unusable.
The DB at EDSC contais some errors (besides my own ones) - some spelling errors, some typos (which will be easy and non critical to correct), and actually 172 pairs of systems with different coordinates given by commanders. These are obvious wrong.
And I bet there are several systems where the trilateration is unsuccessful because there is a typo in 1 or 2 of 6 or 10 distances. Right now for the Auriga site (will go to beta test soon) I have done a complete import from EDSC, and will do export to EDSC and more error hunting this week.

Yes, TornSoul has not been around since December. I personally haven't really noticed EDSC getting slower as it's always been pretty slow. It will slow down as the volume of data increases but at least for the systems we've only got about 25% more than we started with. But I do agree it'll need replacing soon, unless the FD API makes it obsolete.

EDSC not being correctable is a big problem. TornSoul was hoping it would be self correcting but that doesn't seem to work in practice.

You might like to take a look at the cleanup I've been doing. I upload corrected versions of the system and distance data to https://github.com/SteveHodge/ed-systems/raw/master/tgcsystems.json and https://github.com/SteveHodge/ed-systems/raw/master/tgcdistances.json. You can see the manual corrections I'm applying here: https://github.com/SteveHodge/ed-systems/blob/master/tgcfixups.json. Finally the code I use to do the cleanup is https://github.com/SteveHodge/ed-systems/blob/master/updatecache.js. It spits out a list of the suspected bad distances as it runs; I can send you the output if you want to compare to the list you have. It's also able to calculate coordinates for nearly 300 systems that EDSC can't, I expect it would handle most or all of the cases you found. I've chosen not to correct most bad distance data even when it's an obvious typo as I don't think it's worth the effort: the system is pretty good at finding the right coordinates even if there are multiple bad distances. The one type of case I do fix is when distances have been entered under the wrong system name.

You can check out the data for individual systems using my sysinfo page. There are copies at http://www.davek.com.au/td/ed-systems-master/sysinfo.html and http://robert.astronet.se/Elite/ed-systems/sysinfo.html. Wait for it to load the data (which can be quite slow), then just search for a system, e.g. ALRAI SECTOR GR-V B2-3. It reruns the trilateration and will tell you which distances don't match the calculated position of the system and what they should be. It also lists all known nearby systems which is useful for comparison with the nav panel.

 

[Inhumierer]

Hi there, awesome work up to now! Keep it up, we're getting closer!

1. Torn Soul has disappeared, so EDSC may have to be re-written. 2. Dealing with the 'Needle Issue' where trilateration potentially becomes less accurate when dealing with systems a long way and/or in a relatively direct line from known stars.

IF EDSC does get re-written, could I suggest that when a user enters details for a new system, it does a quick calculation of the relative angles of the reference systems to the new target. It will know more or less where the new system is, and also if its coordinates are potentially incorrect (because the relative angles are low - ie the needle effect).

1.) Yes, we are working on a EDSC replacement. EDDB looks very good, waiting for a "real" api. Dumping static files once a day is not an API... I'm working on a similar project, together with sngerous. Currently importing EDSC works pretty well, next thing will be entering distances and submitting, or maybe incorporating EDDB data.
2.) Makes me smile I had the same thought, late December or early January, when I did the triangulation/trilateration in PHP, and I checked the angles. Did not really help. Had good coordinates with some small angles, and had bad coordinates with larger angles. I could not find any relation between angles and preciseness of the calculated coordinates, so I ignored the angles. Recalculating the distances and checking if they are within the 2 digit precision given by the GM was way better.

Besides that: I still think the "needle problem" is in fact a "plate or lens problem", but I was too lazy to do some visualisation, and it's only cosmetical. But would ne nice to see. Anyone got an easy way to make some 3D data visualisation?

 

[Inhumierer]

You might like to take a look at the cleanup I've been doing.

I did, great work! I'll inject this into my copy soon. Thanks! Much appreciated!

You can see the manual corrections I'm applying here: https://github.com/SteveHodge/ed-systems/blob/master/tgcfixups.json.

Is this a fixed location? I could then integrate this in my half-automatic update.

Finally the code I use to do the cleanup is https://github.com/SteveHodge/ed-systems/blob/master/updatecache.js. It spits out a list of the suspected bad distances as it runs; I can send you the output if you want to compare to the list you have. It's also able to calculate coordinates for nearly 300 systems that EDSC can't, I expect it would handle most or all of the cases you found.

Yes, output would be fine, tnx again. A live, online version would be even better.
Edit: In EDSC there are actually 3218 systems with 5 or more distances but no known coordinates. EDDB & your script can handle some of them, will integrate these into my own DB.

You can check out the data for individual systems using my sysinfo page.

Man, you should jump into the boat at EDDB or sngerous' & my site Auriga, coming to a theatre near you soon
You do awesome java stuff, I'm too stupid for this.

 

[Finwen]

I have also started working on a replacement for EDSC. Mainly because i want something for myself with a webinterface and have a backup if EDSC dies.

I think several online databases will be good for redundancy and sharing data between them is really good.

 

[RedWizzard]

2.) Makes me smile I had the same thought, late December or early January, when I did the triangulation/trilateration in PHP, and I checked the angles. Did not really help. Had good coordinates with some small angles, and had bad coordinates with larger angles. I could not find any relation between angles and preciseness of the calculated coordinates, so I ignored the angles. Recalculating the distances and checking if they are within the 2 digit precision given by the GM was way better.

Besides that: I still think the "needle problem" is in fact a "plate or lens problem", but I was too lazy to do some visualisation, and it's only cosmetical. But would ne nice to see. Anyone got an easy way to make some 3D data visualisation?

Larger angles between references definitely helps but I think it's more an artifact of the 1/32 Ly grid and distance error in our case. The answer definitely seems to be to figure out a candidate region in some way and then check each point in that region to see if the distances match.

For 3D visualisation WebGL is fairly easy to get into. I've only had a little play with it but I used it to make a simple visualisation of all the systems we have (http://www.davek.com.au/td/ed-systems-master/systemsvis.html).

- - - - - Additional Content Posted / Auto Merge - - - - -

I did, great work! I'll inject this into my copy soon. Thanks! Much appreciated!

Is this a fixed location? I could then integrate this in my half-automatic update.

Yes, output would be fine, tnx again. A live, online version would be even better.
Edit: In EDSC there are actually 3218 systems with 5 or more distances but no known coordinates. EDDB & your script can handle some of them, will integrate these into my own DB.

Man, you should jump into the boat at EDDB or sngerous' & my site Auriga, coming to a theatre near you soon
You do awesome java stuff, I'm too stupid for this.

It is a fixed location. The mirrors Finwen and maddavo are hosting are also options (http://robert.astronet.se/Elite/ed-systems/tgcfixups.json and http://www.davek.com.au/td/ed-systems-master/tgcfixups.json).

I will definitely switch over to another DB/API once someone has a replacement for EDSC/TGC up and running. I may write one myself but I'm waiting to see what the FD API has first.

Here's a slightly simplified copy of the output of my cleanup process: View attachment 18090

 

[Inhumierer]

Thanks again, very interesting.

I'll take a look at WebGL if I find some spare time.

Our database will be a sufficient replacement for EDSC, but it will take a few weeks until it is ready for use.

 

[Inhumierer]

Yay, WebGL doesn't look trivial if you're not into Javascript...

Anyway I integrated your tgcfixups.json - works fine except 2 systems.

Ditibi is a PITA. Your json tells my php to delete system Ditibi, ok. Then rename "Ditibi (fixed)" to "Ditibi", ok. Next run it deletes Ditibi, but this is already the fixed system, so I have to put it back in by hand.
And with Xi Hugan & Zi Hungan something is messed up badly, they keep flipping every time I run the script.

This reduced my "Cat 1 errors" (system pairs with different distances submitted) from 172 to 155, and the systems without coordinates but at least 5 known references went down to 65.

I will take the whole DB dump from EDDB and your GIT repo and compare it to my database in the next few days, and this will bring down the numbers for sure.

So what do you think is the actually preferred way to enter new distances to find new coordinates? EDSC is unsupported and contains some uncorrectable errors. We need a replacement ASAP, that's for sure.
For the curious: the live dump of these distances & systems can be found at http://the-temple.de/public/sysmaintenance.php (may take some time to load, heavy SQL queries inside)

 

[Finwen]

I am working on a REST interface now for my database. I think multiple replacements for redundancy is good to have.

Not much to see yet. But here is the working getSystem request

 

[RedWizzard]

Ditibi is a PITA. Your json tells my php to delete system Ditibi, ok. Then rename "Ditibi (fixed)" to "Ditibi", ok. Next run it deletes Ditibi, but this is already the fixed system, so I have to put it back in by hand.

And with Xi Hugan & Zi Hungan something is messed up badly, they keep flipping every time I run the script.

This reduced my "Cat 1 errors" (system pairs with different distances submitted) from 172 to 155, and the systems without coordinates but at least 5 known references went down to 65.

The fixes are intended to be applied to the unmodified dataset from TGC/EDSC. As you found there could be problems if you reapply the same fixes to already corrected data. What you could do is keep track of how many fixes you've applied and then just apply new ones I've added (i.e if last time you applied the first 120 fixes then next time just start from the 121st). I'll always append to that file rather than editing existing entries unless I have no option.

One thing that might not be obvious with what I'm doing is that if I have an update of a system name from X to Y and there is already a system Y, any data from X should be merged into Y, specifically distances (removing any duplicates). If the existing data in Y is bad then I'll delete that system or the distances first.

The reason I fixed Ditibi the way I did is that it had a lot of existing distances that were incorrect. It was easier to move the few correct distances and then delete the entire system than deleting all the bad distances. This situation occurred because FD moved Ditibi quite late (after release IIRC).

The Zi Hungan/Xi Hungan/Xi Hugan thing was my mistake. It was renaming Xi Hugan to Zi Hungan and then to Xi Hungan. Zi Hungan should never have been involved. I've fixed this in the fixups. Not sure why it was oscillating for you though.

The reduction in "cat 1" errors is the distances that were pointing to incorrect systems. The reduction of systems without coordinates but with 5+ known references would be due to merging duplicate systems I think.

I will take the whole DB dump from EDDB and your GIT repo and compare it to my database in the next few days, and this will bring down the numbers for sure.

So what do you think is the actually preferred way to enter new distances to find new coordinates? EDSC is unsupported and contains some uncorrectable errors. We need a replacement ASAP, that's for sure.
For the curious: the live dump of these distances & systems can be found at http://the-temple.de/public/sysmaintenance.php (may take some time to load, heavy SQL queries inside)

I'd say keep submitting to EDSC in the meantime. Try not to make any mistakes; I recommend not using the EDSC webpage to add systems as there is very little feedback before you submit. Personally I think my page does a better job If anyone does make any mistakes that they notice after submitting they can let me know (here or via PM) and I'll add them to my corrections.

 

[Athan]

The fixes are intended to be applied to the unmodified dataset from TGC/EDSC. As you found there could be problems if you reapply the same fixes to already corrected data. What you could do is keep track of how many fixes you've applied and then just apply new ones I've added (i.e if last time you applied the first 120 fixes then next time just start from the 121st). I'll always append to that file rather than editing existing entries unless I have no option.

Why not do as EDSC does on its output and have a "lastupdated" field per fixup, then people can apply starting with the first that has a lastupdated after the previous last one they'd applied ?

 

[Inhumierer]

The fixes are intended to be applied to the unmodified dataset from TGC/EDSC. As you found there could be problems if you reapply the same fixes to already corrected data.

I keep a database and poll the latest updates from EDSC (with some small overlapping), and I keep getting bad distances from EDSC, so I wanted to apply your fixes repeatedly. Works well except for these 2 systems. Now I skip the first 495 corrections, no problem

I'd say keep submitting to EDSC in the meantime. Try not to make any mistakes; I recommend not using the EDSC webpage to add systems as there is very little feedback before you submit. Personally I think my page does a better job If anyone does make any mistakes that they notice after submitting they can let me know (here or via PM) and I'll add them to my corrections.

I thought your website uses a local copy of the database. I overlooked the "transmit"-Button Will give it a try, maybe tonight. Got no spare time, my Notebook just came today - I'm very curious if it can run ED :=