Discussion What is the most efficient way to crowdsource the 3D system coordinates

Snuble · Feb 3, 2015

graham.reeds said:
You will only be able to get coordinates for the portion of the galaxy that is hardcoded, the rest of the galaxy is a seeded number.

I see absolutely no reason why the CLIENT (or server if that is part of the network traffic on entering a system) can not tell us the coordinates for the nearest 20LY systems from whereever you are. It effectively have to know this already since it provides this to the target panel already. For the same reason I fail too see why the Client can not just provide coordinates in game since it is able to give us a 2 digit distance across the galaxy.

For mapping purposes, I could live with the distances to the closest systems, but it would make a commanders paperwork so much easier if the coords where provided. I'm afraid FD is overthinking this, a plain text dump when entering a system or visiting a market would be the simplest and most robust approach for 3rd party services.

Athan · Feb 3, 2015

Snuble said:
I see absolutely no reason why the CLIENT (or server if that is part of the network traffic on entering a system) can not tell us the coordinates for the nearest 20LY systems from whereever you are. It effectively have to know this already since it provides this to the target panel already. For the same reason I fail too see why the Client can not just provide coordinates in game since it is able to give us a 2 digit distance across the galaxy.

More to the point the client has to know the relative 3D positions of all stars within ~100ly of the viewpoint so it can actually render them correctly.

If people remember back when we first got accurate co-ordinates in one of the later betas it was because someone used a debugger to find the internal co-ordinates used for that. These were all relative to a single point (down/left/backwards when viewing the core from Sol).

So of course it's easy for the client to have an API call to spit out all stars within X ly of Y position. If any API is only server-side then instead you're asking for something like the cartographics server to spit out all stars within X ly of Y, so not much difference.

RedWizzard · Feb 4, 2015

Athan said:
So of course it's easy for the client to have an API call to spit out all stars within X ly of Y position. If any API is only server-side then instead you're asking for something like the cartographics server to spit out all stars within X ly of Y, so not much difference.

It's pretty clear from what Michael has said that the client is capable of generating the coordinates of all systems in the galaxy. The issue is how many systems it's practical to provide via the API at once. I'd be surprised if we can't get all the systems within 20 Ly at least. But I think more than a couple of hundred Ly is unlikely.

RedWizzard · Feb 4, 2015

I've finally finished writing some code to clean up the TGC data. There's still some bad data in it but it's much better than it was. Currently the data has had about 200 issues fixed (typos, distances applied to the wrong systems, test systems deleted etc), and coordinates calculated for about 220 systems that TGC can't locate. A lot of system names have had capitalisation corrected as well.

The corrected data is available from my github repo:
https://github.com/SteveHodge/ed-systems/raw/master/tgcsystems.json
https://github.com/SteveHodge/ed-systems/raw/master/tgcdistances.json

The file format is identical to a response from TGC, so you should be able to just load them as if TGC had sent the data. Use the date property of the root objects to fetch any more recent data from TGC. E.g.:

tgcsystems.json has contains this:

Code:

{
  "ver": 2,
  "date": "2015-02-04 12:44:11",
  "status": {
    "input": [
      {
        "status": {
          "statusnum": 0,
          "msg": "Success"
        }
      }
    ]
  },
  "systems": [
...
  ]
}

This is the output of a call to TGC with outputmode = 2, date = '2014-09-09 12:13:14Z', and cr = 0, with the fixes applied. To get any newer data call TGC GetSystems with outputmode = 2, date = '2015-02-04 12:44:11Z' (date from the file, with 'Z' appended), and cr = 0. Merge the data by replacing any systems from tgcsystems.json with the updated versions from TGC.

Athan · Feb 4, 2015

RedWizzard said:
The corrected data is available from my github repo:
https://github.com/SteveHodge/ed-systems/raw/master/tgcsystems.json
https://github.com/SteveHodge/ed-systems/raw/master/tgcdistances.json

This won't necessarily work right off the bat for people, depending on how they're pulling the data. My script that works fine with EDSC spits out:

Code:

15:59:58 1$ ./get-edsc.pl
Failed to retrieve GetSystems: 403 Forbidden
Failed to retrieve GetSystems: 
Cookies must be enabled to use GitHub.

I'll see if I can find the minimum necessary tweaks to get it to work. I may end up just pulling that file to local and pointing the script at that, just be aware that it's not as simple as temporarily putting the different URL in.

Athan · Feb 4, 2015

RedWizzard said:
The file format is identical to a response from TGC, so you should be able to just load them as if TGC had sent the data. Use the date property of the root objects to fetch any more recent data from TGC. E.g.:

The entire thing needs to be wrapped in:

Code:

{"d": ... }

in order to exactly match TGC/EDSC output.

Andargor · Feb 4, 2015

RedWizzard said:
It's pretty clear from what Michael has said that the client is capable of generating the coordinates of all systems in the galaxy. The issue is how many systems it's practical to provide via the API at once. I'd be surprised if we can't get all the systems within 20 Ly at least. But I think more than a couple of hundred Ly is unlikely.

In my (rare) spare time, I'm messing with OpenCV to see if I can do 3D reconstruction of the galaxy map as I navigate it. I have no idea if this will be of any use, as the best case is only getting "valid" but anonymous star positions, since the names will still need to be OCRed. But it's a fun exercise

It might be useful to extract extra info using the filters (star types, economies, allegiances, etc.), and then see if they match your list. All this will probably be moot once there is an API.

*experimental goggles on*

Andargor · Feb 4, 2015

Michael has a dump of the "first discovered by" star/player pairs. Would it be worth asking him for it, and integrate with the star DB?

Ref: https://forums.frontier.co.uk/showthread.php?t=109599

EDIT: Maybe ask him for the coords at the same time

RedWizzard · Feb 5, 2015

Athan said:
This won't necessarily work right off the bat for people, depending on how they're pulling the data. My script that works fine with EDSC spits out:

Code:

15:59:58 1$ ./get-edsc.pl Failed to retrieve GetSystems: 403 Forbidden Failed to retrieve GetSystems: Cookies must be enabled to use GitHub.

I'll see if I can find the minimum necessary tweaks to get it to work. I may end up just pulling that file to local and pointing the script at that, just be aware that it's not as simple as temporarily putting the different URL in.

Yeah, I don't think github is really intended for serving files. I'd suggest grabbing a copy and then accessing it locally. I probably won't be updating it all that often anyway, maybe once or twice a week.

Athan said:
The entire thing needs to be wrapped in:

Code:

{"d": ... }

in order to exactly match TGC/EDSC output.

I forgot about that. It always seemed pointless to me

Karic Hovath · Feb 5, 2015

Is there a method to correct coordinates?

In Komovoy and ran a trade run using Trade Dangerous. First jump and hop is to Ditibi. Galaxy Map shows that's a distance of 71.51ly trade nav Komovoy Ditibi --ly 12.13 returns the following:

System JumpLy
---------------
KOMOVOY 0.00
DITIBI 2.43

Slight discrepancy there.

How can I go about helping to correct the coordinates?

Athan · Feb 5, 2015

Karic Hovath said:
Is there a method to correct coordinates?

In Komovoy and ran a trade run using Trade Dangerous. First jump and hop is to Ditibi. Galaxy Map shows that's a distance of 71.51ly trade nav Komovoy Ditibi --ly 12.13 returns the following:

System JumpLy
---------------
KOMOVOY 0.00
DITIBI 2.43

Slight discrepancy there.

How can I go about helping to correct the coordinates?

If you're using the miggy.org route planner (mine) I'll correct the DB, otherwise you'll need to prod whoever runs the one you're using.

Andargor · Feb 5, 2015

Michael posted the discovery data, so it would be cool to merge that with the systems file, for the stars at least:

https://docs.google.com/uc?id=0B5Px3TF3wE0PRXI2M25pRUtPVU0&export=download

EDIT: In CSV format from someone else
EDIT2: The CSV format I posted earlier seems to be missing data, it's a lot smaller than a text dump of the excel (32 MB vs 400 MB)

Karic Hovath · Feb 5, 2015

Fixing Incorrect coordinates

Athan said:
If you're using the miggy.org route planner (mine) I'll correct the DB, otherwise you'll need to prod whoever runs the one you're using.

This was not planned using your planner but, having just checked, your planner has the same error as shown:

#	System Name	Jump Distance	Total Jumped	Total Distance
0	Komovoy	0.00	0.00	0.00
1	Ditibi	2.43	2.43	2.43

Trade Dangerous (https://bitbucket.org/kfsone/tradedangerous/wiki/Home) is the route planner used, loaded with Maddavo's Market Data (http://www.davek.com.au/td/) which gets system data via Elite Dangerous Star Coordinator (http://www.edstarcoordinator.com/).

I'm happy to do the legwork needed to correct the system coords but have no way to submit them anywhere.

Snuble · Feb 6, 2015

http://edstarcoordinator.com/

Go to the system (in game) that you want to get coords for, start getting distances. Remember to open the galaxy map to get distance because the distances in target panel can be off be .0 something and the 1 decimal is not good enough for distances above 10LY

If you get distances for the systems that autocomplete on the link I posted, you most likely get the correct coords for the system in first try.

Finwen · Feb 8, 2015

RedWizzard said:
I've finally finished writing some code to clean up the TGC data. There's still some bad data in it but it's much better than it was. Currently the data has had about 200 issues fixed (typos, distances applied to the wrong systems, test systems deleted etc), and coordinates calculated for about 220 systems that TGC can't locate. A lot of system names have had capitalisation corrected as well.

The corrected data is available from my github repo:
https://github.com/SteveHodge/ed-systems/raw/master/tgcsystems.json
https://github.com/SteveHodge/ed-systems/raw/master/tgcdistances.json

Thanks for your work. Will update my program with your cleaned up data.

RedWizzard · Feb 10, 2015

I've just updated my corrected TGC json files. Get them from https://github.com/SteveHodge/ed-systems.

There is a major problem in TGC now: there are now duplicate versions of Sol, Cayu, Juro, Kato, and Marra that have exactly the same names as the FD versions. The coordinates are also the same so it might not cause too many problems for people using the data but it's a potential issue for people entering data and a big problem for me when verifying the data because TGC is not outputting any new distances entered to those systems. This means there are systems such as HIP 4318 that TGC outputs coordinates for but doesn't output enough distances to verify those coordinates.

Since any new distances to Sol, Cayu, Juro, Kato, and Marra seem to be irretrievable I recommend people stop using those systems as references. I've changed the reference systems on my entry.html page (as all these systems are among my defaults), so please grab a new version from https://github.com/SteveHodge/ed-systems/raw/master/entry.html if you're using that page. I'll probably change the page in the next few days to explicitly reject distances to these systems.

I hope no more systems get duplicates otherwise we're going to need a replacement for TGC quickly (assuming TornSoul doesn't reappear to fix it).

RedWizzard · Feb 15, 2015

Karic Hovath said:
Is there a method to correct coordinates?

In Komovoy and ran a trade run using Trade Dangerous. First jump and hop is to Ditibi. Galaxy Map shows that's a distance of 71.51ly trade nav Komovoy Ditibi --ly 12.13 returns the following:

System JumpLy
---------------
KOMOVOY 0.00
DITIBI 2.43

Slight discrepancy there.

How can I go about helping to correct the coordinates?

Finally got around to looking into this. The coordinates for Ditibi changed sometime between Jan 3 and Jan 10, most likely when 1.04 was released on Jan 4. Lu Yupik, Macalites, and Oguninksmi have also moved and I'm sure there are others.

I've corrected these systems in my corrected TGC data. Unfortunately Ditibi is used as a reference for over 400 other systems. As I don't believe those have all moved I've decided to keep the old Ditibi which I've renamed to "Ditibi (old location)" and added a new Ditibi which has the correct coordinates based on current distances (the new Ditbi will show up in data from the TGC server as "Ditibi (fixed)". The other systems don't have much old data so I've just moved them to their new locations. The latest corrected data is available from my github repo (https://github.com/SteveHodge/ed-systems): tgcsystems.json and tgcdistances.json. There are now nearly 300 corrections (plus almost 1800 capitalisation corrections), coordinates for over 200 systems with no coordinates in TGC. Anyone using TGC data really should check it out.

Snuble said:
http://edstarcoordinator.com/

Go to the system (in game) that you want to get coords for, start getting distances. Remember to open the galaxy map to get distance because the distances in target panel can be off be .0 something and the 1 decimal is not good enough for distances above 10LY

If you get distances for the systems that autocomplete on the link I posted, you most likely get the correct coords for the system in first try.

I'm not sure if submitting corrected distances actually causes TGC to recalculate coordinates - I haven't really seen any evidence that it does. It probably wouldn't work in this case anyway as there are over 400 distances consistent with the old location. This is one of the flaws with TGC: there isn't really any way to correct bad data. If TGC is still relevant once FD release the API we're going to need a new system.

Smacker · Feb 16, 2015

RedWizzard said:
Finally got around to looking into this. The coordinates for Ditibi changed sometime between Jan 3 and Jan 10, most likely when 1.04 was released on Jan 4. Lu Yupik, Macalites, and Oguninksmi have also moved and I'm sure there are others.

I've corrected these systems in my corrected TGC data. Unfortunately Ditibi is used as a reference for over 400 other systems. As I don't believe those have all moved I've decided to keep the old Ditibi which I've renamed to "Ditibi (old location)" and added a new Ditibi which has the correct coordinates based on current distances (the new Ditbi will show up in data from the TGC server as "Ditibi (fixed)". The other systems don't have much old data so I've just moved them to their new locations. The latest corrected data is available from my github repo (https://github.com/SteveHodge/ed-systems): tgcsystems.json and tgcdistances.json. There are now nearly 300 corrections (plus almost 1800 capitalisation corrections), coordinates for over 200 systems with no coordinates in TGC. Anyone using TGC data really should check it out.

I'm not sure if submitting corrected distances actually causes TGC to recalculate coordinates - I haven't really seen any evidence that it does. It probably wouldn't work in this case anyway as there are over 400 distances consistent with the old location. This is one of the flaws with TGC: there isn't really any way to correct bad data. If TGC is still relevant once FD release the API we're going to need a new system.

Thanks RedWizzard, I found the following position changes (TGC vs EDDB):

Code:

System positions differ Alrai Sector WI-T a3-3 [-12.3125,-28.65625,42] vs [-6.75,-25.4375,50]
System positions differ Cephei Sector KR-W b1-0 [-72.96875,13.03125,-57.90625] vs [-75.0625,9.5625,-61.34375]
System positions differ Col 285 Sector NS-U a32-0 [105.34375,-42.25,-4.75] vs [105.125,-26.25,-4.8125]
System positions differ Crucis Sector MD-S b4-6 [56.6875,44,80.03125] vs [56.625,43.90625,79.84375]
System positions differ Ditibi [28.9375,-11.53125,3.46875] vs [68.9375,28.46875,43.46875]
System positions differ lhs 1375 [-9.125,-19.3125,-17.5] vs [-9.625,-21.90625,-16.625]
System positions differ Lu Yupik [21.53125,-19.21875,4.25] vs [70.53125,24.53125,41.15625]
System positions differ Macalites [31.53125,-18.21875,0.46875] vs [76.53125,21.78125,40.46875]
System positions differ Oguninksmii [66,29.84375,53.34375] vs [80,47.84375,53.34375]
System positions differ Tascheter Sector FG-X b1-1 [39.15625,-0.34375,-69.28125] vs [37.75,-3.84375,-68.46875]
System positions differ Tascheter Sector ST-R a4-0 [-10.1875,-36.3125,-56.375] vs [-13.25,-37.03125,-59.1875]

EDDB could be somewhere that can takeover from EDSC as a location for TGC star positions.

RedWizzard · Feb 17, 2015

Smacker said:

Thanks RedWizzard, I found the following position changes (TGC vs EDDB):

Code:

System positions differ Alrai Sector WI-T a3-3 [-12.3125,-28.65625,42] vs [-6.75,-25.4375,50]
System positions differ Cephei Sector KR-W b1-0 [-72.96875,13.03125,-57.90625] vs [-75.0625,9.5625,-61.34375]
System positions differ Col 285 Sector NS-U a32-0 [105.34375,-42.25,-4.75] vs [105.125,-26.25,-4.8125]
System positions differ Crucis Sector MD-S b4-6 [56.6875,44,80.03125] vs [56.625,43.90625,79.84375]
System positions differ Ditibi [28.9375,-11.53125,3.46875] vs [68.9375,28.46875,43.46875]
System positions differ lhs 1375 [-9.125,-19.3125,-17.5] vs [-9.625,-21.90625,-16.625]
System positions differ Lu Yupik [21.53125,-19.21875,4.25] vs [70.53125,24.53125,41.15625]
System positions differ Macalites [31.53125,-18.21875,0.46875] vs [76.53125,21.78125,40.46875]
System positions differ Oguninksmii [66,29.84375,53.34375] vs [80,47.84375,53.34375]
System positions differ Tascheter Sector FG-X b1-1 [39.15625,-0.34375,-69.28125] vs [37.75,-3.84375,-68.46875]
System positions differ Tascheter Sector ST-R a4-0 [-10.1875,-36.3125,-56.375] vs [-13.25,-37.03125,-59.1875]

EDDB could be somewhere that can takeover from EDSC as a location for TGC star positions.

Yeah, those differences are all due to corrections I've made. The EDDB coordinates mostly look like the uncorrected TGC values. I thought EDDB was just price data... is it relaying coordinates from TGC or something?

There are few more you haven't picked up:

Code:

alrai sector ir-w c1-29: coords different: TGC (-54.34375, -10.09375, 88.6875), calculated (-58.78125, -14.3125, 92.90625)
Bob: coords different: TGC (-25.375, 1.125, -20.84375), calculated (-24.53125, -0.4375, -46.65625)
Hyades Sector VO-Q b5-1: coords different: TGC (52.375, -92.34375, -162.625), calculated (48.28125, -79.5, -154.40625)
LFT 179: coords different: TGC (10.59375, -75.875, -9.03125), calculated (13.65625, -70.34375, -16.4375)

Smacker · Feb 17, 2015

RedWizzard said:
Yeah, those differences are all due to corrections I've made. The EDDB coordinates mostly look like the uncorrected TGC values. I thought EDDB was just price data... is it relaying coordinates from TGC or something?

There are few more you haven't picked up:

Code:

alrai sector ir-w c1-29: coords different: TGC (-54.34375, -10.09375, 88.6875), calculated (-58.78125, -14.3125, 92.90625) Bob: coords different: TGC (-25.375, 1.125, -20.84375), calculated (-24.53125, -0.4375, -46.65625) Hyades Sector VO-Q b5-1: coords different: TGC (52.375, -92.34375, -162.625), calculated (48.28125, -79.5, -154.40625) LFT 179: coords different: TGC (10.59375, -75.875, -9.03125), calculated (13.65625, -70.34375, -16.4375)

Thanks, I will check those, I think they are not in EDDB yet.
EDDB has extended system data (coordinates and names based on EDSC) + stations as well as trade data
Take a look here: ROSS where you can edit the data manually (it also scrapes info from the EDDN stream)
Or at the API where you can download json files (updated every 24 hours)
I will endeavour to keep EDDB up-to-date with your files