Discussion What is the most efficient way to crowdsource the 3D system coordinates

[RedWizzard]

OK I'm done for the night, there will be no further changes to my systems.json until tomorrow. I have also verified these Wredguia stars definitely don't exist any more:

I've updated my systems.json with your changes. Everything is the same right now except you haven't got the corrections maddavo pointed out:

G 41804 should be called G 14-6
LP 18994 is an incorrectly named duplicate and should be deleted
LP 32264 is an incorrectly named duplicate and should be deleted
LP 41909 is an incorrectly named duplicate and should be deleted

 

[RedWizzard]

I've confirmed all Wredguia/Wregoe systems from beta 2 have been changed. Searching for them takes you to systems with different names (often "X Sector XX-X x#-#" systems), which is a bug. More of a problem is that searching for the new names (if they are "...Sector..." names) doesn't work. I've ticketed this.

Systems.json updated. Current counts are:
976 systems
810 are reference systems
32 are verified crowdsourced systems
134 are unverified crowdsourced systems.

 

[Snuble]

Looks like another change from Beta2 is that system information is often masked in the galaxy map. Several systems from the new export do not say what population or economy they have. This is revealed when actually visiting a system.

Also it looks like the export is not a complete list of systems with economy. Just now I found one close to the Beta core (Aulin etc) that where not in the export, but had an economy. This was also without me visiting the system, so it was already revealed in galaxy map.

I'm also curious to how the new visit restriction system will work. Will it just deny you the ability to jump into a system if your ranking with the ruling faction is too low, or will it just give you a slap if system security finds you or restrict docking? Time will tell.

 

[RedWizzard]

Looks like another change from Beta2 is that system information is often masked in the galaxy map. Several systems from the new export do not say what population or economy they have. This is revealed when actually visiting a system.

Are they "unknown" systems for you? It would make sense that you can't see economy etc if you don't have system info for them.

I'm also curious to how the new visit restriction system will work. Will it just deny you the ability to jump into a system if your ranking with the ruling faction is too low, or will it just give you a slap if system security finds you or restrict docking? Time will tell.

In FE2 and FFE it was a slap from the authorities if you were caught. I think that could be toughened up a little by also denying landing clearance if you don't have the permit. I'd be disappointed if you were denied the ability to jump though - I don't think that would make sense.

 

[Harbinger]

I've confirmed all Wredguia/Wregoe systems from beta 2 have been changed. Searching for them takes you to systems with different names (often "X Sector XX-X x#-#" systems), which is a bug. More of a problem is that searching for the new names (if they are "...Sector..." names) doesn't work. I've ticketed this.

Systems.json updated. Current counts are:
976 systems
810 are reference systems
32 are verified crowdsourced systems
134 are unverified crowdsourced systems.

Updating to your latest version.

I just noticed, you should remove all the previous measurements to Fu Haiting. The position of this star changed between Beta 2 and Beta 3.

For example, I'm rechecking Hemsut now and previously the measurement was 44.531, currently it's 60.72. That's quite a change in it's position.

EDIT: Fu Haiting moved from (-69.46875, 32.125, -48.875) to (-103, 29.46875, -43.4375) when Beta 3 was released.

 

[Snuble]

Are they "unknown" systems for you? It would make sense that you can't see economy etc if you don't have system info for them.

If it was up to me that information would be part of system scanning. And docks hidden as "unknown signal sources". But still, it is good that the status of a system is unknow even outside of the game until someone visits and share the info. It would make sense that these systems where refinery/extraction type systems. I would think larger populations would be known across the populated galaxy.

 

[TargetLost]

Don't know what you need 3D coords for but is there still a need for 3D coordination calculation?
I guess I could do such but you already have something i read.

Would require a set of fix points (star systems) with know coords in minimum 4 but could be N and for each unknown system at least 4 distances to other known or unknown stystems.
p.s. what's with that grid you mention in this thread. Are all the stars in a grid or is it just an artificial help construct for your calculations?

 

[Smacker]

Updating to your latest version.

I just noticed, you should remove all the previous measurements to Fu Haiting. The position of this star changed between Beta 2 and Beta 3.

For example, I'm rechecking Hemsut now and previously the measurement was 44.531, currently it's 60.72. That's quite a change in it's position.

EDIT: Fu Haiting moved from (-69.46875, 32.125, -48.875) to (-103, 29.46875, -43.4375) when Beta 3 was released.

Using your latest, thanks. Notice a few mistakes:

Missing Stations
'Aulin','Aulin Enterprise',0.0
'Lalande 29917','Cori Terminal',0.0

Missing Systems
'G 14-6',13.5,30.375,9.0625,'Beta3','2014-11-01 06:36:37'

Misnamed Systems
Thottacahuan ==> Thottacahuan W&ng  (&==>a)

Invalid Systems
'G 41804',13.5,30.375,9.0625,'Beta3','2014-11-01 06:36:37'
'LP 18994',-49.9375,28.8125,-33.28125,'Beta3','2014-11-01 06:36:37'
'LP 32264',-27.28125,19.59375,-23.03125,'Beta3','2014-11-01 06:36:37'
'LP 41909',-70.40625,25.4375,-32.40625,'Beta3','2014-11-01 06:36:37'

 

[TornSoul]

p.s. what's with that grid you mention in this thread. Are all the stars in a grid or is it just an artificial help construct for your calculations?

The stars are in a grid (discrete coordinates) with 1/32 LY precision.

We know this from the data supplied by FD for SB2.

We've also reverse engineered that FD is using float vs double precision when calculating distances.
As some distances do not match the ingame reported distances if you use double precision(*)
Just a heads up if you start down this road.

(*) This was verified in multiple instances in SB2, where we had 3 digits of precision for distances - Where in some (rare) cases our math didn't fit the distances reported in the Galaxy map.
Now in SB3, with only 2 digits of precision, this might be harder to verify.

 

[TargetLost]

Thanks.
Well the math is a bit tempting to me. I think it's possible to solve for the whole star net with just the distances and 4 or better more fix points in one step.
Say for ever new system it should be sufficient enough to just tell the distances to it's nighbours which you can read out easily and best have the whole net recalculated so it makes the best out of all data.
But I saw you already have a working system so I guess I spare me the work.

 

[TornSoul]

The math is well understood : http://en.wikipedia.org/wiki/Trilateration

It's fundamental for how GPS works

The nasty bit is when the real world don't play nice (ie. when distances are a bit "fuzzy")

 

[wolverine2710]

Commanders. I´m away this week till friday. No ED access but some internet access. Can read it all but writing time is limited.
It seems the following is going.

1. A few commanders are checking the old SB2 list for validity and RW is updating TOR. Absolutely needed.
2. Have all tool commanders checked if their tool can calculate accurate (or good enough) 3D coords based on the now 2dp GM input.
3. Are commanders already taking new distances for the SB3 sausage?
4. The Great Collector. TornSouls. Are you still convinced that TGR is needed and are you able/willing to setup a alpha TGR so tool commanders can make the needed changes and test it?
5. It seems all tool commanders and other volunteers are still with us. Great work guys/galls!!!!!!

 

[Harbinger]

Using your latest, thanks. Notice a few mistakes:

Missing Stations
'Aulin','Aulin Enterprise',0.0
'Lalande 29917','Cori Terminal',0.0

Missing Systems
'G 14-6',13.5,30.375,9.0625,'Beta3','2014-11-01 06:36:37'

Misnamed Systems
Thottacahuan ==> Thottacahuan W&ng  (&==>a)

Invalid Systems
'G 41804',13.5,30.375,9.0625,'Beta3','2014-11-01 06:36:37'
'LP 18994',-49.9375,28.8125,-33.28125,'Beta3','2014-11-01 06:36:37'
'LP 32264',-27.28125,19.59375,-23.03125,'Beta3','2014-11-01 06:36:37'
'LP 41909',-70.40625,25.4375,-32.40625,'Beta3','2014-11-01 06:36:37'

With regards to missing stations, it's just that I haven't been to that system in Beta 3 so haven't logged the station names/distances yet.

You're probably better off using RedWizzard's latest systems.json from his Github as your primary source of data. He's imported my stations list into it now. In fact I've changed the systems.json file I'm sharing to only include modifications I've made recently to make it easier for RedWizzard. Once I notice RedWizzard has imported them I'll update my shared file as appropriate.

You can of course keep monitoring my shared file to grab data as and when it becomes available.

 

[KevinMillican]

There really is a lot changed outside the core systems:

• Lost Bixby and Wingrove from Styx - replaced by Chu Hub and Gerst Port
• Wolf 654 has lost Diesel Survey which was one of the more interesting outposts in Beta 2 (and no replacement station at all now).

http://www.edpilotlog.co.uk/Station_Photos_list.php

I'm finding that station services are changed as well. This is just touching the surface as I have only been able to try beta 3 this weekend.

It won't stop me developing the edpilotlog tool but I wonder how much input is really worthwhile until FD settle the systems and stations ready for gamma release?

 

[TornSoul]

1. The Great Collector. TornSouls. Are you still convinced that TGR is needed and are you able/willing to setup a alpha TGR so tool commanders can make the needed changes and test it?

Working on TGC - Been running a lot of test today to verify the logic of it works (been posting here while tests have been running )

Good chance that alpha will be ready tomorrow some time.

------

I've used the old distances.json as test data with distances rounded to 2dp.

4 out of 274 distance sets do not find the exact coordinates (but it does with 3dp - Except Alpha Cygni)
Such is life of SB3 I suppose.

Here is a good example of one of those that fail.
The tragedy is that I actually find two candidates (and one is actually the correct one) that both pass the Reverse Distance Check(RDC), but RMS is then used to pick between them, and picks the wrong one.
I don't know that it's possible to resolve that.

Test case

Spoiler

p0: LP 625-34
Distances

Basium 92.33 (né 92.326), coord: [-71.25,49.875,-30.21875]
Eranin 39.11 (né 39.108), coord: [-22.84375,36.53125,-1.1875]
Keries 22.27 (né 22.27), coord: [-18.90625,27.21875,12.59375]
LHS 411 13.65 (né 13.65), coord: [-13.125,23.59375,18.625]
SDSS J1416+1348 22.46 (né 22.458) coord: [-0.6875,27.15625,12]

The two candidates I find are:
A: [-9.46875, 17.8125, 30.4375] (correct) : rms = 0.0022....
B: [-9.46875, 17.75, 30.40625] (wrong) : rms = 0.0019...

And B is then picked as it has the lowest rms value, while A is in fact the correct value.

Let me know what the rest of you get/do.
Would love for this to be a bug in my code tbh..


---

Aside for those interested:
(at least) One not found is due to .net using "bankers rounding" (rounding to nearest even for midtpoint values (ie. x.5))
I must admit to being a bit shocked by that fact - I always assumed "round up" was used in midpoint cases.
Learned something new there...
And the reason it fails is because the distance in the distances.json is 41.765, which when rounding to 2dp gets rounded to 41.76 (and not as I would expect 41.77) - and that's enough to throw it off.

But as this is simply an artifact of rounding it twice (once ingame to 3dp - and then to 2dp when testing), I'm not too worried about that.

Just thought I'd share that little bit of info - It was definitely new to me (and I would have sworn .net didn't round like that...)

 

[Snuble]

Commanders. I´m away this week till friday. No ED access but some internet access. Can read it all but writing time is limited.
It seems the following is going.

1. A few commanders are checking the old SB2 list for validity and RW is updating TOR. Absolutely needed.
2. Have all tool commanders checked if their tool can calculate accurate (or good enough) 3D coords based on the now 2dp GM input.
3. Are commanders already taking new distances for the SB3 sausage?
4. The Great Collector. TornSouls. Are you still convinced that TGR is needed and are you able/willing to setup a alpha TGR so tool commanders can make the needed changes and test it?
5. It seems all tool commanders and other volunteers are still with us. Great work guys/galls!!!!!!

2. If someone would like to put my screenshot+html5 canvas+a couple of clicks solution online, they are welcome. On my internal test system it is working very well and I've stopped using trilateration from distances. The errors I get is mostly because I'm counting wrong (like using 1 instead of 11 as base), but these kind of mistakes are easy to find when comparing the in game distance compared to my caluculated distance. In most cases it is possible to clearly snap to a 1/32 grid, but as I've said earlier, I'm going for "good enough". For a better accuracy one could do several screenshots and my guess is that the average of 4-5 measurments from the screenshots are enough to snap to 1/32 with high confidense.
3. I guess we all are, but also mostly to test the tools. I thin the realisation that the data we gather right now is mostly meaningless come december made most of us more focused on the game rather then data mining.
4. Either a TGC or some 3rd party site that become the place where everyone go.

Other than that, it takes a very long time to go trough even a 20x20x20 box. Just finished getting prices and plotting/verifying stars in range for 17 systems. Each of them give 40 other systems that need to be checked against the database, mostly its a check-confirm-check-confirm process, but once a new system is found, galaxy map have to be opened, two screenshots taken, clicks need to happen and data itself generated.
Then its the gathering of market data. Even with a streamlined tool for entering data, it just takes time. And its the most time consuming part. Its getting very close to doing something similar to the coord screenshots. A semiautomated aproach where the user helps the system understanding where it should look for text, then apply a learned algorithm to mine data from the screenshot. Other deadlines first tough, worst time of the year getting drawn into hobby projects and games

 

[Biteketkergetek]

Test case

Spoiler

p0: LP 625-34
Distances

Basium 92.33 (né 92.326), coord: [-71.25,49.875,-30.21875]
Eranin 39.11 (né 39.108), coord: [-22.84375,36.53125,-1.1875]
Keries 22.27 (né 22.27), coord: [-18.90625,27.21875,12.59375]
LHS 411 13.65 (né 13.65), coord: [-13.125,23.59375,18.625]
SDSS J1416+1348 22.46 (né 22.458) coord: [-0.6875,27.15625,12]

The two candidates I find are:
A: [-9.46875, 17.8125, 30.4375] (correct) : rms = 0.0022....
B: [-9.46875, 17.75, 30.40625] (wrong) : rms = 0.0019...

And B is then picked as it has the lowest rms value, while A is in fact the correct value.

Let me know what the rest of you get/do.
Would love for this to be a bug in my code tbh..

These are my results:

[/d/gocode/src/bitbucket.org/tajtiattila/coords] (master) 2$ ./coords -d 2 -s 'LP 625-34'
# 551 systems total
# using 2 decimal digits and maximum error of 0.005000
? ‘LP 625-34’ has 3 candidate positions
  -9.46875,17.81250,30.43750 (json position): perfect match
  -9.46875,17.87500,30.46875: perfect match
  -9.46875,17.75000,30.40625: perfect match

# warnings:
? ‘LP 625-34’ has 3 candidate positions
  -9.46875,17.81250,30.43750 (json position): perfect match
  -9.46875,17.87500,30.46875: perfect match
  -9.46875,17.75000,30.40625: perfect match
# completed in 3.0871765s

This was the JSON I used. (Note, distances in the json are 3dp values, but they are rounded to 2dp values in my tool due to the -d 2 option).

Spoiler

[
	{"name":"LHS 411", "x":-13.125, "y":23.59375, "z":18.625, "contributor":"FD beta2"},
	{"name":"Eranin", "contributor": "FD beta1", "x":-22.84375, "y":36.53125, "z":-1.1875},
	{"name":"Keries", "contributor": "FD beta1", "x":-18.90625, "y":27.21875, "z":12.59375},
	{"name":"Basium", "x":-71.25, "y":49.875, "z":-30.21875, "contributor":"FD beta2"},
	{"name": "SDSS J1416+1348", "x": -0.6875, "y": 27.15625, "z": 12, },
	{
	  "name": "LP 625-34",
	  "contributor": "RedWizzard",
	  "calculated": true,
	  "x": -9.46875,
	  "y": 17.8125,
	  "z": 30.4375,
	  "distances": [
	    {"system": "LHS 411", "distance": 13.65},
	    {"system": "SDSS J1416+1348", "distance": 22.458},
	    {"system": "Keries", "distance": 22.270},
	    {"system": "Eranin", "distance": 39.108},
	    {"system": "Basium", "distance": 92.326}
	  ]
	}
]

I don't understand why a statistical method can be useful to decide which is the correct coordinate.

All of them positions are correct for the given set of distances. In other words, they need to result in the distance values shown on screen, every single one is correct for the data we have. The only thing we can do in such a case is to find a new reference system and distance that can decide which value is correct.

 

[TargetLost]

The math is well understood : http://en.wikipedia.org/wiki/Trilateration

It's fundamental for how GPS works

The nasty bit is when the real world don't play nice (ie. when distances are a bit "fuzzy")

Of course. The interesting part is to solve it for N unknown systems all at once with least square fit.

 

[TornSoul]

@Biteketkergetek
If I understand your output correctly - You get the same two candidates I do (I only listed two).
And both are "perfect matches" - ie. either satisfy the math.

I don't see you choosing between them, but simply listing you found them?
How do you intend to handle such cases?

 

[Biteketkergetek]

@Biteketkergetek
If I understand your output correctly - You get the same two candidates I do (I only listed two).
And both are "perfect matches" - ie. either satisfy the math.

I don't see you choosing between them, but simply listing you found them?
How do you intend to handle such cases?

I just report them as you see. I don't think they can be handled in any other way, we just need more distances. What could be done though, is to look through sets of additional systems (either nearby or on a predefined list), and ask for distances from them. This listing must take the 2dp precision into account, therefore the tool should ask for a distance from a system where this 2dp distance would be enough to make a decision. It may also happen that no such system known to us exist yet. This idea could be tested by generating distances from additional systems (since we already know the correct beta2 coordinate, which is -9.46875, 17.8125, 30.4375 according to all tools including mine, if 3dp precision input is used).

Btw I list 3 candidates, because in my case the maximum error of 0.005 is really an absolute value. Error in distance must be less or equal than this for me to accept a candidate. I consider this a feature, not a bug, because with this method it does not matter whether the typical floor(x+0.5), the highly unlighly ceil(x-0.5) or the measurement practice round 0.5 towards an even number rule is used for rounding. In the end if there are enough good distances, my approach will yield the same result, as it did for almost all systems from my systems.json, with only 2 warnings.

Spoiler

# 551 systems total
# using 2 decimal digits and maximum error of 0.005000
# showing coords in format: ±whole·frac (frac is a base32 digit)
...snip...
? ‘LP 625-34’ has 3 candidate positions
  -9·P,17·Y,30·N: perfect match
  -9·P,17·4,30·P: perfect match
  -9·P,17·2,30·O (json position): perfect match
? ‘LP 329-18’ has 2 candidate positions
  -26·U,39·4,23·J (json position): perfect match
  -26·V,39·2,23·K: perfect match
# completed in 1m1.1014948s

But rounding in practice, also using floor(x+0.5) will generally produce more values rounded up (since x.xx50 is rounded up, x.xx00 is kept as is, and an equal number of other values will be either rounded up or down).