Discussion What is the most efficient way to crowdsource the 3D system coordinates

[JesusFreke]

I have been through your new data and updated what I can on my TradeDangerous fork. There were a few I couldn't reconcile:

'LP 274-8',-29.71875,45,20.28125,'JesusFreke2','2014-10-18 00:00:00'
'LP 274-8',-29.75,44.9375,20.3125,'Harbinger','2014-10-15 00:00:00'

'Wredguia AT-O B47-0',-103.125,41.9375,-43.25,'JesusFreke2','2014-10-18 00:00:00'
'WREDGUIA AT-O B47-0',-103.125,41.96875,-43.21875,'Combined','2014-10-13 23:00:00'

'Wredguia PC-D D12-74',-116.46875,35.96875,-19.3125,'JesusFreke2','2014-10-18 00:00:00'
'WREDGUIA PC-D D12-74',-116.5,35.96875,-19.28125,'Combined','2014-10-13 23:00:00'

'Wredguia XH-Q B46-4',-109.90625,30.6875,-58.65625,'JesusFreke2','2014-10-18 00:00:00'
'WREDGUIA XH-Q B46-4',-109.9375,30.6875,-58.65625,'Combined','2014-10-13 23:00:00'

'WREDGUIA YD-I C23-11',-128.53125,15.75,-26.65625,'Combined','2014-10-13 23:00:00'
'Wredguia YD-I C23-11',-128.53125,15.84375,-26.46875,'JesusFreke2','2014-10-18 00:00:00'

'WREDGUIA YD-I C23-12',-109.96875,46.75,-45.59375,'Combined','2014-10-13 23:00:00'
'Wredguia YD-I C23-12',-110.03125,46.71875,-45.625,'JesusFreke2','2014-10-18 00:00:00'

'Wredguia ZS-O B47-1',-111.53125,40.3125,-25.21875,'JesusFreke2','2014-10-18 00:00:00'
'WREDGUIA ZS-O B47-1',-111.53125,40.375,-25.1875,'Combined','2014-10-13 23:00:00'

It was one of my first 25 stars so used an averaging method and was converted to 1/32 after the fact so yours is most likely the correct result.

I really need to double check on the coordinates from my first 25 stars.

Yeah, I went back and noticed that as well . I'll get you yet!! Ha! Just kidding . It's great we both have different, but highly accurate methods. It makes for great cross-checking of results.

If anything, I'm still a bit suspicious of my algorithm that tries to map out the candidate region. It "seems" to work... With an emphasis on the "seems"

 

[RedWizzard]

p1-p4 all *exactly* in the same plane = big trouble - agreed.

The critical line (and takeaway) is my last line

"Wether p1-p4 - or indeed p0-p4 all are very close to being in the same plane or not - has no impact on the final calculations."
(emphasis added)
That was the main thrust of my argument.
In all my 1M runs of finding "best fitting plane" it never happened that p1-p4 (or p0-p4) happened to be in the same plane - There was however *many* where it was a very close call (if i recall the distance to the best fitting plane could be out on the 2nd decimal).

If p4 is sufficiently close to the plane of p1..p3 then the precision of the distances could get you into trouble. I did come across a case using 2dp distances where trilateration (with just four references) was picking the wrong candidate. I didn't investigate it closely but I expect either p0 or p4 was close to the plane in that case. IIRC Eranin was involved so if you search the thread for that you could probably find it if you're interested.

On the first two point we do exactly the same.

There is a subtle difference. You're picking 4 references to use for trilateration which means every possible p4 must accurately choose the correct candidate. Whereas I'm not picking a specific p4 at all, I'm just using the entire set of distances to choose the candidate. The difference might be splitting hairs for 5 references but with more references your algorithm will discard any set of references where there exists a subset of 4 that lie in a plane. The more references you have the more likely that is, and I've got some stars with around 20 recorded distances.

 

[Chromatix]

To make testing easier (using TD) I think it would be great to have a convertor which spits out a systems.csv file which can be used to replace the system.csv file as long as NOT everything is stable and NOT all coords are found.

My Python scripts do that.

ETA: I'm sorry to say that "Mirfak" is my fault. I was using the commonly accepted spelling instead of the one in game.

 

[JesusFreke]

I actually have 3 recorded distances that are just *slightly* off with the calculated coordinates. I haven't double-checked the recorded distances yet, but they are extremely close - close enough that it's feasible that it might be a rounding error somewhere, likely due to floating point calculations. I'm planning on checking them out again after I've saved up to upgrade the jump drive on my asp. Going back to a ~12LY jump distance after buying the asp is somewhat painful, and I want to upgrade straight to the "D" drive. (ugh! talk about expensive).

In any case, the 3 suspects are:

LFT 668 <-> Loga. My recorded distance is 64.243, but the closest coord I can find gives a distance of 64.2435019209, just enough that it should round up to 64.244 instead.

HIP 91906 <-> Wredguia UR-Q B46-2. My recorded distance is 19.170, but actual distance using best matching coord is 19.1694999038

Keries <-> Wredguia WH-Q B46-2 131.337 vs 131.336496796

Ok, I went back and double-checked all the recorded distances I have for these systems, and they all checked out. Soooo, this likely means that I'm calculating the distances slightly differently than the game does. I suspect there is some rounding happening on an intermediate result at some point, although I'm not sure where/what.

So, let's take one of the systems, LFT 668. My best coordinate for this system is -19.03125, 25.656250, -24.21875.

The distance from there to loga (-79.78125, 36.53125, -42.0625) is 64.24350192, which should round up to 64.244, but the game shows the distance as 64.243.

The only way I could find to make the math work out is if the distance is rounded to the nearest 1/1024th, before rounding for display. In this case, 64.24350192 would be rounded to 64.24316406, which would then be rounded to 64.243 for display.

I'll need to do some more analysis to determine if the same method of calculating the distance works for the other 2 suspicious distances, and make sure that it doesn't "break" any other stars that have already been calculated.

 

[JesusFreke]

I'll need to do some more analysis to determine if the same method of calculating the distance works for the other 2 suspicious distances, and make sure that it doesn't "break" any other stars that have already been calculated.

I've confirmed that this method of distance calculation also works for the other 2 suspicious distances. Now to implement it in my program and see if any other systems break.

 

[RedWizzard]

Ok, I went back and double-checked all the recorded distances I have for these systems, and they all checked out. Soooo, this likely means that I'm calculating the distances slightly differently than the game does. I suspect there is some rounding happening on an intermediate result at some point, although I'm not sure where/what.

So, let's take one of the systems, LFT 668. My best coordinate for this system is -19.03125, 25.656250, -24.21875.

The distance from there to loga (-79.78125, 36.53125, -42.0625) is 64.24350192, which should round up to 64.244, but the game shows the distance as 64.243.

The only way I could find to make the math work out is if the distance is rounded to the nearest 1/1024th, before rounding for display. In this case, 64.24350192 would be rounded to 64.24316406, which would then be rounded to 64.243 for display.

My theory is that ED is using integer math internally with coordinates having 1/32 Ly units (i.e. effectively fixed point arithmetic). Your 1/1024 could be an artifact of that since 32 * 32 = 1024.

FE2 and FFE used a lot of fixed point arithmetic (and a custom floating format).

 

[JesusFreke]

My theory is that ED is using integer math internally with coordinates having 1/32 Ly units (i.e. effectively fixed point arithmetic). Your 1/1024 could be an artifact of that since 32 * 32 = 1024.

FE2 and FFE used a lot of fixed point arithmetic (and a custom floating format).

Yeah, that's exactly how I came up with 1024 - assuming integer math on the coords, and then a square root that is also represented as 1/32 fixed point of 1/32 fixed point data.., so 1/1024 fixed point in total.

Unfortunately, it doesn't actually work with all the other distances we have. So, back to the drawing board. Except that my drawing board is fresh out of ideas at this point

I tried various levels of decimal rounding of the data at various points, but I couldn't find anything else that works.


Edit: Upon further investigation, I might have been "Doing It Wrong". It's looking like the 1/1024 thing may be the ticket.

 

[RedWizzard]

Yeah, that's exactly how I came up with 1024 - assuming integer math on the coords, and then a square root that is also represented as 1/32 fixed point of 1/32 fixed point data.., so 1/1024 fixed point in total.

Unfortunately, it doesn't actually work with all the other distances we have. So, back to the drawing board. Except that my drawing board is fresh out of ideas at this point

I tried various levels of decimal rounding of the data at various points, but I couldn't find anything else that works.

Bug in their hand-rolled integer square root function?

I found one more distance with the same issue in my data:
LHS 3297 <-> Haras. Shows as 82.840 in ED, but I calculate 82.841
based on LHS 3297 at (-36.46875, 22.6875, -16.53125).

I wonder if we could get the real coordinates out of Michael for a few cases.

 

[JesusFreke]

Yeah, that's exactly how I came up with 1024 - assuming integer math on the coords, and then a square root that is also represented as 1/32 fixed point of 1/32 fixed point data.., so 1/1024 fixed point in total.

Unfortunately, it doesn't actually work with all the other distances we have. So, back to the drawing board. Except that my drawing board is fresh out of ideas at this point

I tried various levels of decimal rounding of the data at various points, but I couldn't find anything else that works.


Edit: Upon further investigation, I might have been "Doing It Wrong". It's looking like the 1/1024 thing may be the ticket.

Ok, yeah. This seems to work well for the data I currently have, although I don't have the lastest stuff from RedWizzard's database. But this is working for those 3 I mentioned, and for everything else I have.

So, the calculation for the final distance check should be:

round(round(calculated_distance * 1024)/1024, 3) == displayed_distance

i.e. round to the nearest 1/1024, and then round the result to 3 decimal places.

EDIT: This new calculation still doesn't seem to work for all cases.

 

[JesusFreke]

I found one more distance with the same issue in my data:
LHS 3297 <-> Haras. Shows as 82.840 in ED, but I calculate 82.841
based on LHS 3297 at (-36.46875, 22.6875, -16.53125).

Hmm, nope. The new distance calculation doesn't work for this :/

 

[JesusFreke]

Bug in their hand-rolled integer square root function?

I found one more distance with the same issue in my data:
LHS 3297 <-> Haras. Shows as 82.840 in ED, but I calculate 82.841
based on LHS 3297 at (-36.46875, 22.6875, -16.53125).

I wonder if we could get the real coordinates out of Michael for a few cases.

I just confirmed that the displayed distance in-game is 82.840

 

[RedWizzard]

So, the calculation for the final distance check should be:

round(round(calculated_distance * 1024)/1024, 3) == displayed_distance

i.e. round to the nearest 1/1024, and then round the result to 3 decimal places.

EDIT: This new calculation still doesn't seem to work for all cases.

Yeah, I'm finding that too. The cases I'm testing are:
1. Taran (-42.5, 45.46875, -3) to Ross 1051 (-37.21875, 44.5, -5.0625), ED shows 5.752
2. Taran (-42.5, 45.46875, -3) to Hagalaz (-50.59375 45.15625 11.09375), ED shows 16.255
3. LFT 668 (-19.03125, 25.65625, -24.21875) to Loga (-79.78125, 36.53125, -42.0625), ED shows 64.243
4. LHS 3297 (-36.46875, 22.6875, -16.53125) to Haras (-118.75, 14.40625, -21.40625), ED shows 82.840

I've tried three variations:
a. round(floor(calculated_distance * 1024)/1024, 3)
b. round(round(calculated_distance * 1024)/1024, 3)
c. round(calculated_distance, 3)

Version "a" works for cases 2, 3, and 4.
Version "b" works for 1 and 3.
Version "c" works for 1 and 2.

Very unsatisfactory

 

[graham.reeds]

Why 1024?

If star coordinates are in 1/32ly surely you should be clamping to 1/32?

 

[JesusFreke]

Why 1024?

If star coordinates are in 1/32ly surely you should be clamping to 1/32?

The coordinates are, yes. The distances aren't. For example, you see distances like 109.213, which isn't a multiple of 1/32, even after rounding.

 

[wolverine2710]

Ticketed WREDGUIA LW-E D11-129 issue

I can't get there either. Long hyperspace sequence then disconnect from server. I'll ticket it.

I also tried to enter WREDGUIA LW-E D11-129. I get very long hyper space sequence. At the end it results in a "could not connect to server error". I've created a ticket for it as well (id: 36268) and copy/pasted the experience from you and Harbinger in the ticket.

 

[Snuble]

SUGGESTION:

Could OP please update first post with links to the various sites/tools that are sharing BETA2 system coordinates?

 

[Biteketkergetek]

The exact distances squared must be a multiple of 1/1024 Ly, because the individual coordinates are a multiple of 1/32 Ly apart.

So it might be better to do the rounding on the squared distance only, if I am not mistaken.

The non-squared distance can be an irrational number even for integer input, for two systems at (0,0,0) and (1,1,1) this will be sqrt(3), which seems pointless to do any rounding on.

 

[wolverine2710]

Bug in their hand-rolled integer square root function?

I found one more distance with the same issue in my data:
LHS 3297 <-> Haras. Shows as 82.840 in ED, but I calculate 82.841
based on LHS 3297 at (-36.46875, 22.6875, -16.53125).

I wonder if we could get the real coordinates out of Michael for a few cases.

You can ask Michael but I don't think its feasible/easy for him. The systems which are not on MB's list are procedurally generated.

I've asked Michael last week two questions- have been given permission to share it.

Q1: Can we get a list of names of all 570 systems - without 3D coordinates.
A: MB: Not really as they're generated. The names I have easy access to I've already provided.

Q2: Can after exiting a HyperSpace jump a file be created with just name and 3D-coordinates. Something I described earlier in this thread.
A: MB: We're busy with the game features so it's not something we can provide at the moment.

Your original question: So it looks as if unless a Fdev is doing some extra work (program) for it Michael can't help us. Still feel free to ask Michael. I'm reluctant atm to ask because I don't want to end up on MB's or FD's "wanted: Dead or alive" list :smilie:

Edit: Going in "outlaw mode". Gonna send him a PM today to check if the number of 570 mentioned in the newsletter is the EXACT number of systems for SB2 or that its is perhaps "rounded" to look good in the newsletter. Would be a waste of time searching for the last 2 needles in the GM haystack if its NOT exactly 570!!

 

[wolverine2710]

My theory is that ED is using integer math internally with coordinates having 1/32 Ly units (i.e. effectively fixed point arithmetic). Your 1/1024 could be an artifact of that since 32 * 32 = 1024.

FE2 and FFE used a lot of fixed point arithmetic (and a custom floating format).

It looks viable to ask Michael Brookes this question as it can probably be answered directly by asking a Fdev. Any one feeling frisky? If not if someone can phrase a concise/clear/unambiguous question for MB I can send him a PM.

 

[wolverine2710]

SUGGESTION:

Could OP please update first post with links to the various sites/tools that are sharing BETA2 system coordinates?

Excellent idea. As a matter of fact it was on my TODO list for today. Will try to figure out what tools (which can be downloaded and used) are out there today. So far I have on my radar : Harbinger, RedWizzard, TunaMage, EDSC. Could be I missed one (or two).

Question1: Would it be possible for the commanders to give me urls (direct download or git repositories) so I'm really using the latest versions of their tools?

Question2: Would it be possible for commanders/authors to send me the urls of their tools which use "our" coords? I certainly do NOT have a complete list of that.