Discussion What is the most efficient way to crowdsource the 3D system coordinates

[wolverine2710]

I finally finished my coordinate verifier tool (visit link to see how it works), and it also thinks that some coordinates are invalid:

LDS 1503
LHS 3297
WREDGUIA WH-Q B46-2

The tool thinks there is no valid coordinate for these systems, in other words, one or more of the distances must be incorrect. All other systems seems to be valid. I used this source.

So either the reported coordinates are incorrect (typo or rounding error), or there is something I didn't think of, and therefore not covered with a unit test in my tool.

AND: you just made it to list 1 ;-)

When I enter https://bitbucket.org/tajtiattila/coords/overview in the browser I get the message: You do not have access to this repository.

Though I am able to access other repositories of you.
https://bitbucket.org/tajtiattila/harmless
https://bitbucket.org/tajtiattila/json-sqlite

 

[TornSoul]

Ok I think I've found out what the problem with EDSC API is (yet to find how to solve it)


It's basically the browser and server being good boys and protecting me from cross-site attacks.

So the reason why injecting the request code in a browser that has EDSC open works, is ofc that it's on the correct domain already.
Which is also why all the other site remote calls works (for populating auto drop downs etc).

But - Being on a different domain (like hurl.it or your own) it's getting blocked.

I'm fairly sure that's the reason anyhow.

Not exactly sure what knobs I need to tweak yet to make it work.

Just posting this here - so the rest of you don't waste time on it - As it seems it's something I need to fix on my (web)server.

I'll let you know when it works.

 

[Biteketkergetek]

AND: you just made it to list 1 ;-)

Thanks! I knew it's worth the effort

When I enter https://bitbucket.org/tajtiattila/coords/overview in the browser I get the message: You do not have access to this repository.

Though I am able to access other repositories of you.
https://bitbucket.org/tajtiattila/harmless
https://bitbucket.org/tajtiattila/json-sqlite

Oh sorry, I haven't realized bitbucket makes repos private by default. I switched it off now.

 

[wolverine2710]

Oh sorry, I haven't realized bitbucket makes repos private by default. I switched it off now.

I'm in ;-)

 

[Biteketkergetek]

I'm in ;-)

I wonder if you can grasp any of it. I just added some comments so I can later, too

 

[Athan]

Ok I think I've found out what the problem with EDSC API is (yet to find how to solve it)


It's basically the browser and server being good boys and protecting me from cross-site attacks.

So the reason why injecting the request code in a browser that has EDSC open works, is ofc that it's on the correct domain already.
Which is also why all the other site remote calls works (for populating auto drop downs etc).

This is the 'same origin policy' stuff in browsers, yes? It'll be a red herring for any non-browser script using the API.

But - Being on a different domain (like hurl.it or your own) it's getting blocked.

Checking hurl.it, it's not even sending things in the right format. You're expecting a json-encoded data array, it's just trying to send normal POST parameters.

I'm fairly sure that's the reason anyhow.

Not exactly sure what knobs I need to tweak yet to make it work.

Just posting this here - so the rest of you don't waste time on it - As it seems it's something I need to fix on my (web)server.

I'll let you know when it works.

FWIW my perl script is now working:

Spoiler

POST http://edstarcoordinator.com/api.asmx/GetSystems
Content-Type: application/json; charset=utf-8

{"data":{"cr":5,"date":"2014-09-01 00:00:00","ver":1}}

$VAR1 = {
          'd' => {
                   'systems' => [
                                  {
                                    'cr' => 5,
                                    'date' => '2014-10-18 03:03:01',
                                    'coord' => [
                                                 '-61.1875',
                                                 '23.59375',
                                                 '-42'
                                               ],
                                    'name' => 'LTT 10482'
                                  },
                                  {
                                    'cr' => 5,
                                    'date' => '2014-10-18 03:03:01',
                                    'coord' => [
                                                 '-75.75',
                                                 '25.40625',
                                                 '-23.78125'
                                               ],
                                    'name' => 'Moscab Kutja'
                                  },
... etc

 

[Biteketkergetek]

Checking hurl.it, it's not even sending things in the right format. You're expecting a json-encoded data array, it's just trying to send normal POST parameters.

That is maybe because everyone expects to send the request in URL encoded format instead of json, when it is documented like having POST arguments. Perhaps an example request would make it clear. I glanced over the example, but it is JavaScript I'm not very familiar with, so I haven't realized the content-type and dataType arguments are for the request itself, not for the response.

At least it explains why the API uses POST instead of GET.

 

[Athan]

Well, TornSoul's example does have:

var data = {data: {ver:1, cr:5, date:"2014-09-18 12:34:56"}};
...
contentType: 'application/json; charset=utf-8',
...
data: JSON.stringify(data),
...

which is what clued me in.

But, yeah, as the API currently stands it could as easily just be straight up POST parameters, i.e. there's no real structure to the query, just one compulsory value and two optional ones.

I can vaguely think of some possible expansions to the API that would be easier with the json method, and it's easy enough to use once you know that's how it works.

 

[JesusFreke]

I finally finished my coordinate verifier tool (visit link to see how it works), and it also thinks that some coordinates are invalid:

LDS 1503
LHS 3297
WREDGUIA WH-Q B46-2

The tool thinks there is no valid coordinate for these systems, in other words, one or more of the distances must be incorrect. All other systems seems to be valid. I used this source.

So either the reported coordinates are incorrect (typo or rounding error), or there is something I didn't think of, and therefore not covered with a unit test in my tool.

I haven't looked at the code, but are you using single-precision floats when performing the square root, as Athan had mentioned?

 

[JesusFreke]

I just thought I'd throw another idea out there, which might be helpful especially for stars that are far away, where we can't get a good set of reference distances to.

I took several screenshots in the galaxy map of a given star, and then using gimp's distance tool, I measured the number of pixels to the nearest grid lines, and was able to calculate the coordinate fairly accurately, accurate enough to get within 1/32 LY at least. The hardest is the vertical measurement, but if you position the camera so that it's exactly aligned with the plane that the grid forms, you can get a screenshot of the star with the above the lower plane and below the upper plane.

 

[RedWizzard]

I finally finished my coordinate verifier tool (visit link to see how it works), and it also thinks that some coordinates are invalid:

LDS 1503
LHS 3297
WREDGUIA WH-Q B46-2

The tool thinks there is no valid coordinate for these systems, in other words, one or more of the distances must be incorrect. All other systems seems to be valid. I used this source.

So either the reported coordinates are incorrect (typo or rounding error), or there is something I didn't think of, and therefore not covered with a unit test in my tool.

LHS 3297 and Wredguia WH-Q b46-2 both have distances affected by this single/double precision discrepancy, so that could be part of the problem. But LDS 1503 does not and it has 16 distances that match so I really don't think there is any way the calculated coordinates could be wrong.

It looks like you're using my stuff so take a look at bulklocate.html. Check the "include known calculated systems" box at the top and you'll be able to find those three systems in the table (click the column headers to sort). Alternatively you can enter the distances in entry.html to see how my algorithm reacts to the data.

 

[RedWizzard]

I just thought I'd throw another idea out there, which might be helpful especially for stars that are far away, where we can't get a good set of reference distances to.

I took several screenshots in the galaxy map of a given star, and then using gimp's distance tool, I measured the number of pixels to the nearest grid lines, and was able to calculate the coordinate fairly accurately, accurate enough to get within 1/32 LY at least. The hardest is the vertical measurement, but if you position the camera so that it's exactly aligned with the plane that the grid forms, you can get a screenshot of the star with the above the lower plane and below the upper plane.

Can you try that with Sag A* and see what you get?

 

[RedWizzard]

It's more of an explanation for why the game displays some wrong figures for distances. Everyone's work here has avoided the 'error' FD have made. But, yes, in theory people could make use of this knowledge to ensure we always use the same figures that FD do.

Edit: And on a quick check, you couldn't, for instance, force perl to use only single-precision floating point to get answers matching the in-game ones. You're stuck with double-precision. I suspect java, javascript, c#, php, python etc are going to face the same issue, being higher level languages.

Edit2: Actually looks like you can force float32 usage in python at least.

Well done! I've confirmed that using single precision floats resolves the discrepancy for all 9 cases I know of and doesn't cause any new ones. So all 2200+ distances in systems.json match except for a couple involving Alpha Cygni (need more data on that out-of-Pill star).

In terms of high level languages, C# and Java both have single and double precision floating point. Javascript has Math.fround() which rounds a double precision float to the single precision equivalent. Wrapping Math.sqrt() with Math.fround() is all that is required here.

 

[RedWizzard]

Ok I think I've found out what the problem with EDSC API is (yet to find how to solve it)

It's basically the browser and server being good boys and protecting me from cross-site attacks.
...
Not exactly sure what knobs I need to tweak yet to make it work.

Just posting this here - so the rest of you don't waste time on it - As it seems it's something I need to fix on my (web)server.

I'll let you know when it works.

Finally got a chance to look at your API stuff - also ran into the cross-domain issue.

Easiest fix might be to use JSONP. Alternatively you could setup CORS. That's probably the correct way to allow cross domain requests these days.

IMHO these API calls should really be GETs. We're not modifying anything on the server and GET has the advantage of allowing caching.

 

[TornSoul]

FWIW my perl script is now working:

Spoiler

POST http://edstarcoordinator.com/api.asmx/GetSystems
Content-Type: application/json; charset=utf-8

{"data":{"cr":5,"date":"2014-09-01 00:00:00","ver":1}}

$VAR1 = {
          'd' => {
                   'systems' => [
                                  {
                                    'cr' => 5,
                                    'date' => '2014-10-18 03:03:01',
                                    'coord' => [
                                                 '-61.1875',
                                                 '23.59375',
                                                 '-42'
                                               ],
                                    'name' => 'LTT 10482'
                                  },
                                  {
                                    'cr' => 5,
                                    'date' => '2014-10-18 03:03:01',
                                    'coord' => [
                                                 '-75.75',
                                                 '25.40625',
                                                 '-23.78125'
                                               ],
                                    'name' => 'Moscab Kutja'
                                  },
... etc

Thank you very much for testing - So that it know it actually works now.

This is the 'same origin policy' stuff in browsers, yes?

Correct.
Not entirely sure how much I've messed up my site/the server trying to get it to work... (from a security point of view)

Checking hurl.it, it's not even sending things in the right format. You're expecting a json-encoded data array, it's just trying to send normal POST parameters.

Thank you for that crucial piece of info - I can stop pulling out my hairs to try and get it to work with hurl.it

That is maybe because everyone expects to send the request in URL encoded format instead of json, when it is documented like having POST arguments. Perhaps an example request would make it clear. I glanced over the example, but it is JavaScript I'm not very familiar with, so I haven't realized the content-type and dataType arguments are for the request itself, not for the response.

At least it explains why the API uses POST instead of GET.

I've added some explaining words at the very top of the api doc site.
Hopefully that will stop people trying with hurl.it etc.

I've also added working JSFiddle links to all methods.
You can basically use those to grab all the data.
Systems fiddle
Distances fiddle

But, yeah, as the API currently stands it could as easily just be straight up POST parameters, i.e. there's no real structure to the query, just one compulsory value and two optional ones.

I can vaguely think of some possible expansions to the API that would be easier with the json method, and it's easy enough to use once you know that's how it works.

Heh yeah - atm it might as well be straight POST.
It's a bit of "preparing for the future" - Just in case.

If I had gone with POST, it would be a huge deal (for consumers) to have to change their code, should I then later decide to "upgrade".

Thanks for the feedback guys.

 

[TornSoul]

Javascript has Math.fround() which rounds a double precision float to the single precision equivalent. Wrapping Math.sqrt() with Math.fround() is all that is required here.

Awesome! Didn't think of that.
Thanks for the pointer.

I'll play around

 

[TornSoul]

Alternatively you could setup CORS. That's probably the correct way to allow cross domain requests these days.

Yup that's what I ended up doing.

IMHO these API calls should really be GETs. We're not modifying anything on the server and GET has the advantage of allowing caching.

Fair point actually.
I just have this somewhere in the back of my mind, that "GET can be dangerous and thus POST should be used if possible"

Nothing I at the top of my head can back up - and might even be out dated info that has somehow got lodged in there...

 

[RedWizzard]

Yup that's what I ended up doing.

Working for me now, thanks.

I just have this somewhere in the back of my mind, that "GET can be dangerous and thus POST should be used if possible"

Nothing I at the top of my head can back up - and might even be out dated info that has somehow got lodged in there...

GET is not encrypted so it's not good for sensitive stuff, but I don't think we're worried about that. I guess you have to be a little careful about parameter validation but that's true of POST too.

TBH I'd set something up myself except I don't really want to have to secure a machine. It's not trivial... So thanks for getting something online.

 

[JesusFreke]

Working for me now, thanks.



GET is not encrypted so it's not good for sensitive stuff, but I don't think we're worried about that. I guess you have to be a little careful about parameter validation but that's true of POST too.

TBH I'd set something up myself except I don't really want to have to secure a machine. It's not trivial... So thanks for getting something online.

POST is only encrypted if you're doing HTTPS (or custom encryption, I guess). And if you're using HTTPS, then the url and the GET params are encrypted as well.

 

[JesusFreke]

Can you try that with Sag A* and see what you get?

I got 25.21875, -20.90625, 25899.96875 for Sagittarius A*, using the screenshot + measure method.

My calculated coordinates were 0.00065, 0.0091 and 0.0015 away from the nearest 1/32 grid, respectively, so 2%, 29% and 5% error.

Edit: oops, previously posted coordinates weren't correct. I went the wrong way with the vertical (middle) one. These still don't match the distances that RedWizzard has though.