Discussion What is the most efficient way to crowdsource the 3D system coordinates

Compiled 'C' using gcc 4.7.2 and glibc 2.13. As I said above, it comes down to hitting 'SQRT' versus 'FSQRT' down at the assembly/CPU level. The latter has much lower accuracy on answers, but appears to be what FD are using in-game. As such there are some edge cases where results go either side of a 'half' point and can affect rounded answers.

Yeah, we had figured that out back in beta 2. But Javascript doesn't have single precision sqrt so I have to use double and convert. If I do it right the answer is the same.
 
Last edited:
TornSoul, note I am now updating my local DB from EDSC once an hour. Given that in the case of no change I'm only querying for anything update at or since the time of the last update I saw I'm thinking I could increase that frequency without causing you issues, yes ?

Yeah - Currently, even doing it once per minute (or even more) should be a non-issue.

Just... Think about the future. So don't get too reliant on a hyper frequent update cycle.

To anyone that ask in general, I tell them to see if their app won't be just fine with an update cycle of 1 hour (most will) - Just so that it's reasonably future proof.
 
Yeah - Currently, even doing it once per minute (or even more) should be a non-issue.

Just... Think about the future. So don't get too reliant on a hyper frequent update cycle.

To anyone that ask in general, I tell them to see if their app won't be just fine with an update cycle of 1 hour (most will) - Just so that it's reasonably future proof.

I've just changed it to 7,22,37,52 minutes past the hour. This came up last night as I quickly got new users including one pointing out I didn't yet have a star he'd just added. Not helped by me not having updated the update script for some other code changes, but still people are going to expect better response than once an hour after adding something, and perhaps most importantly having a good user experience from adding new stars and very soon after having them available in tools will encourage people to keep on adding systems data.

I think this means we bring up some 'live' network of TGC data again, i.e. as TGC gets a new/better position for a star system it broadcasts it out to all listeners. That whole EDMN thing, but for TGC. I'd almost certainly be able to host a 'supernode' for that, i.e. TGC only has to feed one or two hosts and they take care of propogating it out.
 
Some good comments/observations there.

- GetSystems uses POST instead of GET making the API unnecessarily hard to use. It could be implemented using GET, therefore it should just use GET. I can imagine complex queries that would require a structured object, but those should be done likely within a local mirror of TGC anyway.
We've had a long discussion about this already - and most agree with you.
I however, for various reasons, have a preference for POST with everything.
(maybe try a search for POST in this thread for the whole debate and arguments con/pro if you are really interested)


- Both the query and response use semi-undocumented wrappers ("data" and "d", respectively) around the actually documented parameters. I say semi-documented because this can be found out if one carefully looks at the examples.
I just use reflection so at least for the query part I wasted some time figuring it out. I could have saved some time if at least the query and response examples JSON objects mentioned them,
"data" and "d" are standard "wrappers" - Nothing I can do about those. You'll get those no matter what (JSON) API you query.

I'm actually only returning (C# example)
Code:
return new {
          ver = version.currentMinor,
          date = DateTime.UtcNow.ToString(EDSCString.dateFormat),
          status = new { input = InputStatusMessages }
        };

So they are not part of the API - But part of working with the XMLHttpRequest object, hence why it's not mentioned in the API.

or if the API returned a meaningful error message instead of just
Code:
{"Message":"There was an error processing the request.","StackTrace":"","ExceptionType":""}
in this case.
Yes that would be VERY nice.
You'll however have to blame .NET for that one. It's out of my control.
You can often (not always) get more info if you dig into the returned error object - Sometimes it helps, sometimes not.

- "ver" accepts an integer only but returns a floating point value. Perhaps it should be standardized as an integer for symmetry in both places, and use an additional version string in responses if necessary.
Correct - And this is by deliberate design.
Keep in mind that all minor versions are backward compatible with the major version.

Input: Integer; Only the major version. Don't need the minor version. And it would be annoying to have to change this although your program would work fine either way.
Output: Float (really decimal but nvm) - I considered it a service to return the full version number - So your program could alert you in case of a (minor) version change, if you felt like it.
Ie. prompting you to investigate if there was new features you could make use of - Instead of having to check the api pages every day/week/month.

- From the API docs it is not clear which timezone the time stamps (query/response) should be in. Since UTC is mentioned on the page, one can only hope it is indeed UTC.

- Time format should perhaps use RFC 3339. I think it would be the most suitable for the wire format. Basically "T" instead of space and "Z" at the end for UTC.
Fair point, I should perhaps mention that times are UTC somewhere.
Call it programmer bias; in that I would always assume UTC to be default *unless* something else was mentioned (in a global app like this)

Straight "2014-10-11 12:13:14" is easier on the eyes than "2014-10-11T12:13:14Z" - So that's what I went with.
And I'm not aware of any language not able to parse either.
 
Last edited:
people are going to expect better response than once an hour after adding something
In that particular instance, one could argue you should have updated your own backend with the response you got from TGC on submitting - and the system would have been available instantly.

Your update cycle would have to account for systems you'd already updated your backend with though in that case.
And yeah that would be a minor, but still annoying, thing to deal with.

I guess the point I'm trying to make is - It would be doable in that particular scenario, for the user to see his own system right away regardless of what update frequency you use.


I think this means we bring up some 'live' network of TGC data again, i.e. as TGC gets a new/better position for a star system it broadcasts it out to all listeners. That whole EDMN thing, but for TGC. I'd almost certainly be able to host a 'supernode' for that, i.e. TGC only has to feed one or two hosts and they take care of propogating it out.
I'm definitely not adverse to doing this - It is however very long down on the TODO list atm.
 
In that particular instance, one could argue you should have updated your own backend with the response you got from TGC on submitting - and the system would have been available instantly.

Your update cycle would have to account for systems you'd already updated your backend with though in that case.
And yeah that would be a minor, but still annoying, thing to deal with.

I guess the point I'm trying to make is - It would be doable in that particular scenario, for the user to see his own system right away regardless of what update frequency you use.
Except in my case the tool is only a consumer of the data and I point people at EDSC to add missing systems, and thus have no way to know they've added such.
 
"data" and "d" are standard "wrappers" - Nothing I can do about those. You'll get those no matter what (JSON) API you query.

I'm actually only returning (C# example)
Code:
return new {
          ver = version.currentMinor,
          date = DateTime.UtcNow.ToString(EDSCString.dateFormat),
          status = new { input = InputStatusMessages }
        };

So they are not part of the API - But part of working with the XMLHttpRequest object, hence why it's not mentioned in the API.

I think that might be a .net thing. Certainly it's possible to return json without that wrapper in (e.g.) node.js. Anyway, it's obvious from the examples how it works so I don't think it's a problem.
 
Edit: Oh, it's not the square root; it's the addition. I have to apply Math.fround to the results of additions as well as multiplication and the sqrt
itself:
Code:
Math.fround(
  Math.sqrt(
    Math.fround(63.8125*63.8125) +
    Math.fround(-120.46875*-120.46875)+
    Math.fround(42.59375*42.59375)
  )
)
142.82501220703125


Math.fround(
  Math.sqrt(
    Math.fround(
      Math.fround(
        Math.fround(63.8125*63.8125)+
        Math.fround(-120.46875*-120.46875)
      ) +
      Math.fround(42.59375*42.59375)
    )
  )
)
142.8249969482422


Grumble... No matter how crazy I get in C# I always get the same result...
(and yes several of these are equivalent - But desperate times...)

Code:
            decimal a = 63.8125M;
            decimal b = -120.46875M;
            decimal c = 42.59375M;
            
            float A = (float)a;
            float B = (float)b;
            float C = (float)c;
            
            
            //Your code goes here
            Console.WriteLine(Math.Sqrt((double)(a*a+b*b+c*c)));
            Console.WriteLine(Math.Sqrt((float)(a*a+b*b+c*c)));
            Console.WriteLine(Math.Sqrt((float)((float)(a*a)+(float)(b*b)+(float)(c*c))));
            Console.WriteLine(Math.Sqrt((float)((float)((float)a*(float)a)+(float)((float)b*(float)b)+(float)((float)c*(float)c))));
            
            Console.WriteLine(Math.Sqrt(A*A+B*B+C*C));
            Console.WriteLine(Math.Sqrt((double)(A*A+B*B+C*C)));


142.825006290478
142.825006290478
142.825006290478
142.825006290478
142.825006290478
142.825006290478

Open to suggestions...
 
Last edited:
I think that might be a .net thing. Certainly it's possible to return json without that wrapper in (e.g.) node.js. Anyway, it's obvious from the examples how it works so I don't think it's a problem.

Urg yes - You are correct.
If I recall - It's some sort of protection mechanism. Don't recall the specifics ATTOMH
 
Grumble... No matter how crazy I get in C# I always get the same result...
(and yes several of these are equivalent - But desperate times...)

...
Open to suggestions...

Edit: Ok, I got it to work by defining an fround function:
Code:
static double fround(double x)
{
      float f = (float)x;
      return (double)f;
}

double a = 63.8125;
double b = (-120.46875);
double c = 42.59375;
Console.WriteLine(fround(Math.Sqrt(fround(fround(fround(a*a)+fround(b*b))+fround(c*c)))));

142.824996948242

I'm not quite sure why this produces a different result than just using floats. I suspect there are some implicit casts going on that are screwing it up somewhere in that case.
 
Last edited:
Athan told me to post this
"Pantaa Cezisa" was supposed to come from the FD data set but is not in the game anymore

Plotted a rote from eranin to tanmark with 19.6 ly range
the route is like this:
0 Eranin 0.00 0.00
1 LHS 3006 7.51 7.51
2 44 chi Draconis 17.13 24.64
3 Eta Cassiopeiae 16.20 40.84
4 Pantaa Cezisa 15.40 56.24 <-----
5 BD+02 305 13.74 69.98
6 LFT 142 8.23 78.21
7 LFT 163 17.51 95.73
8 53 chi Ceti 18.58 114.31
9 Tanmark 11.68 125.99

looking at the map the real jump from eta cassiopeiae over to bd+02 305 could be "George Pantazis"
This system was not in the DB , so i added it ...

see the screeny too for reference
 
Athan told me to post this
"Pantaa Cezisa" was supposed to come from the FD data set but is not in the game anymore

Plotted a rote from eranin to tanmark with 19.6 ly range
the route is like this:
0 Eranin 0.00 0.00
1 LHS 3006 7.51 7.51
2 44 chi Draconis 17.13 24.64
3 Eta Cassiopeiae 16.20 40.84
4 Pantaa Cezisa 15.40 56.24 <-----
5 BD+02 305 13.74 69.98
6 LFT 142 8.23 78.21
7 LFT 163 17.51 95.73
8 53 chi Ceti 18.58 114.31
9 Tanmark 11.68 125.99

looking at the map the real jump from eta cassiopeiae over to bd+02 305 could be "George Pantazis"
This system was not in the DB , so i added it ...

see the screeny too for reference

And note that was in my local DB as:

Code:
    name      |     x     |  y  |     z     | tgc_id |     createdate      | commandercreate |     updatedate      | commanderupdate | cr
Pantaa Cezisa | -12.09375 | -16 | -14.21875 |  15261 | 2014-11-25 17:25:11 | FD              | 2014-11-25 17:25:11 | FD              |  5
 
Athan told me to post this
"Pantaa Cezisa" was supposed to come from the FD data set but is not in the game anymore
...
looking at the map the real jump from eta cassiopeiae over to bd+02 305 could be "George Pantazis"
This system was not in the DB , so i added it ...

George Pantazis was one of the top tier Kickstarter backers: http://www.polygon.com/2014/6/26/5838526/kickstarter-whales. Probably what happened here is that the system he got to name has been given his name instead of the name he chose. I think you should ticket this as a bug.
 
Some good comments/observations there.


We've had a long discussion about this already - and most agree with you.
I however, for various reasons, have a preference for POST with everything.
(maybe try a search for POST in this thread for the whole debate and arguments con/pro if you are really interested)



"data" and "d" are standard "wrappers" - Nothing I can do about those. You'll get those no matter what (JSON) API you query.

I'm actually only returning (C# example)
Code:
return new {
          ver = version.currentMinor,
          date = DateTime.UtcNow.ToString(EDSCString.dateFormat),
          status = new { input = InputStatusMessages }
        };

So they are not part of the API - But part of working with the XMLHttpRequest object, hence why it's not mentioned in the API.

In my work with XMLHttpRequest and jQuery AJAX so far I've never used such fields. XMLHttpRequest (which isn't mentioned in the docs either btw) and even the jQuery examples access the fields explicitly as "data" and "d", so the assumption that it is standard is incorrect: they must have been added to the examples. XMLHttpRequest does not seem the have a data member according to MDN, and there are certainly APIs not returning "data" either (neither "d"), last time I checked was the Google Directions JSON API.

Keep in mind I've mentioned this only because I've spent ages debugging it, but it is a not an issue for me from now on. As I hear there is at least one other commander who had the same experience.

I don't know much about C#, but in Go one can easily check if an input parameter is null/nil, and can parse the HTTP request and format the HTTP response however she wants. Most notably if JSON is decoded or encoded, no padding is added, unless such padding is specifically asked for.

Btw, I'd suggest to add the optional from_id and to_id parameters to the GetSystems (perhaps GetDistances too), as long as the DB uses AUTOINCREMENT on id, it should be enough to support incremental queries for database initialization.

I'd like to propose an optional JSON stream based query for the data: likely with separate "system" and "error" objects on the wire. In this case neither the server nor the client would have to keep all data in memory. It is not an issue now, but will be once we have millions of systems.
 
I think that might be a .net thing. Certainly it's possible to return json without that wrapper in (e.g.) node.js. Anyway, it's obvious from the examples how it works so I don't think it's a problem.

I agree it is obvious if one uses JavaScript, I guess in that case I would have just copied the function in the docs or the pastebin example and immediately noticed it. But I always use JSON with reflection, and it just works, the only exception being EDSC so far. It's true APIs I've seen so far use GET even for complex queries, therefore such an issue it can't happen at all.

It is not a problem if those fields are there, the problem is that they are easy to miss.
 
In my work with XMLHttpRequest and jQuery AJAX so far I've never used such fields. XMLHttpRequest (which isn't mentioned in the docs either btw) and even the jQuery examples access the fields explicitly as "data" and "d", so the assumption that it is standard is incorrect:
Yes - rw corrected me already.
It's .NET specific.
So used to .NET, I'd forgotten in the moment.


I don't know much about C#, but in Go one can easily check if an input parameter is null/nil, and can parse the HTTP request and format the HTTP response however she wants.
It's not *impossible* in .NET, but it would require to ignore the inbuild parser, and make my own from scratch (with whatever bugs that might introduce).
I'm currently not willing to spend the time and effort on that.



Btw, I'd suggest to add the optional from_id and to_id parameters to the GetSystems (perhaps GetDistances too), as long as the DB uses AUTOINCREMENT on id, it should be enough to support incremental queries for database initialization.
I'm a bit torn on this one.
"In principle" the id's don't (publicly) exist - So maybe it was a mistake to add them to the verbose output. But I thought some might find use for it.
And while yes, they are IDENTITY values, I'm not really keen on building an API around them.
Things can change though - I'll put it on the TODO list way down at the bottom as an "investigate" item.


I'd like to propose an optional JSON stream based query for the data: likely with separate "system" and "error" objects on the wire. In this case neither the server nor the client would have to keep all data in memory. It is not an issue now, but will be once we have millions of systems.
Don't understand this one?
 
I'm a bit torn on this one.
"In principle" the id's don't (publicly) exist - So maybe it was a mistake to add them to the verbose output. But I thought some might find use for it.
And while yes, they are IDENTITY values, I'm not really keen on building an API around them.
Things can change though - I'll put it on the TODO list way down at the bottom as an "investigate" item.

I agree ids should generally not be on the public API. But getting the whole DB is a necessity, and I could not think of a better incremental way to do that so far.

Don't understand this one?

If the JSON encoder and the HTTP handler want to play nicely, the server needs to cache the whole data set being returned in memory, to be able to figure out the value for the Content-Length header. (I'm not sure edsc does that, though.) The client will also need to receive all the data to see the closing ] and } at the end of the wrapper, so it knows it is indeed correct. It is somewhat related to the issue with ids above: everyone needs to get all the TGC data somehow once in a while, and eventually the data will be too large to keep in memory.
 
Back
Top Bottom