Discussion What is the most efficient way to crowdsource the 3D system coordinates

Athan

Athan
Inhumierer said:
I'm not sure how to get the TZ. I could generate a list of available timezones and let registeres users choose & save their preferred timezone. This list has 1781 entries. I didn't know there are this many timezones... Maybe I'll do some little Javascript to read the client's (webbrowser's) time and use it as TZ info.
Click to expand...
Ah yes, good luck with that. You probably want some approach that takes browser local time and compares that to server time to present possible options to the user. Remember DST can cause further headaches, do not go storing just an hours offset, you want to store the (POSIX?) timezone name instead.

Oh, and as for running PHP or such in UTC, it should be as simple as:

Code: 
putenv("TZ=UTC");
The equivalent certainly works for me with perl... on Linux anyway, I'm not going to warranty anything under Windows. Oh, now I look at PHP docs for date() I see there's a date_default_timezone_set(), so perhaps it's not quite so simple in PHP.
 

Inhumierer

Inhumierer
There are a few samples found in the net where people did write javascript to query the GMT offset from the client, but it didn't work reliably from different systems & browsers. So I added a few lines where registered users can choose their timezone from mysql's timezone list and save it. The database connection is set to this timezone, and PHP TZ is set with date_default_timezone_set. Everything shoud be converted into the user's correct timezone when using the web FE.
 

RedWizzard

RedWizzard
Athan said:
That strikes me as exactly the wrong way to do it. ALWAYS use a unique static ID on a data tuple. Anything else that is actual data is subject to change. We know FD have changed/corrected some system names since release and there's nothing to say they won't eventually rename some of the PG names to a more human monicker. Either way you have to deal with merging data from the different names, but using a unique ID means that the name is the only thing you need to change, not also anything that was referring to the system. Remember there may be any number of other programs making use of the data. Think of a simple Explorer Notes app that allows a player to makes notes about the system, you'd want those tied to a unique, never changing, ID for the system, not a name that you may then have to search/replace in the application data due to a rename.
Click to expand...

I think you've misunderstood me a little, the ID is derived from the system name (trimmed and forced to lowercase) but the name is still separate data in it's own right. If the name gets updated in a session I don't have to change the ID. I don't need persistent IDs between sessions, in fact they'd make it harder, so I don't use them.

There's no requirement for IDs to be numeric, a string is perfectly acceptable provided it's unique, and Michael did tell us that system names are supposed to be unique. The game identifies systems by name: the name is already an ID in a public interface. So if you're going to use an ID controlled by someone else, it might as well be the one from the game, no? That is the canonical one, after all. One of the tenants of defensive design is to never trust input, trusting an unnecessary ID is just extra risk (good example is how TGC screwed up Sol and a couple of other reference systems).

Your argument about changes to a system name is reasonable, but in practice in most non-trivial systems you have to deal with updating references anyway (think merges) so generated IDs don't solve the problem (I used to do this stuff for a living so I'm aware of the issues and tradeoffs, also note I'm not currently using a local database - if I was I'd certainly be using primary key IDs).
 
Last edited:

RedWizzard

RedWizzard
Inhumierer said:
That's a good point. I admit that I didn't take a close look at timezones until now - I thought "PHP & MySQL will do that stuff"... ;)
Now I did. The date & time when the last update of a system or distance happened was stored without timezone data. Now it is. All dates & times are stored in UTC, that's good. The API accepts every date & time format MySQL is able to understand. Bad thing: The API gives all date/time data converted to the server's timezone (because the scripts are running there), and that's GMT+2 (Europe/Berlin, CEST). I'll fix that, soon. I think the API sould format everything in GMT, for this is a global notation which every 3rd party tool should understand.
For user output from the web FE it would be nice to know the user's timezone and have everything converted. This is cosmetics, so not at high prio. I'm not sure how to get the TZ. I could generate a list of available timezones and let registeres users choose & save their preferred timezone. This list has 1781 entries. I didn't know there are this many timezones... Maybe I'll do some little Javascript to read the client's (webbrowser's) time and use it as TZ info.
Click to expand...

Just specify that all dates are UTC (GMT) in the api, no need to actually handle timezones (provided php/mysql doesn't do something unexpected to the dates on your behalf). Not that it's hard to handle TZs but too often it gets forgotten. TGC, for example, takes dates without a TZ and I have no idea what TZ it default to. I'm not 100% confident that when TGC says "here's the data up to time X" and then I later say "give me data since X" I'm actually getting all the data. I have to insert an overlap to be sure.
 

RedWizzard

RedWizzard
Zenith said:
I'm not going to go through all 75 pages of posts to find a particular answer, and I've used the search facility without any joy.

I like to trilaterate systems using Finwen's entry page, and also add distances to some uncalculated systems to add to the pool.
Once an uncalculated system has a critical number of reliable measurements from trilaterated systems, does the database perform the calculation and determine the location of the uncalculated system?

I'm thinking there are some systems that are WAY beyond the ability to reach with FSD, but would make good markers to trilaterate against.
Click to expand...

I believe TGC will only calculate coordinates for one system in a particular distance pair. So say you add a system 'A' with 7 distances to reference systems and 1 distance to a system 'B' which doesn't currently have coordinates (but should now be able to be calculated). TGC will only calculate the coordinates for 'A'. The workaround is to submit the 7 reference distances first and then send the distance to 'B' in a separate submission.
 

Inhumierer

Inhumierer
RedWizzard said:
Just specify that all dates are UTC (GMT) in the api, no need to actually handle timezones (provided php/mysql doesn't do something unexpected to the dates on your behalf).
Click to expand...

Sure, that's how I did it. All dates you receive or submit from/to the API are UTC. Timezone setting is only taken into account when using the web frontend, f.ex. manually checking why system x does not have coordinates, check who and when someone transmitted distances etc. And when you have submitted distances, it is nice to have the date & time for them be displayed in your local timezone. In the database everything is UTC.
 
Last edited:

Athan

Athan
RedWizzard said:
I think you've misunderstood me a little, the ID is derived from the system name (trimmed and forced to lowercase) but the name is still separate data in it's own right. If the name gets updated in a session I don't have to change the ID. I don't need persistent IDs between sessions, in fact they'd make it harder, so I don't use them.

There's no requirement for IDs to be numeric, a string is perfectly acceptable provided it's unique, and Michael did tell us that system names are supposed to be unique.
Click to expand...
And we all know that they've managed not to stick to that uniqueness. It would be unwise to assume we've found all the gotchas on that point. IIRC the spreadsheets/lists we got out of MB were for all systems with an economy. I wouldn't assume that every non-PG named system has/had an economy at that point in time.

RedWizzard said:
The game identifies systems by name: the name is already an ID in a public interface. So if you're going to use an ID controlled by someone else, it might as well be the one from the game, no? That is the canonical one, after all. One of the tenants of defensive design is to never trust input, trusting an unnecessary ID is just extra risk (good example is how TGC screwed up Sol and a couple of other reference systems).

Your argument about changes to a system name is reasonable, but in practice in most non-trivial systems you have to deal with updating references anyway (think merges) so generated IDs don't solve the problem (I used to do this stuff for a living so I'm aware of the issues and tradeoffs, also note I'm not currently using a local database - if I was I'd certainly be using primary key IDs).
Click to expand...
And again that's exactly why I'd use a unique, guaranteed never ever changing, ID. Basing that off the current system name is just going to sow confusion if the system name ever does change (again I'm thinking in terms of a never-changing unique ID). You don't want to have "Wimble Land" as your internal unique ID when it then gets changed to "Womble Land", it will cause confusion. Far better for it to be "ID:28563" IMO. Even worse for duplicate system names (again I would NOT assume we've found all of those yet). You'll have people talking about one of them with others thinking they're referring to the other, and depending on the code/application at hand it/they might not realise that two different systems are being discussed.

But, yes, of course on fully public interfaces you will be providing the currently known in-game name for a system as well, any API would be useless without it.

But, eh, it's not like I'm actively coding in this arena. I'm just advocating an abundance of caution so there are as few gotchas as possible. Even if 'you' (generic you) know what you're doing others using an API may not be as well versed so as to be sure they steer clear of those gotchas.
 

RedWizzard

RedWizzard
Athan said:
And we all know that they've managed not to stick to that uniqueness. It would be unwise to assume we've found all the gotchas on that point. IIRC the spreadsheets/lists we got out of MB were for all systems with an economy. I wouldn't assume that every non-PG named system has/had an economy at that point in time.
Click to expand...

There are about 144,000 real world objects in ED, so we don't have most of them. But there won't be any duplicates in the catalog systems (HIP etc), only in the constellation based names. I'd be surprised if there were more than a couple more dupes out there. The dupes screw it all up anyway. If someone enters data for "m Carinae" which ID do you attach that data to? Having generated IDs doesn't help you because the user doesn't know what ID the system they are looking at in game has.

Athan said:
And again that's exactly why I'd use a unique, guaranteed never ever changing, ID. Basing that off the current system name is just going to sow confusion if the system name ever does change (again I'm thinking in terms of a never-changing unique ID). You don't want to have "Wimble Land" as your internal unique ID when it then gets changed to "Womble Land", it will cause confusion. Far better for it to be "ID:28563" IMO. Even worse for duplicate system names (again I would NOT assume we've found all of those yet). You'll have people talking about one of them with others thinking they're referring to the other, and depending on the code/application at hand it/they might not realise that two different systems are being discussed.

But, yes, of course on fully public interfaces you will be providing the currently known in-game name for a system as well, any API would be useless without it.

But, eh, it's not like I'm actively coding in this arena. I'm just advocating an abundance of caution so there are as few gotchas as possible. Even if 'you' (generic you) know what you're doing others using an API may not be as well versed so as to be sure they steer clear of those gotchas.
Click to expand...

I'm not talking about a public interface, I'm talking about internal use. If your own system is confusing you, well you've got bigger problems than merely what you've chosen to use as an ID.

In your example, how is "Wimble Land" actually different to "ID:28563"? They are both opaque keys that are meaningless. The fact that one is somewhat similar to the system's name is irrelevant. Yes, systems with IDs of "Wimble Land" and "Womble Land" could be easily mixed up at a casual glance. So could "ID:28563" and "ID:28553".

Now, IDs in public APIs... in TGC, the IDs are worthless to me: they are a 1-to-1 mapping to system names, and the system names are also unique and static (with the exception of the buggy reference system copies). So they don't add any benefit over using the system name, all they are is more wasted bandwidth. Note that you can't submit data against an ID. And they are not used as references in the data format so I have to check for inconsistencies: in the distance data the reference systems' names and coordinates are reproduced, what if two distances to a system have the same target ID but different names or coordinates? Shouldn't happen, but you can't trust your inputs. So as an API user I have to do more work (well actually I just ignore the replicated data in distances.json, but I hope you see the point), and making your users do unnecessary work is simply bad design. This is not the fault of IDs per se, it's the consequence of blinding adding IDs without thinking. If a TGC replacement is going to have IDs, the data format needs to be normalised (in the RDBMS sense of the word).
 

Athan

Athan
Well, yes, I was addressing the proper way to do things, not a system with all the bugs, misfeatures and lack of features like EDSC (I'm not dissing TornSoul, if he'd not gone AWOL I'm sure he would have addressed all of this long ago, but it's been 5 months now).

Non-normalised DB designs give me the heebie jeebies.
 

RedWizzard

RedWizzard
Athan said:
Well, yes, I was addressing the proper way to do things, not a system with all the bugs, misfeatures and lack of features like EDSC (I'm not dissing TornSoul, if he'd not gone AWOL I'm sure he would have addressed all of this long ago, but it's been 5 months now).

Non-normalised DB designs give me the heebie jeebies.
Click to expand...

Denormalised DBs are often necessary for performance, but they require care (though fully normalised DBs do as well).
 
Last edited:

Inhumierer

Inhumierer
In 1.3 beta CS Camelopardalis is roughly located at -1198/112/-1523.6
Edit: Spelling in"normal " game is CS Cameopardalis
 
Last edited:

Finwen

F
Cl Pismis 19 and Cl Pismis 13 points to the same star in the galaxy map.

Iit looks strange in the jump list with next system 46 ls away....

www Screenshot_0128.jpg
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom