Discussion What is the most efficient way to crowdsource the 3D system coordinates

[Chromatix]

BTW, I can produce a fourth version of my maps at any time, as long as I have a complete set of distance data (since I'm calculating the coordinates from the distances myself). My scripts also happen to produce a few statistics which might be a useful guide to locating the (relatively few) remaining systems.

I fear the quibbling over 0.001 ly measurement discrepancies may be missing the point slightly. There's obviously something odd going on in the game's display code, but it's not going to have a practical effect on jump fuel or stuff like that. I might tighten up my own code so that it is more sure of finding exactly precise coordinates, but that's not a massively high priority for me.

 

[wolverine2710]

BTW, I can produce a fourth version of my maps at any time, as long as I have a complete set of distance data (since I'm calculating the coordinates from the distances myself). My scripts also happen to produce a few statistics which might be a useful guide to locating the (relatively few) remaining systems.

I fear the quibbling over 0.001 ly measurement discrepancies may be missing the point slightly. There's obviously something odd going on in the game's display code, but it's not going to have a practical effect on jump fuel or stuff like that. I might tighten up my own code so that it is more sure of finding exactly precise coordinates, but that's not a massively high priority for me.

A fourth version based upon the now found 571 coords in the bubble would be great. Checked RW's distances.json file. Can't see directly if all 571 sytems have distances BUT the systems.json file has 3D coords for systems. Can't you use that file to create all distances for your script?

I Do like the statistics info. Just checked your site. Your Most isolated 10 systems list gives Hera 11.475 as most isolated. Am I correct to assume that with a ship capable of a 11.475 LY jump range ALL 571 stars can be reached? I ask this because of Snubles post of yesterday which you can find here. Don't know how he measured, manually or using a script but he writes: "The most isolated system that I found in Smackers data is Tring, with 11.325LY to nearest system WREDGUIA DJ-0 B47-1. So inside the pill you need 11.325LY jump capability to visit all "known" systems. Correct? "
Wondering what is the correct jump range to reach the complete SB1 pill?

Can your script also be used to show a commander who wants to upgrade his ship what his island will be? As in will he be stuck or not after upgrading?

I didn't see a link to the source on your site. Am I mistaken or haven't you made your script public?

 

[Snuble]

Hera should be only 7.78LY from WOLF 483. That system is 9.85LY from LHS 3651, and from there is a unbroken link of less then 10LY jumps to the core systems.

 

[gazelle]

I know your list under contributors multiple names (good), but I can't seem to find Gazelles name. What is your policy for the 'contributor' column.

I'm not a 'contributor' of measured distances. I just calculated systems based of the work of the real map surveyor. I just collected all the numbers I could find and run them through my program.

 

[TornSoul]

nope, it's 131.337. Here's a screenie

http://imgur.com/EU062sd

(sorry for the quality, even high-res screen-grabs when using oculus rift aren't very good for some reason)

Super - Thanks for the confirmation.

Very very very odd....

 

[wolverine2710]

I'm not a 'contributor' of measured distances. I just calculated systems based of the work of the real map surveyor. I just collected all the numbers I could find and run them through my program.

The "news systems" list gave me the wrong impression. Thanks for clearing this up. "real map surveyor(s)" it has a nice ring to it. Better then volunteers ;-) Might use that term in the future.

 

[RedWizzard]

@RedWizzard. Just pulled the latest version from your repository, looks nice. Has all data provided here in this forum now been checked/confirmed by your tool. Reason for asking, three days ago Gazelle posted new sytems. I know your list under contributors multiple names (good), but I can't seem to find Gazelles name. What is your policy for the 'contributor' column.

My policy is, "can't accept them with no distance data". With Gazelle's case I don't think I even used the names or checked the coordinates. Generally I've credited anyone who contributed significant numbers of distances. The definition of "significant" has varied a little though generally meant at least 3 distances. Unless I already had a lot of distances for a star. I do plan to compare what I have to what is in TD so some of the credits may change.

It might be worth talking about whether we want a common policy and what it should be. Do we give credit just for a name? Name and coordinates? How much distance data gets you a credit? Etc.

 

[MadDogMurdock]

This question is best in the TD thread, but I will go a bit OT here.
I saw Smackers post. The basic idea of TD is manual input - which is what author kfsone likes. But things can and were automated with emdn-tap which got its data from EMDN. Data for TD could get from Thrudd, it could get from Maddavo.

The big question is, is Thrudd against scraping tools which access ED and/or protocols which was the base of EMDN (ED)? Or is he against automating things? His own tool is automating things - multiple volunteers enter data on his side. What if OCR-ing the market prices succeeds - in line with the new FD data access policy? Would he be against that. Would he be against combining data from multiple sites - data entered manually?. That you have to ask him.

I hesitate to speak for Thrudd and the ethics of tools is grey but the basis seems to be the old pen and paper, but shared by whoever participates, which to me fits with this being a social multiplayer game.

Obviously, it is automating some things like routes but that is based on manual gathering of the base coordinates.

The tool allows direct input of coordinates but also a triangulation method based on the distance to Sol plus 3 other systems. That does leave room for error as discussed widely in this thread.

I'll mention this thread to him because obviously he'd like people to use his tools but he hasn't really done any self-promotion.
I'm sure there's room for collaboration and data sharing that would benefit all these types of tools.

 

[wolverine2710]

My policy is, "can't accept them with no distance data". With Gazelle's case I don't think I even used the names or checked the coordinates. Generally I've credited anyone who contributed significant numbers of distances. The definition of "significant" has varied a little though generally meant at least 3 distances. Unless I already had a lot of distances for a star. I do plan to compare what I have to what is in TD so some of the credits may change.

It might be worth talking about whether we want a common policy and what it should be. Do we give credit just for a name? Name and coordinates? How much distance data gets you a credit? Etc.

Speaking for myself. I'm not doing this thread, coordinates, distances for the credits. Though I have to admit it put a smile on my face when I saw my name show up in kfsone's release notes (OP) for a certain version. Same for seeing my name in the contributors column.

When things get automated it probably is easier then now. Now this thread is full of multiple volunteers supplying distances - though to be honest most things seem to come from commanders who have created programs. Extremely hard to keep of who did what.

When things are automated Multiple tools can/will be uploading data to a central place. The tools could provide for an optional "players name" field which could be collected. This can be used for all kind of statistics stuff if needed at a certain point. It can be shown in the contributor column, it could be shown on a Top50 contributors list etc. Does the central admin needs to have multiple contributors for a certain system before we can sure that the data is valid? If only one entry is needed - assuming enough distances are input we could flag that system as done. Hence only 1 entry in the contributors column. If we need two, it will be two entries etc.... As long as the base is there we can later decide if a 'top list is shown'.

 

[RedWizzard]

Has all data provided here in this forum now been checked/confirmed by your tool.

All data in systems.json has been verified by me. Each system has enough distance data to provide good coordinates* (at least 5 distances). Every system I known of (including the reference systems from Michael) has been seen by me in the navigation panel of a nearby star, in the correct place in the list. I didn't check the distance beyond making sure it was in the right place in the list, but this still give me very high confidence that the data is right.

It also gives me reasonable confidence that there are no more stars to be found via the navigation panel in the Pill. I say "via the navigation panel" because I have seen about a dozen stars that don't consistently show up in the navigation panels as they should. When you look from some systems they are there, in other systems they are not. It would be good if people can verify and ticket this bug because it's one that will cause us problems down the line if it's not squashed. List of the systems I've seen that are affected:

Spoiler

Wredguia LW-E d11-12*
Wredguia XH-Q b46-*
Haras
Manamaya
Zhu Rong
Vaccimici
Tapipinouphinien
Mistana

During this process, (which has taken the last 3 or 4 hours), I found one new system:
Wredguia IF-L b49-3 (-88.375, 35.40625, -4)
Distances

Spoiler

Sol: 95.288
Wolf 497: 90.133
Huokang: 79.104
Demeter: 35.412
Clotti: 11.982
Fu Haiting: 48.806
San Guaralaru: 67.467
Haras: 40.824
Arabha: 38.037

SQL:
INSERT INTO "System" VALUES(,'Wredguia IF-L b49-3',-88.375,35.40625,-4,'2014-10-21 12:43:18');

So I believe that is probably the last system in the Pill, for a total of 572. There is a small chance that one could be hiding in the galaxy map but not visible in the navigation panel - I won't be hunting via the galaxy map but I hope someone does.

* The one exception is Alpha Cygni which is well outside the Pill.

 

[RedWizzard]

I've been comparing the "generate csv" button output with that of the system.csv file by Smacker. Your output IS (basically) the format of systems.csv in TD.

Smacker is now manually updating the systems.csv file in his fork - which is then being pushed into kfsone's master. This takes time every time. Wouldn't it be great if he could just take your generated output CSV OR create a small python scripts which retrieves your system.json file (repository) and creates an systems.csv from it in his fork. TD could even have a command to create on the fly a new systems.csv file. Last action gives a problem when getting the latest version of kfsone by for example using SourceTree (fetch and pull command) or I think git clone from the command line (not a gitter).

The intention is that it output the exact format for TD so it could be dropped straight in. Obviously someone would want to check the changes before checking it into Git though. It's kind of a workaround for the offline nature of my stuff at the moment though: with an online system there would just be a url that you could pull an up to date, dynamically created csv from.

Differences I noticed between your file and Smackers one.

1) His system name entries are sorted alphabetically (ascending). Your csv output depends on the sort order selected in sytems.html. Sort ascending on the system colum and the output is the same.
2) His csv file has a header line, your doesn't. His header line is: name,pos_x,pos_y,pos_z,name@Added.added_id,modified
3) capitalisation scheme. You've adopted the scheme used by ED/FD it seems. Look good. His naming convention sometimes follows your, sometimes not. Examples: Wredguia vs WREDGUIA, Wregoe vs WREGOE. I saw in his pull request he doesn't mind what capitalisation scheme to use.
4) x, y and z coordinates columns seem to match 100%
5) Smackers "name@Added.added_id" column seems to represent TWO things: name author AND "not present" (when system is not in current bubble). TD's systems.csv has no comment column so that is probably the reason. I DO like the "not present" tag but only needed for beta and gamma. Not when the whole bubble is there.
Back to contributors: Smackers has either one specified author or the tag Combined - when data is from more then one person. Iirc the combined tag comes form Bernd's TD fork. You do specify multiple authors. I DO like your crediting style.
6) Modified column (date). Yours is mostly empty. His column is always filled. NOT sure what his date exactly represents. Put in fork, or date of received data? Date/time settings are the same, as in 2014-10-02 14:21:15.
7) I noticed one undefined author in your file: 'Perkwunos',-40.5,46.75,6.75,'undefined',''

1) I can change this so the csv output is always sorted alphabetically independently of the page sort order.
2) will add that.
3) I'd prefer to use what appears to the be the official spelling. I'm not too bothered though.
4) good.
5) in systems.json I've added a "region" entry that has "beyond beta 2" for those stars outside the Pill. I plan to fill this for the rest of the systems too: "alpha", "beta 1", "beta 2", etc. I'll use this to override the credit line where necessary for the csv output.
6) I plan to import dates from TD for this. Mine only has dates for the stuff I've entered I think.
7) Thanks, that's a bug in entry.html.

Main difference between your and smackers file:
1) single vs multiple contributors. I like multiple authors credited by name instead of "combined"
2) "Not present tag". I like that one.
3) capitalisation scheme.

If you both agree that the above is not relevant, I'm totally fine with it.
If you both agree there are some merits in what I wrote, maybe you can sync both files once and for all.

Just my euro cent about syncing both files.

Mostly I agree - many of these things were already on my list to do.

 

[RedWizzard]

BTW, I can produce a fourth version of my maps at any time, as long as I have a complete set of distance data (since I'm calculating the coordinates from the distances myself). My scripts also happen to produce a few statistics which might be a useful guide to locating the (relatively few) remaining systems.

I fear the quibbling over 0.001 ly measurement discrepancies may be missing the point slightly. There's obviously something odd going on in the game's display code, but it's not going to have a practical effect on jump fuel or stuff like that. I might tighten up my own code so that it is more sure of finding exactly precise coordinates, but that's not a massively high priority for me.

A fourth version based upon the now found 571 coords in the bubble would be great. Checked RW's distances.json file. Can't see directly if all 571 sytems have distances BUT the systems.json file has 3D coords for systems. Can't you use that file to create all distances for your script?

Get the distance data from systems.json - it should now be complete for all distances I have. Distance.json is going to end up only have distance data for systems that are not in the main file.

On the 0.001 thing. It may seem like quibbling but it's actually important IMO. It has ramifications for verifying that input data is correct: if a distance is out by 0.001 is that just ED showing the wrong distance or is it a typo from the user or is the star in the wrong place? It could also have indicated that there was a problem with our assumption that everything is on a 1/32 Ly grid. However, I've spent a some time tonight looking for a case which disproves that last one thankfully. I've found a number of cases where the distance between two reference stars is displayed incorrectly:

Tun - Tring shows 83.963 (should be 83.962)
Aeolus - Sigma Bootis shows 55.168 (55.169)
LP 29-188 - LHS shows 42.819 (42.818)
Hyperion - Kamchaultultula shows 50.569 (50.570)

IMO this confirms it's an ED bug rather than a problem at our end. That's worth knowing.

 

[RedWizzard]

Speaking for myself. I'm not doing this thread, coordinates, distances for the credits. Though I have to admit it put a smile on my face when I saw my name show up in kfsone's release notes (OP) for a certain version. Same for seeing my name in the contributors column.

I hadn't really planned on it being particularly rigorous. I just thought it might be fun to see who found what, generate some simple stats, etc.

Anyway I'm off to bed. I think I'm done with star searching for beta 2 so tomorrow I'll probably get back to some minor code improvements. And maybe see if I can figure out boundaries of the Pill (i.e. what's the equation for that cylinder).

 

[TornSoul]

Tun - Tring shows 83.963 (should be 83.962)
Aeolus - Sigma Bootis shows 55.168 (55.169)
LP 29-188 - LHS shows 42.819 (42.818)
Hyperion - Kamchaultultula shows 50.569 (50.570)

IMO this confirms it's an ED bug rather than a problem at our end. That's worth knowing.

In all those cases FD shows the value as being 0.001 too high (ie. none the other way)

Whether that's significant or not - who knows. Just an observation.

EDIT: bah nvm. I can't read apparently. They are mixed...

 

[Smacker]

Smacker is now manually updating the systems.csv file in his fork - which is then being pushed into kfsone's master. This takes time every time. Wouldn't it be great if he could just take your generated output CSV OR create a small python scripts which retrieves your system.json file (repository) and creates an systems.csv from it in his fork. TD could even have a command to create on the fly a new systems.csv file. Last action gives a problem when getting the latest version of kfsone by for example using SourceTree (fetch and pull command) or I think git clone from the command line (not a gitter).

Differences I noticed between your file and Smackers one.

1) His system name entries are sorted alphabetically (ascending). Your csv output depends on the sort order selected in sytems.html. Sort ascending on the system colum and the output is the same.
2) His csv file has a header line, your doesn't. His header line is: name,pos_x,pos_y,pos_z,name@Added.added_id,modified
3) capitalisation scheme. You've adopted the scheme used by ED/FD it seems. Look good. His naming convention sometimes follows your, sometimes not. Examples: Wredguia vs WREDGUIA, Wregoe vs WREGOE. I saw in his pull request he doesn't mind what capitalisation scheme to use.
4) x, y and z coordinates columns seem to match 100%
5) Smackers "name@Added.added_id" column seems to represent TWO things: name author AND "not present" (when system is not in current bubble). TD's systems.csv has no comment column so that is probably the reason. I DO like the "not present" tag but only needed for beta and gamma. Not when the whole bubble is there.
Back to contributors: Smackers has either one specified author or the tag Combined - when data is from more then one person. Iirc the combined tag comes form Bernd's TD fork. You do specify multiple authors. I DO like your crediting style.
6) Modified column (date). Yours is mostly empty. His column is always filled. NOT sure what his date exactly represents. Put in fork, or date of received data? Date/time settings are the same, as in 2014-10-02 14:21:15.
7) I noticed one undefined author in your file: 'Perkwunos',-40.5,46.75,6.75,'undefined',''

Main difference between your and smackers file:
1) single vs multiple contributors. I like multiple authors credited by name instead of "combined"
2) "Not present tag". I like that one.
3) capitalisation scheme.

If you both agree that the above is not relevant, I'm totally fine with it.
If you both agree there are some merits in what I wrote, maybe you can sync both files once and for all.

Just my euro cent about syncing both files.

I already have a Python script that processes the json file. Was meaning to get the case sorted-out now we know the canonical format. The contributors was useful when we were checking for errors, I think that now we have a stable muti-checked group list this should just say "Beta-2-Inferred" or something, unless you still think it has some use other than a very exclusive sort of e-preen

 

[TornSoul]

The contributors was useful when we were checking for errors, I think that now we have a stable muti-checked group list this should just say "Beta-2-Inferred" or something, unless you still think it has some use other than a very exclusive sort of e-preen

I'm in the camp of "Contributor? - Who cares?" as well

 

[TornSoul]

@wolverine2710
"When you read THIS line it means I'm currently updating the OP. Will be removed when ready."

That's still in the OP - Just letting you know in case you thought you had hit submit, but in fact didn't. As it's been there for a good while now.

 

[Harbinger]

A fourth version based upon the now found 571 coords in the bubble would be great. Checked RW's distances.json file. Can't see directly if all 571 sytems have distances BUT the systems.json file has 3D coords for systems. Can't you use that file to create all distances for your script?

I Do like the statistics info. Just checked your site. Your Most isolated 10 systems list gives Hera 11.475 as most isolated. Am I correct to assume that with a ship capable of a 11.475 LY jump range ALL 571 stars can be reached? I ask this because of Snubles post of yesterday which you can find here. Don't know how he measured, manually or using a script but he writes: "The most isolated system that I found in Smackers data is Tring, with 11.325LY to nearest system WREDGUIA DJ-0 B47-1. So inside the pill you need 11.325LY jump capability to visit all "known" systems. Correct? "
Wondering what is the correct jump range to reach the complete SB1 pill?

Can your script also be used to show a commander who wants to upgrade his ship what his island will be? As in will he be stuck or not after upgrading?

I didn't see a link to the source on your site. Am I mistaken or haven't you made your script public?

I ran a check of the calculated distances from all known stars to all other known stars and found that the most distant jump was indeed:

From: Tring
To: WREDGUIA DJ-O B47-1
Distance: 11.325 LY

So as long as your ship can jump this range you can visit any star that we currently know about in the pill.

Also, the shortest jump is:

From: 26 Draconis
To: LFT 1361
Distance: 0.328 LY

 

[wolverine2710]

@wolverine2710
"When you read THIS line it means I'm currently updating the OP. Will be removed when ready."

That's still in the OP - Just letting you know in case you thought you had hit submit, but in fact didn't. As it's been there for a good while now.

Thanks for the headup BUT I'm still updating it, but got distracted today by this thread and RL. Next step, the urls....

 

[KevinMillican]

Elite Dangerous Pilot Log

Another possibility for people to consider; I'm working on an online logbook where I can add comments, typical commodities etc. to systems and stations I visit. (My notebook is just getting too scruffy ! )

Users can add distances to other systems in there as well.

At the moment, registered users can edit details of existing systems and stations, since I uploaded the beta 2 systems that have economies but have not visited them all. In future, only the person who creates a record will be able to edit or delete it.

You get to see everyone's input data and comments. Not sure if this is a good idea long term, but would welcome comments.

http://www.millican.info/edlog