Discussion What is the most efficient way to crowdsource the 3D system coordinates
- Thread starter wolverine2710
- Start date
ShadowGar
Banned
I just updated trade dangerous last night with all of the systems provided by Brookes. I also have been updating the non-market systems with codec's calculations but we are getting reports that a few systems are off. Specifically Hagalaz.
After speaking with Brookes, he said that he does not have a convenient way of collecting systems and compiling a list right now. I'm betting that until an API comes out, what we have is all we are getting.
So figuring out and having a website or program coded that can perform star calculations to determine coords and a method of either OCR scraping or something else that can automate data should be our first priority. Betting on an API that will give us what we want would be wrong. I doubt they would release an API that provides all of the information we had during scraping.
The other issue also is, how many stars are we going to grab? Is it feasible to expect a trade program to have a database of all 400 billion stars? Or do we only list a few thousand systems for trade purposes and keep the traders in that bubble?
Even the collective total of everyone who ever plays ED is never going to make much of a dent in the 400b systems. Ballpark that the start of gamma/retail will have ~150,000 systems available, and then add in whatever gets explored, and for traders, actually gets a station added.
Definitely doable. Although any route-finding is going to need a nice fast lookup on "stars near X,Y,Z" I expect.
ShadowGar
Banned
But without an automated method of obtaining these systems, the developers of these trading programs will have to hand type each system with their respectable stations. So lets say there is only 150,000 stations with markets. Who is going to hand type all 150k?
Do we have a crowdsource method of collecting thousands of peoples hand typed additions or do we expect one person to do all of that?
It took me three days to hand type all 283 known systems with their stations. That was me having to go to each station in game to verify there was a market. Based on my current work it would take me 1,590 days to hand type all 150,000 stations. And that's without adding any market data to those stations. I would say double that time to get all the market data and by the time you finished, all of that data is no good.
For my own amusement I've developed a small bit of software for tracking my progress through the game, including computing coordinates of systems and hyperspace routes (as well as noting down any interesting facts about systems as I pass through).
I'd be happy to share my coordinates with people once I've got some (Have only tried it out on systems near i Bootis so far.). Is there a "shared" coordinate database anywhere that we could sync with? (Sorry I may have missed something in this thread).
I'd be happy to share my coordinates with people once I've got some (Have only tried it out on systems near i Bootis so far.). Is there a "shared" coordinate database anywhere that we could sync with? (Sorry I may have missed something in this thread).
ShadowGar
Banned
Not at the moment. None of the previous developers of trading programs has released any sort of sharing method that I'm aware of. And if they do have a method, its not publicly accessible. But your software would help our current projects if it can determine coords of unknown systems. Myself and others could then drive around the pill like a google mapping car and then take all of those coords and add them to what we already have.
With the 283 recorded systems, I'm missing coord data for 217 stars. Getting those coords would complete the routing issues we currently have. Right now Trade Dangerous can not find proper routes since there are usually 1-3 unknown stars between the known markets.
I can determine the coordinates of unknown systems fairly accurately I think - at least in the small bubble I have tested so far. Here are some numbers I got this evening, in case you can compare to the results you already have:
Code:
LP 271-25 -10.47117 31.83668 7.31337 user-trilat
Ross 52 -8.43655 29.15859 13.31257 user-trilat
OT Serpens -11.12709 30.33942 18.40620 user-trilat
DE Bootis -7.43827 32.62218 17.00016 user-trilat
LHS 399 -13.56303 36.15641 25.78137 user-trilat
LHS 396 -9.87540 30.84284 20.46919 user-trilat
Veren's Stop -12.97398 21.95947 20.27742 user-trilatMy program uses python and numpy because it makes doing the maths easier, so not sure if other people have that kind of setup? I run it on my laptop.
Feel free to download the code here, but be aware that I've only been working on it for a few hours so it's not really heavy in features:
https://www.dropbox.com/s/eqk910w9sz9ikpf/EDtool.zip?dl=0
Last edited:
a distance (in LY i guess) and a directional vector is all thats needed to plot a new 3d point...
I think that the simplest thing is to get 4 distances to known stars. Three distances gets you a single point in space if you are accurate with the distances - the fourth point is to cross check and estimate the error in your measurement. So far I've found the 3 decimal places of distance in the galaxy map sufficient.
It is fairly easy to do the maths to solve for the 3-d coordinates (though I'm an astrophysicist, so maybe it just seems easy to me!)
I picked 4 "arbitrary" reference stars (i bootis, CR draco, Styx and Wyrd) based on the fact that I could spell their names. Probably a better distribution of reference stars would be needed for a larger volume survey.
Alternative approach would be to sit in e.g. i bootis and write down the distances to many stars, then move to another system and write down lots of distances. I prefer the "go there and put it on the map" approach because it's more fun, even if it's not really as efficient.
DOnt know if this is any help but if you want rough figures.
Look top down on the galaxy map and the co-odinates are there
the squares are 10 wide 10 long 10 deep.
Zoom in and it tells you what you are on on the up/down axis
then when the star is exactly on that level, you can work out where it is on the forward/back left/right plane.
Not super accurate but its a good start
Look top down on the galaxy map and the co-odinates are there
the squares are 10 wide 10 long 10 deep.
Zoom in and it tells you what you are on on the up/down axis
then when the star is exactly on that level, you can work out where it is on the forward/back left/right plane.
Not super accurate but its a good start
ShadowGar
Banned
Just got it working, didnt have the imports your using. ;/ Anyways. It is pretty close.
For instance...
-Djedet actual coords, -81.46875, 30.5625, -1.1875
-Djedet by your program ,-81.41270684,29.28272097,-0.57162471
I wonder what we would get if we compared more than 4. Say, 10 places ;p
Having had a quick look at the numbers Brookes kindly provided us, I noticed that they all correspond to increments of 1/32th of a ly. That's something to consider when you're doing trilateration, especially margin of error and things like that.
There are other implications, such as it might be possible to store coordinates in the database as integers (corresponding to the ly coordinates multiplied by 32) which may help when you're wrangling thousands of systems.
There are other implications, such as it might be possible to store coordinates in the database as integers (corresponding to the ly coordinates multiplied by 32) which may help when you're wrangling thousands of systems.
ShadowGar
Banned
I see what you mean. But I wonder if his calculations are good enough just for jump routing. After looking at a few systems and using his program, at most his coords are off by such a slight amount.
Yeah, sorry - I didn't make it to be portable! I guess that the error increases quite a lot when you are far from my 4 reference points. E.g I was able to pinpoint Eranin very precisely (< 0.01Ly). I note that in your example the error is quite small in the x-direction, probably because from djedet all of the reference points appear to be pretty much in the x-direction, and therefore the angular uncertainty in y and z is large.
I expect that the location would be tied down a lot better by picking more sensible reference points.
Actually, I think it would be quite easy for the code to do this rough position estimate then find the best target(s) that would tie down the coordinates more precisely, enabling good positional accuracy everywhere.
I think that probably only at most 6 distances are required - 4 to tie down the rough position and then up to 2 more to refine it.
I'll update the code when I'm finished work and see if it does better for your example.
edit: I'll also pick a better initial set of reference stars for mapping the "pill".
Those four systems are rather close together (all near the middle of the B1 bubble), so you'll get less accurate results for any "interesting" systems further out. Since we have coordinates for all of the B2 *populated* systems, you could (and should) choose widely-spaced reference systems from that list.
For my part, I'm collecting measurements to certain well-known distant stars as well, so that we have a good reference basis when the bubble expands yet again. I've established fairly conclusively that Sol really is at 0,0,0, and I have eight of the "navigation stars" pinned down fairly well too. Sagittarius A* still has a large uncertainty, but considering its distance from the available references, that's not surprising (see above). Additional measurements might make it converge better.
I think I need to add one more point of input-validation to my own Python scripts, and then I can post them here. (Incidentally, I eventually caved and used the least-squares code; it's really slow, but it saves me some mental effort.) I post-validate the results for sanity; having noticed the 1/32 ly quantisation early on, I require the surviving distance measurements to fall within 1/100 ly of the calculated position. I should also try to estimate how much small errors in the distance measurements perturb the position.
Yes. The stumbling block will be extracting the procedurally generated stars. I know from reimplementing the FFE code in Java that the procedural generation code is quite sensitive. An off by one error in a lookup or a signed operation that should be unsigned and suddenly a star is completely different. It wouldn't be that hard for Michael, but I'm not surprised he doesn't have time for it right now.
I think you need all the stars in the inhabited area of the galaxy if you want to do route planning. Do we know how big the inhabited area is? It's obviously not the whole 400 billion. Even several thousand could be awkward to handle.
If you're not doing route planning then there is really no need to have any star that doesn't have a market. Though FD has talked about systems developing over time so the list would probably not be static.
My wild guess would be that we're unlikely to see more than about 100,000 inhabited star systems. Even that might be an order of magnitude too high, but I don't know how far out FD wants to scale the inhabited galaxy. Taking into account the uninhabited systems between and surrounding them, we can extrapolate a need to handle a maximum of about a million stars for route-finding.
That's a tractable problem. Not necessarily easy - at those scales it would be rather a good idea to switch to a faster language than Python - but tractable, in that we could tackle it using ordinary PCs and obtain optimal, complete results in a reasonable time.
Scaling up to a billion star systems would be genuinely difficult. Scaling up to 400 billion would be very impractical. But a million or so is not a problem.
That's a tractable problem. Not necessarily easy - at those scales it would be rather a good idea to switch to a faster language than Python - but tractable, in that we could tackle it using ordinary PCs and obtain optimal, complete results in a reasonable time.
Scaling up to a billion star systems would be genuinely difficult. Scaling up to 400 billion would be very impractical. But a million or so is not a problem.
Yes - I suspect it will be a while before we get to more than a few thousand systems.
Debugging and balancing markets and stuff is probably going to be an issue for FD for a while, so keeping it reasonably tight would make sense.
I guess you may be able to visit a lot of systems in the full version, but be constrained by habitable systems, fuel supplies and repairs and things will limit the range.
Debugging and balancing markets and stuff is probably going to be an issue for FD for a while, so keeping it reasonably tight would make sense.
I guess you may be able to visit a lot of systems in the full version, but be constrained by habitable systems, fuel supplies and repairs and things will limit the range.
The easiest way to get a systems coord is to visit it, open the galaxy map, and get distance to 4 other known systems. Then do trilateration.
As for hand typing, that is something that have to be shared among 1490 players, not 1 player typing for 1490 days.
I'm still hand typing the unknown in the "short side" of the current bubble (as seen from the beta 1 systems), but post the ones your missing and I'll fill them in as soon as I visit the systems.
As for hand typing, that is something that have to be shared among 1490 players, not 1 player typing for 1490 days.
I'm still hand typing the unknown in the "short side" of the current bubble (as seen from the beta 1 systems), but post the ones your missing and I'll fill them in as soon as I visit the systems.
And we will also be constrained by available data. I suspect that we won't really be able to ask for a tailored list of a million systems' coordinates. The download size would not be prohibitive, mind, but the difficulty of specifying *which* million we want would be more problematic.
We should probably ask for the initial set of inhabited systems, as we have now. That would be an excellent set of anchors to build everything around.
However, a crowdsourcing tool (like the beta1 market dumps) would make *that* problem tractable, and would scale naturally to the regions of space that players actually explore.
All we need for that is an API which provides usable information; the distances to the adjacent jump-reachable systems would be a reasonable start (and should be available without any special action by the player), while the distances to more distant systems might require the player to open the galaxy map (which he's likely to do anyway if he's exploring).
Such a tool could also be self-validating, in theory. Other players could be used to confirm the range to newly-discovered systems from their present locations.
We should probably ask for the initial set of inhabited systems, as we have now. That would be an excellent set of anchors to build everything around.
However, a crowdsourcing tool (like the beta1 market dumps) would make *that* problem tractable, and would scale naturally to the regions of space that players actually explore.
All we need for that is an API which provides usable information; the distances to the adjacent jump-reachable systems would be a reasonable start (and should be available without any special action by the player), while the distances to more distant systems might require the player to open the galaxy map (which he's likely to do anyway if he's exploring).
Such a tool could also be self-validating, in theory. Other players could be used to confirm the range to newly-discovered systems from their present locations.