Discussion What is the most efficient way to crowdsource the 3D system coordinates

TGC Alpha 0.3 is done - It really deserves more than just a 0.1 increment :cool:

A bunch of extra goodies added to the search filters.
And yes "search filters" is now a thing.
As all those filters have now been grouped together in an appropriately named "filter" object (this is a breaking change)

A sample search with all the boxes ticked could look like
Code:
var query= {
  data: {
    ver:2, 
    outputmode:1, 
    filter:{
        knownstatus:0,
        systemname: "sol",
        cr:5,
        date:"2014-09-18 12:34:56",
        coordcube: [[-10,10],[-10,10],[-10,10]],
        coordsphere: {radius: 123.45, origin: [10,20,30]}
    }
  }
};

Also - All the filters are now available for both GetSystems and GetDistances.
And they can be mixed and matched as you please.

The newly added extra filters are:
- systemname
- coordcube
- coordsphere

They should be nearly self-explantory.

If not - Heres the apidoc link: http://edstarcoordinator.com/api.html
 
I was working with paper and pencil when i came to my solution :)

edit: bold: correted your solution it has to be DMM^2

As was I - Did it a few times as it didn't match yours.
Then wen't online to see if there wasn't a solver that could tell me what the problem was.


DMM^2 - yeah that's a "typo" that it isn't there.
 
Beta3.04 234 valid Systems out of the Beta 2 File. Check File: Beta3.04FixPointSystems.txt in my DropBox
~
Calculated Best 4 Fix Ref-Systems (placed at first position in the File):
~
PARCAE;-8.125;55.09375;-17
HIP 7338;-96.5;33.21875;-68.65625
LHS 3549;-29.625;6.125;-1.9375
AO QIN;-62.90625;46.78125;1.53125
~
Simulated Results: (The 230 simulated as entered with distances to the 4 Ref systems rounded to 2 digits after the point)
~
The Distance of the calculated position to the exact position in the File:
Average Distance Error 0.005638
MinDistanceError 0.000521
MaxDistanceError 0.016730
~
Difference between the simulated entered Distance to the Ref (rounded 2 digi after point) and the calculated one from the calulated position.
Average Distance Match Error: 0.0018824
MinDistanceMatchError : 0.0000097
MaxDistanceMatchError : 0.0112492
~
This means for Systems WITHIN the Space Range of this 234 Systems I expect the above accurracy when entered the in ED displayed distances
of players current new System to those 4 selected Ref Systems with my calculation method.
~
If it's really so in real..I don't know.. it is not to exculde that I made a mistake somewhere.
It took my laptop almost 2 days brute force to calculate the best 4 ref systems. :)
~
end of simulation :)
 
Parsing the netlog to get uncharted star systems is certainly doable assuming the commanders wish to participate in this way and of course FDEV have no objections to data collection in this way. However the filename is no longer netLog.log, from looking in my logs folder it appears to have changed to netLog.timestamp.01.log thereby subsequent restarts will have a different filename as the timestamp will have progressed.

Also the timestamp appears to be off by a couple of months. My netLog created today is named netLog.1411091647.01.log, that timestamp would put the date at Fri, 19 Sep 2014 01:54:07 GMT. Anyway regardless you could search the directory for the most recently updated file named netLog*.log and access that.

Just FYI, in case you missed it, Michael Brookes had mentioned something about not relying on the netLog for data. I believe the implication was that it would be going away. But there's no reason we can't use it for now.
 
Calculated Best 4 Fix Ref-Systems
Best fix systems - Are not valid for very long.
As distances increase to the fix systems (as our livingspace expands) the accuracy goes down.

For a given set - Sure it might be worth to find them.
But we don't have a fixed set (in the future).


The Distance of the calculated position to the exact position in the File:
Average Distance Error 0.005638
MinDistanceError 0.000521
MaxDistanceError 0.016730
~
Difference between the simulated entered Distance to the Ref (rounded 2 digi after point) and the calculated one from the calulated position.
Average Distance Match Error: 0.0018824
MinDistanceMatchError : 0.0000097
MaxDistanceMatchError : 0.0112492
That ought to be plenty of accuracy. Only 1/32th LY accuracy is needed (and even 0.0112492 is well below that)

Try running it again, but this time round your candidates to the 1/32th grid.

And then run the Match error again.
Pretty good chance you'll nail all of them exactly I think.
 

Harbinger

Volunteer Moderator
Just FYI, in case you missed it, Michael Brookes had mentioned something about not relying on the netLog for data. I believe the implication was that it would be going away. But there's no reason we can't use it for now.

Yeah, I'm aware of that. Purely answering wolverine2710's query on whether or not it can be done currently.

I can see how the log-files can be used to gather new uncharted star names

This is the only thing we could do with it, if the commander passes through a system which is uncharted we could log the name so that a volunteer could go and get the distance information. It may help to find overlooked systems but I'm pretty sure it's usefulness will end there.
 
How to waste time --- get into OCR. That list of systems on left HUD. So tempting to mine from screenshot. It might be the single biggest timesink when searching for new systems. Got as far as actually tracing each individual letter, but none of the free OCR tools like the slightly rotated letters. If someone want to have a go at recognizing 10x10 pixel slightly distorted but perfectly human readable characters in javascript - I've got all the code for getting each char in browser without going to server.

Look up "planar homography" - you can use that to remove the perspective transform. I've used it to take a camera image of a tabletop and transform it into a true top-down view. The simplest version requires the user to specify 4 points that should be in a rectangle. In this case the corners of the holographic display. Trickier would be to do some image analysis and determine the corners automatically. Anyway, see if your OCR software likes this image any better... I just used my test harness so there is some unnecessary scaling going on but I suspect it'll be undistorted enough for your purposes.
skj4NQZ.png
 


This is where I'm at. I think its most to gain during the bw filter to look more at the orange components then at the average on each pixel. A learning network should be able to do the numbers pretty accurate, not so sure about the letter.
 
Best fix systems - Are not valid for very long.
As distances increase to the fix systems (as our livingspace expands) the accuracy goes down.

For a given set - Sure it might be worth to find them.
But we don't have a fixed set (in the future).



That ought to be plenty of accuracy. Only 1/32th LY accuracy is needed (and even 0.0112492 is well below that)

Try running it again, but this time round your candidates to the 1/32th grid.

And then run the Match error again.
Pretty good chance you'll nail all of them exactly I think.

Yes if players go deeper into space somewhen 4 better fixes should be evaluated. However that requires that you have precise coordinates form systems out there. You can calulate them with those 4 but you don't have it garantet acurrate. So as farther out you go and as more second generation fix points you take the more uncertain everything will become. The grid lock to 1/32 and compareing the distances to a lot of systems for a new fix out ther will help.
~
Byway I haven't checked those 4 fix systems position within ED if they really still at the exact positon like we have from the file. My invalide filtering was only done with one fix reference. That's not 100% sufficient and a check in ED (alldistances of the 4 fix systems to each other) would have to take place first before useing.
Also this check has to be repeated for every new ED release.
~
With:
CalulatedX = Math.Round(32.0 * CalulatedX) / 32;
CalulatedY = Math.Round(32.0 * CalulatedY) / 32;
CalulatedZ = Math.Round(32.0 * CalulatedZ) / 32;
~
I get this:
Average Distance Error 0.0
MinDistanceError 0.0
MaxDistanceError 0.0
~
Average Distance Match Error: 0.0024294
MinDistanceMatchError : 0.00000049071
MaxDistanceMatchError : 0.00499461
~
This means I got indead the exact position for ever of the 230 Systems if I grid lock them to 1/32 useing the 4 fix references.
The Match Error might look confusing at first but is correct.
The Match Error is to expect to be +/-0.005 since the entered Distances are rounded to 2 digi after the decimal point
(I used the distance Error (= absolut error value) for the Match which emans -0.005 abecomes +0.005 and the average will be the half of that
~
I quickly did an additional simulation by adding +1 for the entered Distances to the RefSystem1 and got this:
Average Distance Error 1.08
MinDistanceError 0.211
MaxDistanceError 2.58
~
Average Distance Match Error: 0.4775
MinDistanceMatchError : 0.00008375
MaxDistanceMatchError : 2.44034
~
That says it is possible to detect faulty distance inputs with the match error. However it's not possible to say which of the 4 distance is the wrong one.
Forexample I saw when I debugging that this min error is from system 117 and infact to RefSystem1 ! the one we manipulated. However the distances to the other refsystem then were way of. So you just have to check every of the 4 distances and if one is off the whole input is invalid.
~
Edit to add: might have to clarify one point but you can read it from the code too: What I call distance error is in truth the position error. I called it distance error because I calculate the distance of expected position (from file) to calculated position.
And the Distance Match Error is the difference between the entered distance (the point 2 digi) and the distance we can calculate with the calculated position to the ref system.
 
Last edited:

wolverine2710

Tutorial & Guide Writer
@Tornsouls. Finally having a look at EDSC. Done a weebit of testing and so far no errors.
In fact EDSC after a small period on the rack is spinning like a cat instead of screaming like a pig ;-)
A few things I found so far and/or questions about GetSystems.

  1. commandercreate and commanderUpdate always seems to be empty.
  2. commanderUpdate. When exactly is this set/changed.
  3. A resultscount which returns numbers of systems would be nice. Especially for testing and I think in general. Otherwise a tool author has to loop to get this value.
  4. From your ealier post I knew I had to set ver:0 otherwise an error occurs.
  5. Zlib or another compression type as input parameter. That would reduce the bandwith greatly.

More tomorrow.
 
  1. commandercreate and commanderUpdate always seems to be empty.
  2. commanderUpdate. When exactly is this set/changed.
  3. A resultscount which returns numbers of systems would be nice. Especially for testing and I think in general. Otherwise a tool author has to loop to get this value.
  4. From your ealier post I knew I had to set ver:0 otherwise an error occurs.
  5. Zlib or another compression type as input parameter. That would reduce the bandwith greatly.

1: Because those are the systems I seeded the DB with (FD data)
There is in fact one system that has non-empty commandercreate/update - See if you can find it :D

2: Every time cr is updated.
Which is every time a distance/name/coordinates are confirmed by someone else.

3: You don't have to loop anything - systems.length (javascript) will tell you - it's an array afteral.
Can't imagine any languages out there without a length/count feature for arrays?

4: Hopefully there isn't an actual *error* - But a proper result returned containing a message informing you version number is wrong?

5: The server is set to accept requests for compressed responses (both on static and dynamic content).
So it's up to whatever client you are using to set the correct headers for that to occur.

---

EDSC itself (Now just an input form - as TGC has taken over the trilation/storage part) - is actually not operational as such atm.
It's actually currently running of another database (with beta2 data) - To not mess with TGC.

I'm working on changing that though.

In a day or two - It should be fully operational again (with some extra bits as well), and hooked up to TGC itself.


And once I'm done with that... I'll have a look at storing (static) system/station data in the TGC as well.
--

Thanks for testing!
 
Last edited:
Hey all,

Updated my app a bit, Solar Systems grid now includes a black market column whose value is derived from the black market values of the stations in the system. For example, if any station in the system has a black market, the system is flagged as having a black market.

The add/edit dialog of the Neighbours grid in system details includes an autocomplete text box that searches system names as you type.

http://elitesystems.azurewebsites.net/
 

wolverine2710

Tutorial & Guide Writer
Hey all,

Updated my app a bit, Solar Systems grid now includes a black market column whose value is derived from the black market values of the stations in the system. For example, if any station in the system has a black market, the system is flagged as having a black market.

The add/edit dialog of the Neighbours grid in system details includes an autocomplete text box that searches system names as you type.

http://elitesystems.azurewebsites.net/


Looks very nice. I've just added a note for a system and saved it - worked. Later removed the note - worked. I also like the API very much. This tool can be very handy to input stations and there distances etc. As in uploading them to TGR once Tornsouls has support for that. I mentioned this before I believe. Your tool presents itself as a spreadsheet. It would be great if common windows command like page up, page, home, going to the bottom could be implemented. Using the buttons for me is less intuitive.

One thing to consider. Atm I know of two API's for getting info and uploading system/station data. I know that thrudd is considering an API for downloading and uploading prices info. Its entirely possible it will be possible that we can download system, station info as well. Haven't had time to compare to compare yours and Torunsouls one. Having API's is GREAT but we should strive to minimize to number of API's and its json format. It looks as if we more or less decided that TGC is TOR (the one reference for all data except for prices) and other tools can upload data to it and retrieve data from it. Does this mean that TGC has the best API and JSON format and others should follow that? I can't say. I'm pretty sure every API, JSON format has it advantages and disadvantages. As TGC is still in Alpha and yours seem to Alpha/Beta as well perhaps NOW is the time to start to look into the API and its JSON format. Iirc you use POST and GET while Tornsouls is only using POST - one reason, better support from within .Net/c#). Perhaps you and Tornsouls can work together, combine forces and see if perhaps a unified API or at least a unified JSON format can be made. If third party tools take off and start to use your or Tornsouls data I can imagine those authors would like to see one or two API's instead of multiple ones - as in 5+. If there is a good base I think other authors MIGHT also consider using that as it would mean interoperability is MUCH better.

Again. I can't and won't judge as this point which is better but I DO know that with regard to API's and JSON formats LESS IS MORE.
 
Last edited:
Looks very nice. I've just added a note for a system and saved it - worked. Later removed the note - worked. I also like the API very much. This tool can be very handy to input stations and there distances etc. As in uploading them to TGR once Tornsouls has support for that. I mentioned this before I believe. Your tool presents itself as a spreadsheet. It would be great if common windows command like page up, page, home, going to the bottom could be implemented. Using the buttons for me is less intuitive.

One thing to consider. Atm I know of two API's for getting info and uploading system/station data. I know that thrudd is considering an API for downloading and uploading prices info. Its entirely possible it will be possible that we can download system, station info as well. Haven't had time to compare to compare yours and Torunsouls one. Having API's is GREAT but we should strive to minimize to number of API's and its json format. It looks as if we more or less decided that TGC is TOR (the one reference for all data except for prices) and other tools can upload data to it and retrieve data from it. Does this mean that TGC has the best API and JSON format and others should follow that? I can't say. I'm pretty sure every API, JSON format has it advantages and disadvantages. As TGC is still in Alpha and yours seem to Alpha/Beta as well perhaps NOW is the time to start to look into the API and its JSON format. Iirc you use POST and GET while Tornsouls is only using POST - one reason, better support from within .Net/c#). Perhaps you and Tornsouls can work together, combine forces and see if perhaps a unified API or at least a unified JSON format can be made. If third party tools take off and start to use your or Tornsouls data I can imagine those authors would like to see one or two API's instead of multiple ones - as in 5+. If there is a good base I think other authors MIGHT also consider using that as it would mean interoperability is MUCH better.

Again. I can't and won't judge as this point which is better but I DO know that with regard to API's and JSON formats LESS IS MORE.

I completely agree, having multiple APIs defeats the purpose of a single repository of data. If TGC is going to provide all possible data entities an application might want (systems, ships, equipment, stars, planets etc.) then I can switch my repository layer to call that API instead of my backend database, if TGC is only going to provide a subset then we would need a discussion into which API provides what. In the meantime I can make my JSON format mimic TGC to keep things simple.

I want to flesh out the service over the next week or so to allow the API to be used for adding/updating station and system details but I need to have a think about authentication first. The intention will be that it is RESTful, so you POST a new system, PUT an updated system and DELETE to well, delete a system.

I did PM TornSoul to ask him if he thought there was a way forward and to express the fact that I didn't want to duplicate his efforts but he hasn't replied yet. I'm more than willing to collaborate on a solution though.
 

wolverine2710

Tutorial & Guide Writer
I completely agree, having multiple APIs defeats the purpose of a single repository of data. If TGC is going to provide all possible data entities an application might want (systems, ships, equipment, stars, planets etc.) then I can switch my repository layer to call that API instead of my backend database, if TGC is only going to provide a subset then we would need a discussion into which API provides what. In the meantime I can make my JSON format mimic TGC to keep things simple.

I want to flesh out the service over the next week or so to allow the API to be used for adding/updating station and system details but I need to have a think about authentication first. The intention will be that it is RESTful, so you POST a new system, PUT an updated system and DELETE to well, delete a system.

I did PM TornSoul to ask him if he thought there was a way forward and to express the fact that I didn't want to duplicate his efforts but he hasn't replied yet. I'm more than willing to collaborate on a solution though.

Sounds like a plan. Timezones, and working times and a whole lot of other reasons could be the reason he has not been able to respond (yet).

As your tool is web based would OAuth or OpenID not be an option for you. From the wiki: OAuth is a service that is complementary to, and therefore distinct from, OpenID. A few links: OAuth wiki, OpenID, OAuth 2.0. Haven't used it myself so don't know how it exactly works. Basically to be a deterrent it should be possible to determine the account/username of the person logging in....

A very long shot but still worth a PM to Michael Brookes. Perhaps the forum login or even better the login of the Elite Dangerous store (where you have to register your ED) could be used for authentication. If so and then you or other third party ED tools could use that. It would for sure be a deterrent for pranksters. OAuth where users can login with for example their google account would be a deterrent as well. Just thinking out loud.....
 
Yeah he seems like he's really busy at the moment so I didn't want to bug him :)

For now I'll just mimic his JSON structure and we can take it from there.

Yeah I'll look into OAuth. Microsoft provides a nice walkthrough for implementing OAuth in MVC applications so I'll take a look at doing that, I wouldn't want to implement a system of my own from scratch, I'd like to actually play Elite from time to time!
 

wolverine2710

Tutorial & Guide Writer
Yeah he seems like he's really busy at the moment so I didn't want to bug him :)

For now I'll just mimic his JSON structure and we can take it from there.

Yeah I'll look into OAuth. Microsoft provides a nice walkthrough for implementing OAuth in MVC applications so I'll take a look at doing that, I wouldn't want to implement a system of my own from scratch, I'd like to actually play Elite from time to time!

Playing ED, what exactly is that again ;-).
Haven't played in the last 3++ weeks, away a lot and these forums/threads are so addictive. Will be away again from Friday till Tuesday and where I am I can't play ED. Perhaps its better so, not burned out for when Gamma hits next week Friday.....
 
Last edited:

Michael Brookes

Game Director
Sounds like a plan. Timezones, and working times and a whole lot of other reasons could be the reason he has not been able to respond (yet).

As your tool is web based would OAuth or OpenID not be an option for you. From the wiki: OAuth is a service that is complementary to, and therefore distinct from, OpenID. A few links: OAuth wiki, OpenID, OAuth 2.0. Haven't used it myself so don't know how it exactly works. Basically to be a deterrent it should be possible to determine the account/username of the person logging in....

A very long shot but still worth a PM to Michael Brookes. Perhaps the forum login or even better the login of the Elite Dangerous store (where you have to register your ED) could be used for authentication. If so and then you or other third party ED tools could use that. It would for sure be a deterrent for pranksters. OAuth where users can login with for example their google account would be a deterrent as well. Just thinking out loud.....

This isn't something we'll be able to look at any time soon I'm afraid.

Michael
 
Back
Top Bottom