Discussion What is the most efficient way to crowdsource the 3D system coordinates

[RedWizzard]

Fixed.

And it was indeed the null coords that was the issue.
I'd missed *one* spot where I was converting the coords (hope there isn't more lurking...)

Yep, all good here now.

The null coordinates: I've been in great doubt about how to handle (display/transmit) that actually..
I've settled on just putting null there, as that's what's in the DB - and at least JavaScript will handle that just nicely.
It seems to be most consistent - although it ain't so pleasing on the eyes. Which really shouldn't matter one bit (still annoys me though )

Nulls works for me. I guess the alternative would be to leave out the coordinate array entirely.

 

[TornSoul]

Nulls works for me. I guess the alternative would be to leave out the coordinate array entirely.

Yeah I had that thought as well - Turns out that will be impossible to do, without a MAJOR change in the code (so no! )

Turns out you can't have conditional fields in an object inside a lambda expression in c#.

Which sorta makes sense, considering the (in all other cases VERY convenient) type checking that is performed with lambdas.


It *would* be neat I think (to remove coords entirely on nulls) - But then again, not *quite* consistent, as you'd then actually have to check if the coords are present, before passing it on to a constructor or similar.

One other option would be to set coord to null - instead of an array of nulls.
But that again sorta runs into the consistency issue - Not as severe but still...

 

[TornSoul]

Something else:

Filtering.

I'm going to add the option to filter on starnames (so you only pull anything containing "era" or whatever).

And at some time in the future, I'm sure it'll be necessary to filter on "range of coordinates" (ie. to get every star/distance in say a 20ly cube).

And we can already filter on cr and date.

It starts to become a whole bunch of filter options...

So.. The thought is, if I should bite the bullet now, and put all of those into a structure, instead of adding them one by one by one....

Ie. perhaps something akind to

var query = {
  ver:0,  
  outputmode:2, 
  filter: {
    systemname: "era",
    cr:1, 
    date:"2014-09-18 12:34:56",
    coord: [[-10,10],[-10,10],[-10,10]]
  }  
 }

Thoughts?

 

[RedWizzard]

Filtering.

I'm going to add the option to filter on starnames (so you only pull anything containing "era" or whatever).

And at some time in the future, I'm sure it'll be necessary to filter on "range of coordinates" (ie. to get every star/distance in say a 20ly cube).

So.. The thought is, if I should bite the bullet now, and put all of those into a structure, instead of adding them one by one by one....

Thoughts?

Probably worth doing pretty soon. I'm expecting to switch to loading reference stars and names for autocomplete on demand, as bringing in all data clearly won't scale. Filtering by volume is also high on my list of desired features. Ideally it'd be nice to have two options for specifying the volume: as a range of coordinates (as you've done in your example) and also as an origin and distance "give me all stars within 25 Ly of [x,y,z]".

 

[TornSoul]

Alright I'll do that then.

So Alpha 0.3 will have breaking changes (what Alpha's are for )

Radius search thing - good idea.

So something like

var query = {
  ver:0,  
  outputmode:2, 
  filter: {
    systemname: "era",
    cr:1, 
    date:"2014-09-18 12:34:56",
    coordcube: [[-10,10],[-10,10],[-10,10]]
    coorddist: {dist:25, origin:[10,20,30]}
  }  
 }

Is probably how it's going to end up looking like for alpha 0.3

 

[TargetLost]

That helps a ton - Thanks!
I still have no idea what eq1 represents though
Ie. why is that formula correct, what does it "do" (is it adding vectors or...)

The rest I can follow.

~
It's simple phytagoras.
~
The 3 points: M1,M2,P build a triangle. M1M2 is the base line and the blue line is the height of the triangle.
~
~
.........P
.........|
M2 ------------ M1
........Pk
~
We can split it in two rectangle triangles: M2,Pk,P and M1,Pk,P
Both rectangle triangles have the same hight PkP so we calulate PkP for the left triangle and PkP for the right triangle and set this equal.
left:
a^2+b^2=c^2, for the left side is: M2Pk^2+PkP^2 = DM2^2
for the right side: M1Pk^2+PkP^2 = DM1^2
~
for Pk we know it's somewhere on the line given by M1M2 and the position is what we would like to know.
makeing it with a factor k we just have to care about one unknown only solvable with one equation eq1
~
Set PkP^2 left == PkP^2 right and you get eq1
~

Probably true

Any idea if this is resistant to wrongly reported distances?

If not, perhaps (as JesusFreke has been suggesting for the other method) leave out 1 pair, and then try for each such combination (as one should then be wrong, and thus be identifiable)

~
You will always get a specific position with which you can calulate back the distances to the reference systems and compare with what was entered.
In case wrong distances were entered you should get notable differences (error) although I haven't tested taht.
With additional references (take 5 instead 4) you get better fail entry detection.
~
Note this difference: It requires 4 good references to get a definitve position. With 3 only you theoretical get a line for the possible positions. (different to the trilateration where you get 2 possible possition with 3 references)
If A*x = b is solveable you get a specific position. If it is not due to lack of information your math lib should report unsolvable error and you know.
You can fix it by adding one more ref system or check the 4 ref system for distinct enough postions to each other.
It's rather unlikely you have that case. When you pregive the 4 refsystems to use then you have ruled this out anyway.




~

That last bit I'm not sure how to handle (by hand) either

With more than 4 reference systems you are in fact getting an over determined system as well, which needs to be handled.

~
I see you are not that familiar with the least square approach. That methodic is something very very usefully.
http://en.wikipedia.org/wiki/Least_squares
~
The Err formula (Err is actually a sum of i planes. But of course I only solved it for 1 Element/plane.
For building the Matrix A you have to sum them all up!!
You don't solve for each plane one after the other. You put every single plane into A and b and then you right away get your result P(X,Y,Z) and finish.
Maybe I can give you some code example so you will see.
Problem is a bit..I used some code that I might not distribute.
~
Edit to add links for solveing a linear eq sets (don't think you need them thought but might be helpfull for the one or the other)
A*x = b
http://en.wikipedia.org/wiki/System_of_linear_equations
http://en.wikipedia.org/wiki/Gaussian_elimination
3*3 you can probabily well code out directly too.

 

[Snuble]

How to waste time --- get into OCR. That list of systems on left HUD. So tempting to mine from screenshot. It might be the single biggest timesink when searching for new systems. Got as far as actually tracing each individual letter, but none of the free OCR tools like the slightly rotated letters. If someone want to have a go at recognizing 10x10 pixel slightly distorted but perfectly human readable characters in javascript - I've got all the code for getting each char in browser without going to server.

 

[TornSoul]

~
It's simple phytagoras.
~
The 3 points: M1,M2,P build a triangle. M1M2 is the base line and the blue line is the height of the triangle.
~
~
.........P
.........|
M2 ------------ M1
........Pk
~
We can split it in two rectangle triangles: M2,Pk,P and M1,Pk,P
Both rectangle triangles have the same hight PkP so we calulate PkP for the left triangle and PkP for the right triangle

Aha - So that's what your'e doing with eq1.
From just looking at it that wasn't at all obvious (to me)

Also - In your picture, the formula for k is slightly wrong.
It should be: k = 0.5 + (...) (not 0.5 - (...))

Unless you are doing something else not relating to modifying eq1.

I rather like the "trick" with k btw.

EDIT: On 2nd thought, now that I know what eq1 is all about (finding Pk).
Wouldn't *any* point on DMM do just as well (very easy to find the equation for the line between M1/M2 - and just pick a point from there)

Why does Pk have to be the one at the corner of the right triangles?

EDIT EDIT : Answering myself - To ensure the plane created goes through P ofc.

--

And I am in fact familiar with the least squares method and it's uses.

Never actually had to use it for anything though afair

 

[TargetLost]

Aha - So that's what your'e doing with eq1.
From just looking at it that wasn't at all obvious (to me)

Also - In your picture, the formula for k is slightly wrong.
It should be: k = 0.5 + (...) (not 0.5 - (...))

Unless you are doing something else not relating to modifying eq1.

--

And I am in fact familiar with the least squares method and it's uses.

Never actually had to use it for anything though afair

~
It should be minus. That's how I have used it in Maxima
k : 0.5 - (DM2*DM2 - DM1*DM1) / (2*DMM*DMM);
k is zero when P is exactly on M1 (DM2 = DMM and DM1 = 0)
and it's 1 when P is exactly on M2 (DM2=0 and DM1 = DMM)
Might be i have done something wrong in the drawings because I made them yesterday in a hurry.
~
I have uploaded slightly modified C# code that I can distribute. So you can see how it's all implemented.
~
That's where it starts:
public double Calculate(List<EDSystem> iEDSystems, int iFixPoints)
~
I give him a list of all the known good 230 systems (Beta2 file) and with iFixPoint I define the number of FixPoints to use (normaly 4)
That mean he will take the first 4 in the list as the reference.
Then I simulate the others like they were entered with ED-Distacnes (2 digi) toward the 4 fix references.
I store all that data into a Matrix and a Vector and pass it to my solveing method which I a bit unluckly named Gauss too
-> Gauss(vParameterVect, vXpYp, vConstants);
private ERes Gauss(EVector oParameters, EMatrix iKnown, EVector iConst)
~
vConstants (iConst) contains the coordinates of the 4 fix systems. 1 dim vector: [ref1x,ref1y,ref1z,ref2x,ref2y,ref3z ..... ]
vXpYp (iKnown) is a distance matrix N*N of all Systems versus all Systems. In there I stored the 2 digi rounded distance from the systems of interest to the FixSystems and pick them out form there. I made it like that for Newton where I wanted to find solutions with even unknown to unknown system distances. A distance value of 0 is the marking to skip that.
vParaneters (oParameters) is the output. For each simulated unknown system the 3 calculated cooords. Its a 1 dim vector containing this chain: [x1,y1,z1,x2,y2,z2...]
Ignore all the other stuff in this class (newton and test). It's not used.

 

[TornSoul]

Might be i have done something wrong in the drawings because I made them yesterday in a hurry.

Yup - It's actually more than just the x/- sign which I missed at first.

If I solve eq1 from the drawing I get

k = (DM1^2 - DM2^2)/(2*DMM) + 0.5

Notice how DM1 and DM2 are reversed.

-----
a = DM2 (yes DM2 not DM1)
b = DM1
c = DMM
x = k

Solve for x

 

[Harbinger]

So any of the other system data: Star Type, Economy, etc., is not being maintained specifically by anyone in this mapping group?

I've been submitting allegiance, economy, government, population, security and station data (not star types) as I visit populated systems to RedWizzard's systems.json which is what we consider our central repository. You can see an example of this in the 4th system listed in systems.json:

Spoiler

  {
    "name": "HR 8474",
    "x": -97.96875,
    "y": 21.3125,
    "z": -36.125,
    "contributor": "FD",
    "region": "Beta2",
    "contributed": "2014-10-21T17:16:31.000Z",
    "allegiance": "Federation",
    "economy": "Refinery",
    "government": "Democracy",
    "population": 263055,
    "security": "Medium",
    "stations": [
      {
        "station": "Eisele Dock",
        "distance": 190,
        "type": "Civilian Outpost",
        "faction": "Green Party of HR 8474",
        "commodities_market": true,
        "black_market": true,
        "shipyard": false,
        "outfitting": true,
        "largest_pad": "medium"
      },
      {
        "station": "Clark Point",
        "distance": 326,
        "type": "Industrial Outpost",
        "faction": "HR 8474 Solutions",
        "commodities_market": false,
        "black_market": false,
        "shipyard": false,
        "outfitting": false,
        "largest_pad": "medium"
      },
      {
        "station": "Galvani Landing",
        "distance": 326,
        "type": "Industrial Outpost",
        "faction": "Traditional HR 8474 Order",
        "commodities_market": false,
        "black_market": false,
        "shipyard": false,
        "outfitting": false,
        "largest_pad": "medium"
      }
    ]
  }

With a galaxy regeneration likely to come at the Preview Build there's very little point dedicating much time to this right now though as it could all change. I'm primarily focussing on charting stars that are based off real star data so very unlikely to change positions in the next build.

 

[Athan]

How to waste time --- get into OCR. That list of systems on left HUD. So tempting to mine from screenshot. It might be the single biggest timesink when searching for new systems. Got as far as actually tracing each individual letter, but none of the free OCR tools like the slightly rotated letters. If someone want to have a go at recognizing 10x10 pixel slightly distorted but perfectly human readable characters in javascript - I've got all the code for getting each char in browser without going to server.

It might be better to find a transform to 're-square' the panel before attempting OCR. I can play around with GIMP and Tools > Transforms > Perspective and get something rough, but really you'd want it automated and quickly repeatable.

 

[wolverine2710]

It might be better to find a transform to 're-square' the panel before attempting OCR. I can play around with GIMP and Tools > Transforms > Perspective and get something rough, but really you'd want it automated and quickly repeatable.

You might want to have a look at ImageMagick. Its a command line toolbox with zillion possibilities. And in the OCR thread someone also used a tool/program to change the perspective. Now with Sb2/SB3 the UI is flat for the market but this was not so with earlier builds. Also ImageMagick has LOTS of tools to sharpen/enhance/convert to grey etc etc. This can also make a difference with OCR-ing.

 

[wolverine2710]

Something completely different. I had this plan for a captains log, post #1200. This could be used by commanders who don't want to crowd source but in effect supply us for free a list of star systems out there. I have posted a captains log request in Slopeys BPC thread and PM-ed him. The BPC already uses the netlog.log and parses it to get the current system in the BPC. Perhaps he's willing/able to have an extra tab in the BPC for a simplified captains log which uploads newly found systems to TGC.

 

[Harbinger]

Parsing the netlog to get uncharted star systems is certainly doable assuming the commanders wish to participate in this way and of course FDEV have no objections to data collection in this way. However the filename is no longer netLog.log, from looking in my logs folder it appears to have changed to netLog.timestamp.01.log thereby subsequent restarts will have a different filename as the timestamp will have progressed.

Also the timestamp appears to be off by a couple of months. My netLog created today is named netLog.1411091647.01.log, that timestamp would put the date at Fri, 19 Sep 2014 01:54:07 GMT. Anyway regardless you could search the directory for the most recently updated file named netLog*.log and access that.

 

[Athan]

Parsing the netlog to get uncharted star systems is certainly doable assuming the commanders wish to participate in this way and of course FDEV have no objections to data collection in this way. However the filename is no longer netLog.log, from looking in my logs folder it appears to have changed to netLog.timestamp.01.log thereby subsequent restarts will have a different filename as the timestamp will have progressed.

Also the timestamp appears to be off by a couple of months. My netLog created today is named netLog.1411091647.01.log, that timestamp would put the date at Fri, 19 Sep 2014 01:54:07 GMT. Anyway regardless you could search the directory for the most recently updated file named netLog*.log and access that.

No, it's in YYMMDD order, shame they dropped the 'CC' then it would be very close to ISO 8601 format, the One True Date/Time format.

Also note the extra ".01". Last night during a single session I had both a .01. and a .02. netLog file created (the latter having a different timestamp in the name too), so it's no longer even sufficient to watch the one file.

 

[Harbinger]

Ahh didn't notice, it just looked so close to current unix timestamps. I see it now YYMMDDhhmm

The 01, 02 part isn't really that important as long as only one file is being modified (If it's modifying more than one it could be problematic). You just check the last modified attribute of the file and pick the filename that was updated most recently.

 

[TargetLost]

Yup - It's actually more than just the x/- sign which I missed at first.

If I solve eq1 from the drawing I get

k = (DM1^2 - DM2^2)/(2*DMM) + 0.5

Notice how DM1 and DM2 are reversed.

-----
a = DM2 (yes DM2 not DM1)
b = DM1
c = DMM
x = k

Solve for x

Ah I see.
That's exactly the same solution you have got just written different.

You can simple check it with wxMaxima:
~
original:
eq1: DM2^2 - ((1-k)*DMM)^2 = DM1^2 - (k*DMM)^2;
solve(eq1,k);
~
my solution:
ratsimp(0.5-(DM2*DM2-DM1*DM1)/(2*DMM*DMM));
~
your solution:
ratsimp((DM1^2 - DM2^2)/(2*DMM*DMM) + 0.5);
~
each of it will output:
(DMM^2-DM2^2+DM1^2)/(2*DMM^2)

for k. It's just a matter of taste or how you solved it. I was working with paper and pencil when i came to my solution

edit: bold: correted your solution it has to be DMM^2

 

[Snuble]

I dont want it uploaded to server. I'm very familiar to Imagemagick, but using a server-client configuration come with a cost. A client only approach can run from wherever. Be it a cheap host or from a local folder.

Think I'll have a look at the deep learning networks and see if they produce better results. The toolbox is html5/canvas/javascript. It does not support native perspective transformation (can't change the bottom row of 3x3 matrix). But yes, it might be that the same 4x4 matrix can be used, but chance is that not everyone play with the same FOV as I do, and then the matrix found in GIMP most likely is off.

However, I do rotate each "line" before get shapes, so I dont think there is much to get there. I think it might be more to gain from analyzing the orange values and get a better black/white balance.

 

[Little Angel]

No, it's in YYMMDD order, shame they dropped the 'CC' then it would be very close to ISO 8601 format, the One True Date/Time format.

Also note the extra ".01". Last night during a single session I had both a .01. and a .02. netLog file created (the latter having a different timestamp in the name too), so it's no longer even sufficient to watch the one file.

There are a couple of folks I've talked to who are interested in a captain's log. More interested in a log than trading or route planning, in fact.

I can see how the log-files can be used to gather new uncharted star names... but for mapping purposes, how can it help find the coordinates? Is there enough data in the logs to calculate a viable position?