Discussion What is the most efficient way to crowdsource the 3D system coordinates

CMDRKNac

C
Little Angel said:
Is "Destination" a real star in the game?

Screenshot%202014-12-04%2009.41.41.png

I noticed two stars near Dahan in the beta's star list (B2?): Destination and Training

I got the impression these were on the list from the FD dump and represented stars only accessible in the tutorials.
Click to expand...

Just a note on this:

I haven't implemented automatic deletion if a system is NOT in EDSC (not sure I will because in the future I will accept user input in my own database which then I'll try to push to EDSC one so there may be temporary inconsistencies between both). So there may be some systems which shouldn't be there (I don't know if it's so, last time I checked a couple days ago I had the same number as there are in EDSC but Torn Soul may have removed some of those dummy systems).

If anyone got a list of those systems and can post I can remove them ¡manually (sucks to have those in there!)
 

JesusFreke

J
wolverine2710 said:
Was AWOL for a week (holdiday, no internet). Have to do a LOT of reading up. Just a few questions - haven't (yet) read the last week posts ;-(

What is the status of TGC by TornSoul?
Is is already possible to add other data then distances. Ie adding Systemnames to TGC and other data?

Is currently TOR by RedWizzard leading or is it TGC by TornSoul?
RedWizzard: I think you mentioned last week you wanted to add TGC support to your tool. Has this been implemented?
JesusFreke: I think you mentioned last week you wanted to add TGC support to your tool. Has this been implemented?
Click to expand...

Yep. Although only locally - I haven't updated my repository. So far I've submitted around 250 new stars to TGC
 

TornSoul

T
wolverine2710 said:
What is the status of TGC by TornSoul?
Is is already possible to add other data then distances. Ie adding Systemnames to TGC and other data?
Click to expand...

No static data yet.
We've had a long'ish debate on the format of the data - and some research into what is needed to record (so has been a changing target)
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
TornSoul said:
No static data yet.
We've had a long'ish debate on the format of the data - and some research into what is needed to record (so has been a changing target)
Click to expand...

Thanks for the update. My backlog is NOT getting much smaller atm ;-(
Updating the third party tools list as we speak. If anyone is missing his tool in this list, please send me a PM. Each and every tool mentioned will be put in there!!!
 
Last edited:

TornSoul

T
TGS has been updated.

It will no longer fail to find coords if there exist a wrong distance in the DB already.

It will further more report which distance(s) that appear to be wrong - And the message will suggest to resubmit these.
It will NOT (although it could) suggest what those distances should be to make it a correct distance.
- The reason for this is to avoid people simply entering that suggestion, without actually confirming it on the Galaxy Map.

------

Info dump:

I've changed TGC to use the less stringent test for correctness that both JesusFreke and rw are using.

Ie.
- Minimum 5 distances must match the found coordinates perfectly (zero error)
- The found coordinates must have at least 2 less invalid distances than the next best candidate (compared by number of invalid distances)

--

High-level algorithm:

Each 3 combination of all references are generated and put through trilateration and yields a candidate (or not as the case might be)
Each candidate from each 3-combo and it's 8 neighbors (= 17^3 = 4913 gridpoints, which is overkill... ) in each cardinal direction is tested for correctness (reverse distance check) vs all the reference distances.


Some caps are put in:

There is one star that has 135 references (if I recall - and maybe it was just in the test DB).
All 3-combos of references results in almost half a million (not necessarily unique) candidates.
To avoid this I pick 25 random ones from the references, and do the 3-combo with just those.

After all 3-combos has generated their candidates - The unique ones are picked.
These are ordered by lowest RMS (on distance) - and the lowest 20 are picked.

These 20 (or less as the case might be) are then put through the cube-search (adding each neighbor up to 8 grid points away as further candidates)

On these the above "5 distance, 2 better" criteria is applied.


-------

EDIT

Here's the full algorithm in "pseudo code"
Code: 
Pick 25 random reference systems

For each 3-combo of reference systems
  Candidates = trilateration(3-combo)
  For each candidate (zero or two) 
    Calculate RMS based on *all* 25 reference systems (not just one p4)
  Add the candidate with the lowest RMS to the candidates list
  
Eliminate duplicate candidates

Pick the 20 candidates with lowest RMS values

For each candidate
  Add all neighbors within a radius of 8 gridpoints to the candidates list (17^3 candidates for each candidate)

Eliminate duplicate candidates (there will be alot of overlap - as candidates found in trilateration will be close)

For each candidate
  Do a reverse distance check to all 25 reference systems
  - Record those distances that do not match
  
Sort list of candidates by number of distances that do not match (lowest first)

Pick the first one and verify that:
  - At least 5 distances match perfectly (zero error)
  - That the "next best" candidate, has at least 2 more bad distances (bad = does not match perfectly)
  
If this checks out - We've found the coordinates of a new system
otherwise the coords are rejected.

In either case - TGC reports back any distance to the final candidate that do not match perfectly (if any)
 
Last edited:

TornSoul

T
TGC static data (output) version 2

Code: 
{
  name: "HIP 107457",

  coord: [-116.25,28.34375,-39.3125],

  cr: 5,
  commandercreate: "",
  createdate: "2014-11-07 13:38:07",
  commanderupdate: "",
  updatedate: "2014-11-07 13:38:07",
  
  detail: {
    permit : null/string,
    economies: [Refinery, Industrial], //Order is preserved - Duplicates are allowed
    
    controllingfaction: faction1, //has to match a faction in the factions list
    government: "Democracy",
    state: "None",    
    allegiance: "Federation",
    population: 31839,
    security: "Medium",

    cr:1,
    commandercreate: "",
    createdate: "2014-11-07 13:38:07",
    commanderupdate: "",
    updatedate: "2014-11-07 13:38:07"
  },
  
  factions:[
    {
      name: faction1,
      government: "Democracy",
      allegiance: "Federation",
      influence: 55.0,
      state: "None",
      //Relationship is not included - It's assumed to be per commander

      cr:1,      
      commandercreate: "",
      createdate: "2014-11-07 13:38:07",
      commanderupdate: "",
      updatedate: "2014-11-07 13:38:07"      
      
    }  
  ],
  
  stations: [
    {
      name: "Cenker Outpost",
      faction: faction1, //has to match a faction in the factions list - Which then gives Government/Allegiance/Influence/State
      facilities: ["Commodities", "Refuel", "Repair", "Re-Arm", "OutFitting"] //Order is NOT preserved - Duplicates are NOT allowed
      economies: [Refinery, Industrial] //Order is preserved - Duplicates are allowed
      
      distance: 6450,
      type: "Industrial Outpost",
      blackmarket: false, //is kept seperate      

      cr:1,      
      commandercreate: "",
      createdate: "2014-11-07 13:38:07",
      commanderupdate: "",
      updatedate: "2014-11-07 13:38:07"
    }     
  ]
}

System
Details are no longer derived from factions.
Faction name still has to exist in the factions list.
economies are now an array : Order is preserved - Duplicates are allowed

Station
Faction name still has to exist in the factions list.
Government/Allegiance/Influence/State is still derived from faction

economies are now an array : Order is preserved - Duplicates are allowed
facilities are now an array : Order is NOT preserved - Duplicates are NOT allowed

blackmarket : Is still kept separate.


Comments/opinions
 
Last edited:

Athan

Athan
TornSoul said:
TGC static data (output) version 2

Code: 
{
  name: "HIP 107457",

  coord: [-116.25,28.34375,-39.3125],

  cr: 5,
  commandercreate: "",
  createdate: "2014-11-07 13:38:07",
  commanderupdate: "",
  updatedate: "2014-11-07 13:38:07",
  
  detail: {
    economies: [Refinery, Industrial], //Order is preserved - Duplicates are allowed
    
    controllingfaction: faction1, //has to match a faction in the factions list
    government: "Democracy",
    state: "None",    
    allegiance: "Federation",
    population: 31839,
    security: "Medium",

    cr:1,
    commandercreate: "",
    createdate: "2014-11-07 13:38:07",
    commanderupdate: "",
    updatedate: "2014-11-07 13:38:07"
  },
  
  factions:[
    {
      name: faction1,
      government: "Democracy",
      allegiance: "Federation",
      influence: 55.0,
      state: "None",
      //Relationship is not included - It's assumed to be per commander

      cr:1,      
      commandercreate: "",
      createdate: "2014-11-07 13:38:07",
      commanderupdate: "",
      updatedate: "2014-11-07 13:38:07"      
      
    }  
  ],
  
  stations: [
    {
      name: "Cenker Outpost",
      faction: faction1, //has to match a faction in the factions list - Which then gives Government/Allegiance/Influence/State
      facilities: ["Commodities", "Refuel", "Repair", "Re-Arm", "OutFitting"] //Order is NOT preserved - Duplicates are NOT allowed
      economies: [Refinery, Industrial] //Order is preserved - Duplicates are allowed
      
      distance: 6450,
      type: "Industrial Outpost",
      blackmarket: false, //is kept seperate      

      cr:1,      
      commandercreate: "",
      createdate: "2014-11-07 13:38:07",
      commanderupdate: "",
      updatedate: "2014-11-07 13:38:07"
    }     
  ]
}
Comments/opinions
Click to expand...

How about storing at least a boolean for permit required, if not some extended data (some seem to be per region rather than specific system?). I keep running into my calculated routes being no good because they include a Permit Required system :S
 

TornSoul

T
Athan said:
How about storing at least a boolean for permit required, if not some extended data (some seem to be per region rather than specific system?).
Click to expand...

Darnit - I keep forgetting that one.

I'll edit my post.

I think a string is best atm. as we have no idea what attributes it might have.
 

wolverine2710

Tutorial & Guide Writer
wolverine2710
Little Angel said:
I'm slightly confused, and could use some straightening out...

What is TGC vs EDSC?
Click to expand...

TornSoul first created EDSC. A tool to enter distances and to calculate coordinates. The tool tells you when you have collected enough good data. After that he created TGC (The Great Collector). Its a web-api. You upload distances and then it calculates coordinates. You can also get data from it. Its planned to store more static data, like distance from sun to station, station info like has BM etc.. EDSC now uploads its distances to TGC which then calculates the coordinates. This is how I interpreted his posts.

Unfortunately he didn't/couldn't buy a separate domain for TGC and placed it on the EDSC website. That is were the confusing is coming from. I've seen the ESC mentioned in other threads as well, when they actually meant TGC.
 

TornSoul

T
Little Angel said:
I'm slightly confused, and could use some straightening out...

What is TGC vs EDSC?
Click to expand...

EDSC is simply an input tool for TGC.

TGC is the API that you can submit data to and request data from.

TGC is hosted along side EDSC as I didn't want to buy another domain for something no ordinary user is ever going to see anyhow (and thus don't need an easy to remember URL for)


I think that for most people who haven't followed this thread the two (EDSC/TGC) are synonymous - and because of the website, often just refer to the whole thing as EDSC
 
Last edited:

CMDRKNac

C
TornSoul the static data schema/structure/output is production or are you still in research phase? May start making the necessary changes if it's more or less set.

The only sad part for me... no storage of individual commodity data, as I was hopping we could use TGC as a centralized database to keep the current (don't need to keep historical, each one could do in their own db's if needed) market data for the various trading tools that exist. IMO this area is were we need most of the resource pooling if we want to stand a chance to have somewhat decent data outside of the game.
 

TornSoul

T
The static data is still in the "draft stage" - So input and corrections are encouraged.

Way back when - It was agreed in this thread that TGC would be static data only.

Hence commodity data is not a part of TGC.
 

Little Angel

Little Angel
CMDRKNac said:
TornSoul the static data schema/structure/output is production or are you still in research phase? May start making the necessary changes if it's more or less set.

The only sad part for me... no storage of individual commodity data, as I was hopping we could use TGC as a centralized database to keep the current (don't need to keep historical, each one could do in their own db's if needed) market data for the various trading tools that exist. IMO this area is were we need most of the resource pooling if we want to stand a chance to have somewhat decent data outside of the game.
Click to expand...

I was thinking about this... and I felt that this was data I *didn't* want stored through the EDSC. Maybe it's just my gut, but I felt that it was best to keep static and dynamic data separate from each other. There are a number of other tools that store trading data already (and many use the EDSC to maintain their star list). I was considering approaching one of these tool makers as a way of handling that data. It's so transient, trading data, and needs a whole host of peripheral tools to check how fresh or stale the data is... and most of the effective tools already have this. It would save having TornSoul write all of that support structure around dynamic trading data, as well as static star data.
 

Snuble

S
TornSoul said:
The static data is still in the "draft stage" - So input and corrections are encouraged.

Way back when - It was agreed in this thread that TGC would be static data only.

Hence commodity data is not a part of TGC.
Click to expand...

You know, you could extend TGC to listen to the firehose and just keep a snapshot (as in not storing the complete history) of market data... That would in fact be very helpfull if someone for some reason have downtime on their listening and need a 1 time update for the market data.
That data is sort of static. A bit less static, then system and station factions and government, but still, pretty static...
 
You must log in or register to reply here.
Back
Top Bottom