Discussion What is the most efficient way to crowdsource the 3D system coordinates

[TornSoul]

I did PM TornSoul

Bliff... I just don't think to check for those... (as in, this is the 3rd ever I've gotten so..)

If you hadn't posted that you had PM'ed me, who knows when I'd noticed I had a PM hanging

---

TGC :
The intend behind the TGC has been to have it be The Source™ of system data for ED - That all other tools could (not should) then sync with.

It was debated back and forth here what TGC should and should not store.

It was agreed that initially (as a sort of Beta run) it should have the most basic and not more (ie. coordinates of systems)
Simply to get it up and running.
The most simple being what it has now (alpha 0.3) (a few extra bits and bobs - but that's pretty much it - data wise. It has some pretty nifty filters though)

The thought was then to add on to that when we where satisfied the TGC was doing it's job.

It was also agreed that TGC was best suited for static data.

So on the TODO list are things like:
- Stations in a system,
- Black/commodity markets availability
- Economy/allegiance/government etc.
- other bits and bobs we can come up with.


This at least has been my understanding and what I signed up to provide.


Tools that wish to maintain their own DBs of market data are encouraged to do so, and also encouraged to submit static data to the TGC, if their tools provide their users a way of submitting data.

The TGC itself does not have a "user interface" and is reliant on as many tool makers out there as possible to submit data.

One such tool is EDSC (which I've been neglecting badly while working on TGC ).
Ie. EDSC is simply an input form to TGC (and a user of TGC data ofc)

The roles obviously gets a bit confusing as TGC is hosted along side EDSC.
But I didn't want to buy yet another domain just for TGC, as no "enduser" will ever have to interact with TGC anyhow (and thus no easy to remember URL is needed).
Endusers interact with the tools, that then interact with TGC.

I hope to see *a lot* of tools, like yours, out there, who will all help with gathering the data.
Over the years, some tools will disappear, and new and (better) ones will turn up.
But the data stored in TGC will remain for those newcomers to take advantage of.


If you wish to contribute data to TGC or not, is completely up to you of course.
But it would help everyone if you did - and helping everyone with data is really the only reason for TGC to exist in the first place.

 

[TornSoul]

A quick FYI:

I've worked on setting up a test server for TGC.
It gets created from a fresh backup of the real TGC DB every 24 hours (at 1:15 UTC).
This way you have a bit of retention on your test data, if you want to test some particular scenarios.

I'm adding a simple "test" toggle to the options for calling the API to switch between one or the other DB.

This way tool makers can hammer the test DB without worrying about polluting the real DB, which can be an issue when you are testing your submit functions (as I've found out myself working a bit on EDSC today)

Once done - I'll "upgrade" to Beta status, as it seems everything is fairly stable.

And then as mentioned, I'll work on getting the last static data added to TGC as well.

 

[Little Angel]

Funny, TornSoul, I was just loggin' on to ask about that.

I'm going to try and integrate your tool into my tool, and was wondering what the best way to do that would be.

I did look at the jsfiddle site that was linked, but I didn't understand the output, which seemed to be an error.

 

[TornSoul]

Use ver=0 while in Alpha

It's set to 2 in the fiddles - as that will be the value, once I move it to beta (probably tomorrow night - ie. 12-20 hours from now).

 

[Harbinger]

Just a quick FYI. Be sure to keep an eye on systems.json on RedWizzard's Github. I've personally submitted 80+ new systems (star catalogue systems) in the last few days so there's lots of new data in there.

 

[Little Angel]

Do RedWizzard's and TornSoul's systems collaborate?

The main interest for me with EDSC is that my tool will be able to push data to it without the user having to do anything more than fill out a form internally, and press submit. I'm hoping the POST will do the heavy lifting.

From looking at RedWizzard's github page, it looks like I would have to be doing manual commits.

What I'm hoping is that people use this companion tool and then log stars for themselves, and then submit that data as a matter of course, rather than convincing people to stop what they are doing and search for stars, if they don't want to.

 

[RedWizzard]

Do RedWizzard's and TornSoul's systems collaborate?

Not at the moment. Once TGC is up I'll submit all the data from systems.json. I might continue to maintain systems.json until TGC includes station and demographic data, if so then I'll probably manually forward any new systems that appear in systems.json. Once TGC has all the data I'll stop maintaining systems.json.

 

[TornSoul]

TGC is now live : ver=2

A "test" toggle has been added to the input parameters - Please use it when testing your apps.

A mirror of the live DB is made every day at 1:15 UTC to test against.


EDSC has also been updated and is now fully functional - and can be used to submit data if you prefer that over the coding way.

EDSC has a toggle for "test"ing - and also an "advanced" toggle - For when you just want to enter individual distances.

And commander name has been added (it will recall that name between sessions - So you only need to enter it once)

Enjoy.

--

Work now commences on ver 2.1 - Which will include all the other static data.
I'll post a draft here once I got it, so you can see if I've forgotten something or not.

 

[Rhulan]

Bliff... I just don't think to check for those... (as in, this is the 3rd ever I've gotten so..)

If you hadn't posted that you had PM'ed me, who knows when I'd noticed I had a PM hanging

No worries, thought that might be what was happening. The forums should fire off an email or something

It was agreed that initially (as a sort of Beta run) it should have the most basic and not more (ie. coordinates of systems)
Simply to get it up and running.
The most simple being what it has now (alpha 0.3) (a few extra bits and bobs - but that's pretty much it - data wise. It has some pretty nifty filters though)

The thought was then to add on to that when we where satisfied the TGC was doing it's job.

It was also agreed that TGC was best suited for static data.

So on the TODO list are things like:
- Stations in a system,
- Black/commodity markets availability
- Economy/allegiance/government etc.
- other bits and bobs we can come up with.


This at least has been my understanding and what I signed up to provide.


Tools that wish to maintain their own DBs of market data are encouraged to do so, and also encouraged to submit static data to the TGC, if their tools provide their users a way of submitting data.

The TGC itself does not have a "user interface" and is reliant on as many tool makers out there as possible to submit data.

One such tool is EDSC (which I've been neglecting badly while working on TGC ).
Ie. EDSC is simply an input form to TGC (and a user of TGC data ofc)

The roles obviously gets a bit confusing as TGC is hosted along side EDSC.
But I didn't want to buy yet another domain just for TGC, as no "enduser" will ever have to interact with TGC anyhow (and thus no easy to remember URL is needed).
Endusers interact with the tools, that then interact with TGC.

I hope to see *a lot* of tools, like yours, out there, who will all help with gathering the data.
Over the years, some tools will disappear, and new and (better) ones will turn up.
But the data stored in TGC will remain for those newcomers to take advantage of.


If you wish to contribute data to TGC or not, is completely up to you of course.
But it would help everyone if you did - and helping everyone with data is really the only reason for TGC to exist in the first place.

Thanks for that explanation, very detailed. In that case, what I might do for now then is to use your API to retrieve the coordinates for systems, and display them as read-only. The rest of the system details can be sourced from my database (which is populated from RW's JSON). Then as TGC adds more data to its schema, I can remove them from mine and retrieve them from TGC instead. We'll have to have a discussion about how to go about authentication between the systems, but that can wait for now.

 

[TargetLost]

Just verified my 4 Fix Points in ED Beta 3.05
They are correct.
I flew to each one and the read out ED Distances match exactly with the distances that you can calulate out of their given coordinates rounded to 2 digits after the point.

The check File ( FixPointSystemCheckList.txt ) is in my DropBox:
https://www.dropbox.com/sh/9g15l3um92vu0x3/AACldLmEAPC_2JzM4K_wtT02a?dl=0

Fix Point System Check-List (Last checked Beta 3.05)

Fix Point System Coordinates

1) PARCAE -8.125; 55.09375; -17
2) HIP 7338 -96.5; 33.21875; -68.65625
3) LHS 3549 -29.625; 6.125; -1.9375
4) AO QIN -62.90625; 46.78125; 1.53125

( SOL 0;0;0 )


Fix Point System Distance Checks

PARCAE HIP 7338 104.68
PARCAE LHS 3549 55.56
PARCAE AO QIN 58.43
PARCAE SOL 58.23

AO QIN PARCAE 58.43
AO QIN HIP 7338 78.99
AO QIN LHS 3549 52.66
AO QIN SOL 78.41

LHS 3549 PARCAE 55.56
LHS 3549 HIP 7338 98.27
LHS 3549 AO QIN 52.66
LHS 3549 SOL 30.31

HIP 7338 PARCAE 104.68
HIP 7338 LHS 3549 98.27
HIP 7338 AO QIN 78.99
HIP 7338 SOL 123.00

so if you use those 4 as references you should be fine in beta 3.05

 

[TornSoul]

OK so I imagine an output something like this, if a system is requested with the Verbose flag.
ie. it won't be possible to request an individual station - you pull a full system.
I can't see pulling individual stations making much sense.
Let me know if you disagree.

Spoiler

{
  id: 1,
  name: "HIP 107457",
  coord: [-116.25, 28.34375, -39.3125],
  cr: 5,
  
  commandercreate: "",
  createdate: "2014-11-07 13:38:07",
  commanderupdate: "",
  updatedate: "2014-11-07 13:38:07",
  
  detail: {
    allegiance: "Federation",
    economy: "Extraction",
    government: "Corporate",
    population: 31839,
    security: "Medium",
    
    cr:1,
    commandercreate: "",
    createdate: "2014-11-07 13:38:07",
    commanderupdate: "",
    updatedate: "2014-11-07 13:38:07"
  },
  
  stations: [
    {
      id: 24,
      name: "Cenker Outpost",
      distance: 6450,
      type: "Industrial Outpost",
      economy2: "Refinery",
      faction: "HIP 107457 Incorporated",
      commarket: true,
      blackmarket: false,
      shipyard: false,
      outfitting: true,
      maxpad: "medium",
    
      cr:1,  
      commandercreate: "",
      createdate: "2014-11-07 13:38:07",
      commanderupdate: "",
      updatedate: "2014-11-07 13:38:07"
    }
  ]
}

It's mostly, but not quite rw's format.

I've stolen the "detail" idea from Biteketkergetek over here as it makes sense with regards to adding the create/update info.

I have a strong feeling it could be different people doing coords and others filling in the details.
Details and station data may or may not be the same - who knows.
But I think it makes sense keeping that data grouped.
And if not, I would not know how to properly deal with someone updating one thing but not the other etc.

Note that I've added a "economy2" property to stations - As some do in fact have multiple.
I've yet to see any with 3 - but easy to add if it becomes necessary.

Also note that a cr property have been added to each grouping (detail/station) as a whole.
I don't think it's feasible (or makes much sense) having a cr property for every individual datapoint.

It will simply get updated whenever someone makes an edit/update.
Another option would to be get rid of the cr property for detail/stations.
I'm honestly not sure how much sense it makes - as info can be submitted piecemeal.


Submitting:
I'm thinking of allowing for submitting individually
- details
- station info
(ie. two new endpoints)

Let me know what you think.

Unless I hear otherwise, this will be what i'll work towards.

 

[wolverine2710]

Off Topic. TGC is being used to store static information. The following might be used for dynamic data - ie prices information.

Perhaps interesting for getting more and more up to date price information. "EDDN - Elite Dangerous Data Network. Trading tools sharing info in an unified way". Basically EMDN by Andreas but then filled with data NOT gathered by scraping the ED program/protocols. Hence data gathered in a legal way. In this case data entered manually in trading tools. I've PM-ed trading tools authors. Since I wrote the thread already someone PM-ed me to say he's very interested in creating EDDN on an Azure platform. I hope the TT authors respond and are positive about the idea. Fingers crossed.

Edit: When a viable OCR solution exists, that data could also be send to EDDN.

 

[Biteketkergetek]

I've stolen the "detail" idea from Biteketkergetek over here as it makes sense with regards to adding the create/update info.

For the record, the idea came from Smacker. I just converted them to JSON from his SQL.

Numeric IDs caused problems while updating the SQL files of TradeDangerous, I believe they've switched over to CSV because of those problems. They aren't necessary either, because system names should be unique. (I still support duplicate names, we'll see if they will be really unique in the end. I kept both Charas (link to real and duplicate) from the Beta3 data from FBdev for the time being, with the prefixed ID calculated from their coordinates).

 

[wolverine2710]

TGC is now live : ver=2

A "test" toggle has been added to the input parameters - Please use it when testing your apps.

A mirror of the live DB is made every day at 1:15 UTC to test against.


EDSC has also been updated and is now fully functional - and can be used to submit data if you prefer that over the coding way.

EDSC has a toggle for "test"ing - and also an "advanced" toggle - For when you just want to enter individual distances.

And commander name has been added (it will recall that name between sessions - So you only need to enter it once)

Enjoy.

--

Work now commences on ver 2.1 - Which will include all the other static data.
I'll post a draft here once I got it, so you can see if I've forgotten something or not.

One word: PERFECT.

 

[Little Angel]

OK so I imagine an output something like this, if a system is requested with the Verbose flag.
ie. it won't be possible to request an individual station - you pull a full system.
I can't see pulling individual stations making much sense.
Let me know if you disagree.

<snip>

It's mostly, but not quite rw's format.

I've stolen the "detail" idea from Biteketkergetek over here as it makes sense with regards to adding the create/update info.

<snip>


Submitting:
I'm thinking of allowing for submitting individually
- details
- station info
(ie. two new endpoints)

Let me know what you think.

Unless I hear otherwise, this will be what i'll work towards.

The only bit of detail that seems to be missing from the station data that is available in Biteketkergetek's is "distance".

This is somewhat useful when planning a route because some stations are Soooooo far away from the entry point / nav beacon (if there is one) that it can make trading there a pain in the aspic. Having that information can help a commander decide if the route is worth the time or not.

So, I'd argue to include it.

<Hoping to get some time testing my tool with your database tomorrow or over the w/e...>

 

[Little Angel]

[Additionally]

I noted in the station data "max pad size".

This is something I've been guessing is true, especially at small "platform" stations.

I have no clue what pad sizes there are, nor what ships correspond to what pad size.

How would a commander, when at a station, know what the max pad size actually is?

 

[Harbinger]

In my experience of charting a small quantity of stations for systems.json. Although I added the pad size, knowing the station type is actually sufficient to determine the maximum pad size. [Orbis/Ocellus/Coriolis] Starports can take any vessel, [Civilian/Industrial/Commercial/Scientific] outposts can take medium vessels and unsanctioned (pirate) outposts can only take small vessels.

There's no way of knowing the outpost type from the system map though.

 

[Snuble]

If that is the case, then max pad size is not needed. Would be better to have that as metadata for stations.

 

[Little Angel]

Yeah, I'd agree, I'd rather have the station type... and then the tool itself could use that if it needed to to get additional details, if the station types are static in detail.

 

[Little Angel]

Such short star names: