Discussion What is the most efficient way to crowdsource the 3D system coordinates

[wolverine2710]

TornSoul the static data schema/structure/output is production or are you still in research phase? May start making the necessary changes if it's more or less set.

The only sad part for me... no storage of individual commodity data, as I was hopping we could use TGC as a centralized database to keep the current (don't need to keep historical, each one could do in their own db's if needed) market data for the various trading tools that exist. IMO this area is were we need most of the resource pooling if we want to stand a chance to have somewhat decent data outside of the game.

For dynamic data there is EDDN (see my profile). Its not public available yet but if you PM me I can give you the repo url and you can look at the wiki site. EDDN is basically EMDN by Andreas. It relays data is receives. Anyone can collect the data from the firehose and for example make available the last entries etc etc... EDDN could also do that. I'm planning to create an ELK stack (google) for it, Especially Kibana is worth looking into. I've already been contacted by two commanders who are willing to setup a database with the data. Back from holdiday, so a backlog and haven't been able to give EDDN the love it deserves. EliteOCR could for example be feeding EDDN.

Edit: kfsone (Trade Dangerous) and Thrudd have also said they want to support EDDN. Aslo Maddavo (creator of a merging tool for TD data) said he was "in". Slopey has not responded to a PM so I don't know what he is planning.

Edit2: I DO think its good to separate static and dynamic data.

 

[CMDRKNac]

For dynamic data there is EDDN (see my profile). Its not public available yet but if you PM me I can give you the repo url and you can look at the wiki site. EDDN is basically EMDN by Andreas. It relays data is receives. Anyone can collect the data from the firehose and for example make available the last entries etc etc... EDDN could also do that. I'm planning to create an ELK stack (google) for it, Especially Kibana is worth looking into. I've already been contacted by two commanders who are willing to setup a database with the data. Back from holdiday, so a backlog and haven't been able to give EDDN the love it deserves. EliteOCR could for example be feeding EDDN.

Edit: kfsone (Trade Dangerous) and Thrudd have also said they want to support EDDN. Aslo Maddavo (creator of a merging tool for TD data) said he was "in". Slopey has not responded to a PM so I don't know what he is planning.

Yes I saw the thread, but wasn't sure if the project was still alive or not. Good to see it is, I'm definitively interested (was going to offer myself to setup an API for storing commodity data in my own db to compliment TGC), so I hope the project continues. Will keep checking.

That data is sort of static. A bit less static, then system and station factions and government, but still, pretty static...

My thoughts exactly, but didn't want to push TornSoul (I don't know if he is paying for the hosting or using some free service, also is additional work to do so his choice ultimately). I think pretty much everything is 'dynamic' in the game, but most stuff rarely ticks, once a day, and most of the time nothing significant won't change (unless some big event is going on). I'm not sure how often the market data ticks, I guess is either 5min like missions or also daily (haven't been playing much lately unfortunately lol), FD has designed the game to reduce data transit substantially to keep operating costs low, it's not like the usual MMO in that regard.

Given the ways we have to collect and input data right now are hampered by Frontier (understandably), I'm not sure we are going to get many manual input ticks, and we may be 'fine' with a sort of 'static' database for most practical purposes.

 

[wolverine2710]

With EDDN we were extremely lucky. A hosting firm offered us free hosting. A 20 Mbps connection, a good server with 80GB of storage. That should go along way ;-) I must contact jamesremuscat so we can set things in motion. A POC is already running. Problem: was away a week for work and then a week on holiday... A sleeping/dying project (in the eyes of the commanders) isn't a good project ;-(

 

[Finwen]

One thing i miss about a system is if the star is scopable or not. Scopability could be derived from Spectral Star type.

I think a optionally field for star type (ex: M3) would be nice. From that i could plot a nicer map with star colors and my route show if the stars allow fuel scoping.

 

[JesusFreke]

Gotta say you guys are doing a great job, every system I visited last night was already on the database!

Keep in mind that any (almost?) system with an economy will be there already, since those were provided as a big dump by frontier (thanks!!).

Other than that, I've been focusing my distance collection in a rough sphere around shinrarta dezhra. I currently have all systems near it in something like a 30LY radius, with various offshoots up to probably 40-50LY out. I probably don't have all those stars submitted yet though.. I don't submit them until I get at least 5 distances and a redundancy of 1, which helps catch any input errors before I submit the data.


I've been keeping track of every system that I enumerate the near stars from the nav panel for, along with the distance to the furthest one in the nav panel. This gives me a set of spheres of known space. Then, I go through and find every 3-way intersection between the spheres (using trilateration..). For each such intersection, I check if there is a 4th sphere that contains that intersection point. If so, then the intersection is an interior intersection and is ignored. If the intersection is not contained by any other sphere then it is an "exterior" intersection - meaning it is adjacent to some area that hasn't been checked for stars yet. So I find and print all such exterior intersections. For each such intersection, I print the distance to the central star that I'm working from (in this case, shinrarta dezhra), the distance to where I currently am, and the nearest known star to that point.

These exterior intersection points are the candidates for which star I go to next to enumerate neighbors. I'll either go directly to the nearest known star to that intersection, or look for the nearest unknown star in the rough direction away from the central star.

 

[RedWizzard]

What trade data is that? ingame or out of game tools?

In-game. If you're at a nearby system you can often see the test systems listed in the systems that a particular good is exported to.

 

[RedWizzard]

TGC static data (output) version 2

System
Details are no longer derived from factions.
Faction name still has to exist in the factions list.
economies are now an array : Order is preserved - Duplicates are allowed

Station
Faction name still has to exist in the factions list.
Government/Allegiance/Influence/State is still derived from faction

economies are now an array : Order is preserved - Duplicates are allowed
facilities are now an array : Order is NOT preserved - Duplicates are NOT allowed

blackmarket : Is still kept separate.

Comments/opinions

Looks good to me.

 

[RedWizzard]

Was AWOL for a week (holdiday, no internet). Have to do a LOT of reading up. Just a few questions - haven't (yet) read the last week posts ;-(

What is the status of TGC by TornSoul?
Is is already possible to add other data then distances. Ie adding Systemnames to TGC and other data?

Is currently TOR by RedWizzard leading or is it TGC by TornSoul?
RedWizzard: I think you mentioned last week you wanted to add TGC support to your tool. Has this been implemented?
JesusFreke: I think you mentioned last week you wanted to add TGC support to your tool. Has this been implemented?

TGC support is mostly done for my tool but not yet pushed to Github. I'll get that done tonight (in the next 12 hours). That's priority #1.

I have a few other things I'm working on:

• Verify the reference systems that were in earlier betas but aren't in gamma. These are done but I need to write a tool to update systems.json. I'm verifying them by checking the distance to each one from two reference systems about 200 Ly apart. There are 41 that still exist and have the expected distances. There are 45 that can't be found in the galaxy map - I suspect these have been renamed.
• Verify the contributed systems from earlier betas. I've checked them all from one reference system, just need to check about 100 from the second reference system. Then merge this data into systems.json.
• Write a tool to grab the TGC data and compare it to systems.json so I can submit the data I have that isn't in TGC.
• Some enhancements for my entry page (optmisation and reference system suggestions are the main ones).

 

[bollwerk]

Not sure if this is the appropriate place to post this, but I get an error when trying to input data into http://edstarcoordinator.com/default.html
Is the site working?
I've tried in both Chrome and IE.
The error message is just a very generic "woops. The server returned an error. That's not supposed to happen."

 

[RedWizzard]

Not sure if this is the appropriate place to post this, but I get an error when trying to input data into http://edstarcoordinator.com/default.html
Is the site working?
I've tried in both Chrome and IE.
The error message is just a very generic "woops. The server returned an error. That's not supposed to happen."

Yes, looks like something is wrong. Static pages are loading ok but dynamic stuff such as autocomplete and the actual submission are not.

 

[CMDRKNac]

there is any OCR solution to capture system names and distances to other systems? I guess the ones used for trade tools can be refactored to use for this, that would speed up the process after gamma.

 

[RedWizzard]

there is any OCR solution to capture system names and distances to other systems? I guess the ones used for trade tools can be refactored to use for this, that would speed up the process after gamma.

Unfortunately we need distances from the galaxy map so OCR isn't practical. The nav panel sometimes shows distances that are out by 0.01 Ly. It also only shows two decimal places if the distance is less than 10 Ly.

OCR on the nav panel would work to collect system names and it might help for economy/faction data in the galaxy map.

 

[Snuble]

It might be as someone suggested that the galaxy map (and maybe also the "leaked" coordinates?) uses a mass center for the system that is different from the entry star. I'm in LHS 266, 154360Ls from LHS 266 A, and distances have changed. Its not longer the same systems that are off by 0.1. BD+27 is now off by 0.01, while Yin sector have same distance in galaxy map and nav panel where it at the A star had 0.01 wrong distance.

I dont think there is any way around this but to use the galaxy map for gathering distances.

 

[RedWizzard]

Yes, looks like something is wrong. Static pages are loading ok but dynamic stuff such as autocomplete and the actual submission are not.

Just tried a submit and am getting a cross-origin request blocked error.

Edit: seems to be working again now.

- - - - - Additional Content Posted / Auto Merge - - - - -

It might be as someone suggested that the galaxy map (and maybe also the "leaked" coordinates?) uses a mass center for the system that is different from the entry star. I'm in LHS 266, 154360Ls from LHS 266 A, and distances have changed. Its not longer the same systems that are off by 0.1. BD+27 is now off by 0.01, while Yin sector have same distance in galaxy map and nav panel where it at the A star had 0.01 wrong distance.

I dont think there is any way around this but to use the galaxy map for gathering distances.

I agree, it'll have to be the galaxy map. 150,000 Ls is 0.005 of a Ly so it makes sense that it is affecting the distances. The problem is that any distance from the primary star or center of mass (whichever the galaxy map uses) could tip the rounding over so it shows a different value in some cases.

 

[TornSoul]

Edit: seems to be working again now.

Just did a iisreset - No idea what the problem was... IIS being IIS I guess.

 

[TornSoul]

One thing i miss about a system is if the star is scopable or not. Scopability could be derived from Spectral Star type.

I think a optionally field for star type (ex: M3) would be nice. From that i could plot a nicer map with star colors and my route show if the stars allow fuel scoping.

I didn't realize some stars weren't scoopable....
I agree that that would actually be a nice thing to know.


Raises the issue that there's often more than one star in a system though.

Some could be scoopable others not.

So do we save info for all stars - or just if the system as such has _a_ scoopable star.

Do we save it as a boolean - or the actual star type while at it?

Does anyone have a full mapping of which types are scoopable and which are not?

 

[Athan]

FGE gathered some info and the conclusion was:

Can Fuel Scoop


Class = A
Class = B
Class = F
Class = G
Class = K
Class = M
Class = O


Cannot Fuel Scoop


Class = DA (White Dwarf)
Class = DC (White Dwarf)
Class = Y
Class = T
Class = L
Class = TTS
variable stars aren't scoopable
----

I can confirm Y/T/L/TTS as not scoopable myself. Given those are some form of Brown Dwarf and thus low mass, together with jump-in supposedly always being at the most massive object in the system, I think we can conclude that only the main star type matters for scoopability. If the main is unscoopable then nothing else in the system is going to be scoopable.

 

[TornSoul]

If the main is unscoopable then nothing else in the system is going to be scoopable.

Right. So a simple bool should be good enough?

 

[Harbinger]

Given those are some form of Brown Dwarf and thus low mass, together with jump-in supposedly always being at the most massive object in the system, I think we can conclude that only the main star type matters for scoopability. If the main is unscoopable then nothing else in the system is going to be scoopable.

It's not the physical size of the object that affects the jump in location but it's mass. A white dwarf can have a massive mass and as such be the jump in target but there may well be a much larger M class star with a lesser mass in the system which is in fact scoopable.

The star types contained within a system is the one thing that is available via the INFO tab on the Galaxy Map, even if you know nothing further about the system.

 

[Athan]

Yes, on reflection it's probably an idea to just store the spectral type of all stars in the system. Then tools can use/present that as scoopable or not as needed. Whole lot of data gathering though.