Discussion What is the most efficient way to crowdsource the 3D system coordinates

wolverine2710

Tutorial & Guide Writer
wolverine2710
Dejay said:
I think the best way would be to OCR the distances to stars in the navigation panel.
Click to expand...

OCR-ing solutions are becoming a very viable solution. At least for the market prices. I think the same mechanism can/will be applied to the nav menu when everything is working near perfectly. EliteOCR by Seebek to me seems to be the best OCR solution out there atm - as in wrt accuracy. zxctypo has trained a neural net for the numbers (which is now in EliteOCR) which has a > 99.99 accuracy. The OCR-ing of the commodities names was not 100% in the past but as it is a fixed list a Approximate string matching algorithm could take care of that and find the best fitting sytem name. I believe that it is in the current version or the next - not 100% sure. The system names (without economy) are NOT all known so that is a challenge. zxctypo is atm training the NN for names as well. Curious about the accuracy. BUT even for that a fixed list can be setup. Suppose when TGC has support for all static data someone could run a tool which uploads the current system to the TGC so we have the exact names (from the netlogs). Slopey newest BPC has a flight log which could be send to TGC.

That all said, it is said that we have to jump through that many loops. I hope in the future a public available web-api will exist and the coordinates, name of a system will be in it. Would make life so much easier. Life could be so much easier if we just ignore FD's wishes and hack the ED/protocols, but we don't. We are good Ed citizens. But it will happen in the future by others.....
 

sngerous

sngerous
TornSoul said:
JesusFreke thank you for the list!

I've got a similar (thankfully smaller) list from rw in PM (I'll have to compare them)

-----

Before I do anything though I'd like your opinion on the following:

So far I've been manually updating the TGC - and for renaming etc that works fine, as the "updateddate" will be updated, so that tools can automatically capture these changes.

The problem is however whenever I do a deletion.
Situation:
Someone submits a system (but miss types the name) along with enough distances to calculate coordinates.
The miss typed system is of a (correct) system *already* in the DB with coords (and potentially distances).
We thus end up with two systems (well, same system, but with two different names) with the same coordinates.

Obviously the wrong one needs to be deleted.

This sadly was a situation I didn't anticipate - And thus the system isn't really geared to handle deletions.
Or rather - It isn't geared to report those deletions to users of TGC.

Which can be an issue for tool makers relying on the data (they can ofc (and should) do their own sanity checks - but it's annoying).

So what I would propose would be to simply set cr=-1 for entries that have been "deleted", as this will update the "updateddate" field, and thus a pull from TGC will show that this entry needs to be deleted in your local DB (or marked similarly)

Would this be an acceptable way forward?
And pitfalls with this approach you guys can spot?
Or an even better way of handling it? (that doesn't require me to rewrite the whole DB/code setup :p)

Even with the cr=-1 change, I'll need to touch all my stored procedures etc (so that cr=-1 entries are ignored) but I think that's the lesser evil atm.

Thoughts?


EDIT:

On second thought...

If I'm going through all the trouble of updating all stored procedures (to ignore deleted entries), I might as well change all the tables and add a boolean "isValid" column instead - To avoid overloading the cr value.
As that can always have unforeseen consequences - depending on (code) logic in both my own code and that of tool makers - depending on the cr value.

Thoughts?
Click to expand...

What about spam? How do you avoid the database being flooded with random data from a program?

When thinking of my app, i would probably avoid any non-valid or weak data and only consider cr5 systems because it is better to display nothing when there is some, giving the user the opportunity to enter a system himself, than to have something that is wrong which result in a user becoming angry.
I'd say it seems more reliable to miss something than to have something wrong.

So, yea, reading your edit, i'd answer: yes, hiding cr and replacing it with a "valid" boolean is more comfort most db users will need.
 

JesusFreke

J
TornSoul said:
There are system name/coords combos that do not belong together in TGC.
Proven by the Pareco case. And it's unlikely to be the only case.

There might be some that don't exist anymore.
Etc.

I'm *guessing* it's a low percentage (or we'd have heard from more people).

But just how polluted TGC is atm (using the gamma data) is impossible to say.
And likewise, it's impossible to say, if current valid data will actually stay valid in perpetuity.

So apparently TGC will need to become a curated depository...

I for one do not want to have to be the one to fix stuff in TGC whenever changes to the Stellar Forge happens, for the next many many years (hopefully ED will exist a long time)


And frankly it irks me the wrong way that base system data (name/position) is not "fixed" - but can change.
That's just wrong for a persistent universe(/galaxy) - That ED is supposed to be.


So atm I'm not quite sure what to think tbh.

There are several options - None of which I particularly like.

I'll have to think it over for a bit - And also see what the rest of you think/suggests.
Click to expand...

In the back of my mind for a while, I've been thinking what we actually need is some sort of... wiki-database or something (with an appropriate api of course), with the sort of crowdsource/wiki/community-driven things you would expect. user reputation, higher editing/mod privs with high rep, etc.

Some potential solutions? I haven't actually looked at these much..

http://foswiki.org/About
http://semantic-mediawiki.org/wiki/Semantic_MediaWiki
https://code.google.com/p/database-wiki/
 

Dejay

Dejay
JesusFreke said:
In the back of my mind for a while, I've been thinking what we actually need is some sort of... wiki-database or something (with an appropriate api of course), with the sort of crowdsource/wiki/community-driven things you would expect. user reputation, higher editing/mod privs with high rep, etc.

Some potential solutions? I haven't actually looked at these much..

http://foswiki.org/About
http://semantic-mediawiki.org/wiki/Semantic_MediaWiki
https://code.google.com/p/database-wiki/
Click to expand...

I agree that something like "wowhead" is needed (provided a huge boost to WoW over the years, a kind of meta community).
I think a specialized interface to a database is best. elitetrader.co.uk is really great for example. But a collaborative effort would of course be better.

The ideal 3rd party tool platform would be a "HTML5 / javascript" application that can run on a webserver or locally on your PC. You could support addons to this platform as a HTML5 applet. Ideally with a browser that supports "overlay" like steam does, ideally also for VR (placing a 3D window someplace in your VR space as an overlay to your normal game. Now I'm just dreaming of course haha. But imho something like this is what VR needs.
 

JesusFreke

J
Dejay said:
I agree that something like "wowhead" is needed (provided a huge boost to WoW over the years, a kind of meta community).
I think a specialized interface to a database is best. elitetrader.co.uk is really great for example. But a collaborative effort would of course be better.

The ideal 3rd party tool platform would be a "HTML5 / javascript" application that can run on a webserver or locally on your PC. You could support addons to this platform as a HTML5 applet. Ideally with a browser that supports "overlay" like steam does, ideally also for VR (placing a 3D window someplace in your VR space as an overlay to your normal game. Now I'm just dreaming of course haha. But imho something like this is what VR needs.
Click to expand...

Yeah, regarding the VR window thing, I'd love to get even something like a little VR text editor/note taker type of thing, that you could alt-tab to something. Or possibly a VR chrome window.. :D
 

RedWizzard

RedWizzard
TornSoul said:
On second thought...

If I'm going through all the trouble of updating all stored procedures (to ignore deleted entries), I might as well change all the tables and add a boolean "isValid" column instead - To avoid overloading the cr value.
As that can always have unforeseen consequences - depending on (code) logic in both my own code and that of tool makers - depending on the cr value.

Thoughts?
Click to expand...

I like the idea of a separate "valid" field. I think the problem with using CR is that it's always a filter so presumably even though a deleted system has a update date that would include it in the data being fetched, the CR will exclude it (unless we request CR = -1), right? The net effect would be to make the CR filter useless because you'd have set it to -1 to detect any deletions in the data you already have. Then again, maybe everyone is already filtering on CR=0 or CR=1 (at least I am) so it might not matter.
 

RedWizzard

RedWizzard
JesusFreke said:
In the back of my mind for a while, I've been thinking what we actually need is some sort of... wiki-database or something (with an appropriate api of course), with the sort of crowdsource/wiki/community-driven things you would expect. user reputation, higher editing/mod privs with high rep, etc.

Some potential solutions? I haven't actually looked at these much..

http://foswiki.org/About
http://semantic-mediawiki.org/wiki/Semantic_MediaWiki
https://code.google.com/p/database-wiki/
Click to expand...

I've had the same thoughts. I have actually played a bit with Semantic MediaWiki and my experience was that it was pretty easy to incorporate data from a database in a wiki page but it wasn't so easy to put data back into the database. I'm not sure it would work for our case. I haven't looked at the other two.

A NoSQL databases could be an option as well. I had a look at CouchDB and it looks reasonably well suited to this sort of thing: it stores JSON documents with revision history and has users and privilege levels (though not particular fine-grained I believe). The various similar systems would be reasonable alternatives (e.g. Redis).

All that said, given the possibility that FD may release an api, I think we might be best to curate TGC as it is now for say 6-12 months and see what happens. If it doesn't look like anything is coming from FD at that point then we could re-evaluate. I'm willing to help share the load if TornSoul finds it too much.
 

sngerous

sngerous
I have written some code the last days. It's planned as an enhancement for my cockpit running on my tablet and should work as a journal. The current project's name is "Auriga Log".

Prototype Video

The language is Python and the Kivy library. Kivy states to be cross-platform. I already can run the same app on Windows and on my Android tablet (Kivy launcher) without any adaption.

My vision so far is that it doesn't give any more information than the ED main game gives. No revealed systems from start which are actually unexplored, no trading information if not entered by the user himself. But the user data will persist over the lifetime of a commander and ofc collected in the db.

Inhumierer has called to provide the hardware/support for the database. I have written a test Servlet to query EDSC which worked well but is not online yet.
Later, it's planned to have a servlet as the background service for the app client. It will also query EDSC and duplicate at least the strong system data. Also, the Auriga user should have the option to enter system data which the servlet then will also push back to EDSC.

Actually, entering system/station/market data should be one big basis of Auriga Log to finally make it useful, together with giving better orientation and ovieview about your part of the galaxy. If commanders find it useful, that data could be also useful for other tools.

Ok, tell me what you think. (If you have more experience of cross-platform/overall capabilities of Kivy, Kivy on Mac/IOS, please tell me.)
 
Last edited:

wolverine2710

Tutorial & Guide Writer
wolverine2710
sngerous said:
I have written some code the last days. It's planned as an enhancement for my cockpit running on my tablet and should work as a journal. The current project's name is "Auriga Log".

Prototype Video

The language is Python and the Kivy library. Kivy states to be cross-platform. I already can run the same app on Windows and on my Android tablet (Kivy launcher) without any adaption.

My vision so far is that it doesn't give any more information than the ED main game gives. No revealed systems from start which are actually unexplored, no trading information if not entered by the user himself. But the user data will persist over the lifetime of a commander and ofc collected in the db.

Inhumierer has called to provide the hardware/support for the database. I have written a test Servlet to query EDSC which worked well but is not online yet.
Later, it's planned to have a servlet as the background service for the app client. It will also query EDSC and duplicate at least the strong system data. Also, the Auriga user should have the option to enter system data which the servlet then will also push back to EDSC.

Actually, entering system/station/market data should be one big basis of Auriga Log to finally make it useful, together with giving better orientation and ovieview about your part of the galaxy. If commanders find it useful, that data could be also useful for other tools.

Ok, tell me what you think. (If you have more experience of cross-platform/overall capabilities of Kivy, Kivy on Mac/IOS, please tell me.)
Click to expand...

Please send me a PM when its publicly available, otherwise its possible I miss your announcement. I will then add it to my 3rd party tools list.
 

sngerous

sngerous
CMDRKNac said:
sngerous that's actually a very interesting project, not-cheat mode game aid. Good idea. Meanwhile I've open my database to upload market data and can be used in a similar way if you wish (just browse where you have been), but can be updated by anyone (registered) so you may get more up to date data than you would get ingame.

Here is an example system: http://www.elitedangerouscentral.com/systems?q=LHS 331&type=system
Click to expand...

Do you base upon EDSC?

Edit: Just saw the station details. Looks pretty good. Is there any way to get raw database records?
 
Last edited:

RedWizzard

RedWizzard
TornSoul said:
The other one is the one from MBs Gamma list (correct coords) - Don't know why I didn't check the coords before...

So yeah - There's a 12 Aquarii

at both [-57.0625, 57.03125, 68.8125] (MBs Gamma list )
and
"Duplicate" : [-295, -270, 303] (GM map)
Click to expand...

One of the 12 Aquarii systems is gone. They have removed the one from MB's spreadsheet (the one at [-57.0625, 57.03125, 68.8125]).
 
Last edited:

Athan

Athan
RedWizzard said:
The duplicate 12 Aquarii is gone. They have removed the one from MB's spreadsheet (the one at [-57.0625, 57.03125, 68.8125]).
Click to expand...
No they haven't. They may simply have switched which you find on a search. I'm looking at the -57.0625, 57.03125, 68.8125 one on the in-game map right now.
 

Rhulan

R
Hey all, just a quick one to say that I've been away with work for the last month but I'm back now and fully intend to carry on working on my web application. When I left I was just starting to integrate with EDSC so I'll read all the posts I've missed and carry on with that effort.
Any feedback/ideas/improvements send me a PM or name and shame here :)
 

donpost

D
Guys I don't know exactly when this happened, but a system has MOVED sometime in the last 5 days or so.
.
The system is Sun Wen. It is in the database as being 5.93Ly from Bohmshohm but it's now 41.70Ly away..
.
EDIT: Oh you're already aware!
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom