[CLOSED] EDDB - a site about systems, stations, commodities and trade routes in Elite: Dangerous.

[themroc]

Hi there just wanted to let you know that I've been playing with creating a mobile app using your data via htto (thanks for doing this project!!) - i couldnt get the contact form to work so I'm posting here...

Indeed! Thx for pointing it out, got to fix that.

thought it best if I point out the json isn't gzipped - which makes such large files pretty heavy going.

Secondly it looks like you update the files on a chron job whether there have been changes or not (i.e. the last modified updates nightly) which may result (if use is widespread) in heavy server loads for you.

I'm aware of this bandwidth/load issue. Generally bandwidth and load is no problem on the server I'm using. But I will add gz version for people with crappy internet.

Dayly changes in stations and systems are very likely, so the full file has to change of course. And again, traffic is unlimited and the 100MBit I'm running on should be more than enough to server the site, the json and an upcoming REST API.

Also... the json files are only relevant for developers. So I don't expect a widespread usage. Of course I'm having an eye on my logs very carefully and will react accordingly. Start small and react on demand ...

And finally you have a couple of Stations listed as [.....] 0rbital (number zero not O) - not sure if this is correct assuming its an OCR thing but thought I'd point it out while I was here

That I have to fix indeed.

Updating the Maddavo data will end up in eddb eventually. Additional to this, eddb will introduce a data manipulation/creation interface for people, a kind of backend thing, very soon. Maddavo data will still be imported, but eddb will have unique data from that point on.

 

[themroc]

Hi there just wanted to let you know that I've been playing with creating a mobile app using your data via htto (thanks for doing this project!!)

Sorry I didn't read it fully in the first place.
Well in this case you should mirror the data to your place and do the incremential update service yourself. My JSON dump is not meant to serve as feed for a widespread app usage. It's more as the base for whatever developers do with it. If you have more questions, please contact me on IRC (weblink here).

i couldnt get the contact form to work so I'm posting here...

fixed!

 

[MagmaiKH]

Cool initiative. A question about the database sql scheme. Did you create that from scratch? The reason I ask this is because of the thread "[GENERAL] Common Use SQL Game Database Source", in which an extensive scheme has been described/discussed?

I DO like the idea of validating (prices) data but is the EliteOCR (and RegulatedNoise) OCR data that bad? Afaik EOCR does ONLY update data to EDDN (and create a .BPC file for Slopeys the BPC) when its at least 1920x1080. Otherwise its not uploaded. I haven't checked the data in EDDN myself so I hope you interpret this a question and not some kind of rude remark. How much of the received data was 'incorrect' after it has been validated by your tool?

QAA deliberately ignores all data from RegulatedNoise because it's SNR is so bad.

- - - - - Additional Content Posted / Auto Merge - - - - -

Hello Commanders,
Updates:

• 19-01-15: As of now the core data of eddb gets dumped as JSON every night for everybody to download.

Awesome themroc - thanks!

Would it be possible to add line-breaks in the .json files? It would make it easier on my parser to go line-by-line rather than search for chunks.

 

[themroc]

Would it be possible to add line-breaks in the .json files? It would make it easier on my parser to go line-by-line rather than search for chunks.

You're not the first one requesting this. But I have to wonder... who still writes his own JSON parser? Every modern language has a built in JSON parser which will totally be able to read my data. What are you using?

 

[Cedhric]

Looks very good, themroc. Just one suggestion - could you put a button on the web form to swap the Origin and Target stations? That would make it very quick to suggest a 2-station run for trading.

 

[themroc]

Looks very good, themroc. Just one suggestion - could you put a button on the web form to swap the Origin and Target stations? That would make it very quick to suggest a 2-station run for trading.

Round trip route finding as well as multi stop routes is in the works. Before finishing that I wanna release a backend first, so data can easily be corrected. This is highly important since the data quality is not perfect. So please be patient

 

[MagmaiKH]

You're not the first one requesting this. But I have to wonder... who still writes his own JSON parser? Every modern language has a built in JSON parser which will totally be able to read my data. What are you using?

C# but I can't report progress if the parser has to read it all in one phat chunk.

 

[themroc]

C# but I can't report progress if the parser has to read it all in one phat chunk.

Does plain JSON parsing take that long for you?
However, generally that doesn't make any sense for JSON. It's not CSV where one line equals one set of data. Since JSON can have sub objects, a one line solution is only possible manually knowing how the data looks like.

How would you be able to parse better with such a version?

 

[Cedhric]

Round trip route finding as well as multi stop routes is in the works. Before finishing that I wanna release a backend first, so data can easily be corrected. This is highly important since the data quality is not perfect. So please be patient

Brilliant, many thanks. And I'm not rushing you. You've got five minutes.......

 

[wolverine2710]

Very good tool with LOTS of potential. Is this the new TGC - minus coordinates calculations?

Added to the OP of the EDDN thread, section "Tools created for EDDN and/or using it"

Name: eddb - Elite: Dangerous Database
Author: themroc
Description: Routeplanner.
Website: eddb
Source available: No
Thread: eddb - a site about systems, stations, commodities and trade routes in Elite: Dangerous.
Compatible with Release: Yes
EDDN support: SUB
Value added service: Archiving
Comments: Much more functionality is planned, like a REST-API.

 

[themroc]

minus coordinates calculations?

I think as soon as a backend for eddb is in place and people can add/correct data, a coordinates calculator will likely be implemented. There's a lot on my roadmap right now and i have limited time...

 

[MagmaiKH]

Does plain JSON parsing take that long for you?
However, generally that doesn't make any sense for JSON. It's not CSV where one line equals one set of data. Since JSON can have sub objects, a one line solution is only possible manually knowing how the data looks like.

How would you be able to parse better with such a version?

Oh ... I see your point if it's only one blob.

I was thinking each [ ... ] chunk of data would have it's own line and there would be many chunks in one .json file - like one per station or something.

 

[wolverine2710]

I think as soon as a backend for eddb is in place and people can add/correct data, a coordinates calculator will likely be implemented. There's a lot on my roadmap right now and i have limited time...

I don't mind when eddb does not have a distances to coordinates calculator. TGC aka edsc api can be used for that. Its working as it should THOUGH TornSoul has been MIA since the 16th of December 2014, which is suboptimal in case of problems. In principle RedWizards tool or any of the other tools could take of that - calculating coordinates. Afaik those tools can check if a coordinate could be reliable calculated. When those tools can upload the coordinates to eddb its all well. One disadvantage: no way to spot pranksters. But you could of course implement their algorithms in your tool. Remember: calculating coordinates is NOT trivial. It took months before the algorithms were perfected - due to reducing to only 2 decimal precision in the Galaxy Map in later beta's.

Tracking to many tools - can't remember it all. What language are you using? With luck one of the commanders is using the same language ;-)

 

[wolverine2710]

Oh ... I see your point if it's only one blob.

I was thinking each [ ... ] chunk of data would have it's own line and there would be many chunks in one .json file - like one per station or something.

Wrt memory usage. For Java there are sax parser like JSON parsers. Probably also for other languages. With XML you have DOM (reads all in memory) and SAX like parsers (NOT storing all in memory).

 

[themroc]

I don't mind when eddb does not have a distances to coordinates calculator. TGC aka edsc api can be used for that. Its working as it should THOUGH TornSoul has been MIA since the 16th of December 2014, which is suboptimal in case of problems. In principle RedWizards tool or any of the other tools could take of that - calculating coordinates. Afaik those tools can check if a coordinate could be reliable calculated. When those tools can upload the coordinates to eddb its all well. One disadvantage: no way to spot pranksters. But you could of course implement their algorithms in your tool. Remember: calculating coordinates is NOT trivial. It took months before the algorithms were perfected - due to reducing to only 2 decimal precision in the Galaxy Map in later beta's.

Tracking to many tools - can't remember it all. What language are you using? With luck one of the commanders is using the same language ;-)

What will we do without TGC? Who knows whats gonna happen...
Anyone knows if people still update TGC? Maddavo said he is receiving some new system data. So maybe his list is the new "master" list? For sure we do need one central point where everybody knows its being spread like EDDN is doing with prices. If TGC is abandoned, it gives me a bad feeling.

I'm not implying that eddb has to be the place for systems. Just pointing out a possibility if all goes wrong or the majority thinks it's a good idea

 

[sutex]

Can EDSC be used by RegulatedNoise DEV to incorporate , It's at GIT , https://github.com/stringandstickyta...Noise/releases. given us uses of RN are relying on good data to build our own local saved databases

 

[Smacker]

Very nice I have been investigating this. Some Station Corrections:

Renaming Denberggateway to Den Berg Gateway in 37 Librae [EDDN]
Postfix Spacing Chelistation => Cheli Station in Atins [EDDN]
Postfix Spacing Malzbergstation => Malzberg Station in Auradjan [EDDN]
Postfix Spacing Tapinascity => Tapinas City in Auradjan [EDDN]
Postfix Spacing Tanibase => Tani Base in Bamin [EDDN]
Postfix Spacing Vetulanihub => Vetulani Hub in BD-04 797 [EDDN]
Postfix Spacing Herreshofffreeport => Herreshoff Freeport in G 123-49 [EDDN]
Postfix Spacing Filipchenkogateway => Filipchenko Gateway in Gendenwitha [EDDN]
Postfix Spacing Kanwarterminal => Kanwar Terminal in Gendenwitha [EDDN]
Renaming Ooicolony to Doi Colony in Guaras [EDDN]
Postfix Spacing Parazynskiport => Parazynski Port in HIP 1427 [EDDN]
Renaming Bokeniport to Bokeili Port in HIP 65636 [EDDN]
Postfix Spacing Hannuport => Hannu Port in Kedo [EDDN]
Postfix Spacing Meuccisurvey => Meucci Survey in Koskit [EDDN]
Postfix Case Tasaki freeport => Tasaki Freeport in LHS 2819 [EDDN]
Postfix Spacing Maskelynesettlement => Maskelyne Settlement in LHS 3728 [EDDN]
Postfix Spacing Lilienthalcity => Lilienthal City in LHS 3728 [EDDN]
Postfix Spacing O'connorsettlement => O'connor Settlement in LHS 53 [EDDN]
Postfix Spacing Staffordplatform => Stafford Platform in LHS 53 [EDDN]
Postfix Spacing Chargaffport => Chargaff Port in LTT 16218 [EDDN]
Postfix Spacing Aliport => Ali Port in Mannodava [EDDN]
Renaming Kavanoiport to Kavandi Port in Pataheimr [EDDN]
Postfix Spacing Godeldock => Godel Dock in Quivira [EDDN]
Postfix Spacing Alihub => Ali Hub in Rana [EDDN]
Postfix Spacing Barsantiport => Barsanti Port in Sadhen [EDDN]
Postfix Spacing Becquerelcolony => Becquerel Colony in Shangdi [EDDN]
Postfix Spacing Alphonsihub => Alphonsi Hub in Siluror [EDDN]
Postfix Spacing Boroagearena => Boroage Arena in Soturicari [EDDN]

 

[wolverine2710]

Can EDSC be used by RegulatedNoise DEV to incorporate , It's at GIT , https://github.com/stringandstickyta...Noise/releases. given us uses of RN are relying on good data to build our own local saved databases

For starters, you meant edsc and not eddb?

Not totally sure what you mean. Edsc is a tool to input distances for a system which does not have coordinates yet. It (website GUI) helps with suggestions for good reference points and in the end tells you if all is OK and then the coordinates can be submitted to TGC (The great collector) aka edsc api. Unfortunately Tornsoul did not create a domain for TGC so their is confusion. Sometimes edsc is mentioned, sometimes TGC. The TGC API allows you to download coordinates (and distances) AND you can upload distances with the API. In the background TGC periodically checks all distances and decides if enough good data is available to calculate a new coordinate. After a while that new coordinate is inserted into the coordinates list and can be retrieved with the API.

Usage of TGC is for everyone so RN can use the data. I thought it would already use them, but perhaps RN gets the data from another tool. That other tool could be maddavo's list or TD's list which is very likely to use the TGC. Or perhaps RN is just using the list with almost 20K coordinates supplied generously by Michael Brookes. Note: Tornsoul has not released his code.....

 

[wolverine2710]

Very nice I have been investigating this. Some Station Corrections:

Renaming Denberggateway to Den Berg Gateway in 37 Librae [EDDN]
Postfix Spacing Chelistation => Cheli Station in Atins [EDDN]
Postfix Spacing Malzbergstation => Malzberg Station in Auradjan [EDDN]
Postfix Spacing Tapinascity => Tapinas City in Auradjan [EDDN]
Postfix Spacing Tanibase => Tani Base in Bamin [EDDN]
Postfix Spacing Vetulanihub => Vetulani Hub in BD-04 797 [EDDN]
Postfix Spacing Herreshofffreeport => Herreshoff Freeport in G 123-49 [EDDN]
Postfix Spacing Filipchenkogateway => Filipchenko Gateway in Gendenwitha [EDDN]
Postfix Spacing Kanwarterminal => Kanwar Terminal in Gendenwitha [EDDN]
Renaming Ooicolony to Doi Colony in Guaras [EDDN]
Postfix Spacing Parazynskiport => Parazynski Port in HIP 1427 [EDDN]
Renaming Bokeniport to Bokeili Port in HIP 65636 [EDDN]
Postfix Spacing Hannuport => Hannu Port in Kedo [EDDN]
Postfix Spacing Meuccisurvey => Meucci Survey in Koskit [EDDN]
Postfix Case Tasaki freeport => Tasaki Freeport in LHS 2819 [EDDN]
Postfix Spacing Maskelynesettlement => Maskelyne Settlement in LHS 3728 [EDDN]
Postfix Spacing Lilienthalcity => Lilienthal City in LHS 3728 [EDDN]
Postfix Spacing O'connorsettlement => O'connor Settlement in LHS 53 [EDDN]
Postfix Spacing Staffordplatform => Stafford Platform in LHS 53 [EDDN]
Postfix Spacing Chargaffport => Chargaff Port in LTT 16218 [EDDN]
Postfix Spacing Aliport => Ali Port in Mannodava [EDDN]
Renaming Kavanoiport to Kavandi Port in Pataheimr [EDDN]
Postfix Spacing Godeldock => Godel Dock in Quivira [EDDN]
Postfix Spacing Alihub => Ali Hub in Rana [EDDN]
Postfix Spacing Barsantiport => Barsanti Port in Sadhen [EDDN]
Postfix Spacing Becquerelcolony => Becquerel Colony in Shangdi [EDDN]
Postfix Spacing Alphonsihub => Alphonsi Hub in Siluror [EDDN]
Postfix Spacing Boroagearena => Boroage Arena in Soturicari [EDDN]

Great you took the time to check this. Afaik currently only EOCR and RN are uploading to EDDN atm. The spacing issue is most likely coming from RN. The 1.8 release notes state: "Remove incredibly-dumb bug that has been eating the spaces in station names... d'oh..."

 

[wolverine2710]

What will we do without TGC? Who knows whats gonna happen...
Anyone knows if people still update TGC? Maddavo said he is receiving some new system data. So maybe his list is the new "master" list? For sure we do need one central point where everybody knows its being spread like EDDN is doing with prices. If TGC is abandoned, it gives me a bad feeling.

I'm not implying that eddb has to be the place for systems. Just pointing out a possibility if all goes wrong or the majority thinks it's a good idea

I probably didn't express myself well enough. I would LOVE to have eddb TGC functionality for uploading distances and then calculating coordinates. I could imagine you would give this a low priority because there is a lot on your plate wrt things to implement.

YES, there should be at least one reference tool which all others can rely on.

I believe there is a similarity between edsc and eddb. Both haven't made the code public, which means in case of a problem (RL issues, abandoned tool) the community has a problem. Are you planning on releasing the code?