ED Astrometrics: Maps and Visualizations

Update to the above. I've narrowed the problem to decoding compressed data (although encoding seems fine). While I continue to investigate the underlying problem, I have a work-around in place of having the map-maker script use BMPs to load the logos and other image elements, rather than PNGs. If that works, the maps should begin updating again soon.

It's been a while since I've had to track something like this down. What a pain... LOL.
 
So the work-arounds seem to be working, and the maps are updating again. I figured I'd write up a "post mortem" just in case anyone is interested in this sort of thing, though the problem is not actually corrected yet. I'm just working around it.

This is the sort of thing that makes me crazy, as a system administrator. I mean, worse than space madness. ;) But even so, it's also fun and interesting to do this kind of detective work. It simply gets maddening when it's almost always hitting a server or process that you want to quickly restore to service, with no immediately apparent solution. Thankfully it doesn't happen often, but there's always a risk with updating software on a system that's been humming along automatically just fine.

It all started with an OS update (yum update in CentOS) last week. I forgot to check that the images were still updating for several days, but no biggie, they're just images. :D Marx mentioned that the nebula maps weren't up to date and seemed to be missing some data. This involved two problems, first that I didn't have the spreadsheets pulling on the right schedule, but secondly that it could no longer read the XLSX file from Google Docs. It complained of some sort of stream error. I figured at first that this might be a problem with Google mangling it or using a file feature that the Perl modules don't understand. I was only using Excel format because I've worked with the Perl module for that before and it contains all of the sheets in one file.

At this point I also noticed that the other maps hadn't been updating due to a PerlMagick (ImageMagick) error, so I figured that was a separate problem, and focused on the spreadsheet first. I switched it to use ODS format instead, and got a nearly identical error. (What?) So to sidestep this first issue, I switched to CSV format (no reliance on Perl modules) and this script started crashing on the image manipulation too.

Now here's where PerlMagick (a wrapper for the ImageMagick libraries) can be irritating. The crash and error message usually happen long after the actual problem has occurred. Like, maybe hours later, in the case of the map-maker script. It would die on executing a Composite or Write command on one of the images, saying that no image was present. Turning on some additional warning messages (that might otherwise be invisible), I discovered that it was actually failing right at the start, when it tried to load PNG image assets. Most of the other operations would happily (and silently) try to perform operations on an empty data set. The warnings indicated that it thought that the PNGs were corrupted on disk. I knew that wasn't true.

So what do all of these problems have in common? They're all perl modules that are failing to read complex data formats and parse them. I figured the most likely culprit would be compression, since I'm not sure what else they could be doing in common that wouldn't also affect everything else running in Perl. As a test, I started switching the scripts to use BMP and TGA formats internally, since they're uncompressed (though TGA has a compression option, it's rarely used). This worked, thus giving me more evidence that it's an compression decoding issue. The command-line ImageMagick still works, and converts back to PNG before uploading the images to the web server.

So what I know is this-- low level compression in Perl is failing, but everything else is still running fine. I've tried rolling back the linux packages that pertain to Perl, ImageMagick, and Zlib, but nothing has solved this yet. In the meantime, the scripts are doing fine working with other formats. So at least I can relax now. :D
 
Last edited:
I am certainly interested in post mortems. Not that I work in that profession but more than once reading such reports helped me with a problems some time later :)
 
Turning on some additional warning messages (that might otherwise be invisible), I discovered that it was actually failing right at the start, when it tried to load PNG image assets. Most of the other operations would happily (and silently) try to perform operations on an empty data set. The warnings indicated that it thought that the PNGs were corrupted on disk. I knew that wasn't true.
That certainly sounds annoying. You'd think a problem as serious as failing to load a file would merit a visible-by-default error message at least! 🤔

Marx mentioned that the nebula maps weren't up to date and seemed to be missing some data. This involved two problems, first that I didn't have the spreadsheets pulling on the right schedule, but secondly that it could no longer read the XLSX file from Google Docs.
I thought you were pulling POI data directly from EDSM now, but I guess not? Comparing your interactive map to the EDSM one, the sets of POIs are close but not identical. (I think your POIs are a strict subset of the EDSM ones, but it's hard to be sure with all the overlapping pins and such.)
 
I thought you were pulling POI data directly from EDSM now, but I guess not?
He was talking about the nebula data for the nebula maps (see the Maps and Charts, not the interactive map), which is pulled from here. Some real nebulae are still missing from the map, because I forgot to fill out the reference systems - should be done relatively soon though.
 
Last edited:
That certainly sounds annoying. You'd think a problem as serious as failing to load a file would merit a visible-by-default error message at least! 🤔
Now that I know that it does this, I'm tempted to modify every..single..function call to perlMagick with a wrapper that checks to see if it threw a warning, and display it myself. At least problems might be easier to identify in the future if I do that.
 
I've switched the interactive map over to use Leaflet instead of the Google Maps API. I did this mainly because Google did something that I was afraid they might do-- They sometimes shut off your access to force you to acknowledge billing conditions, etc, even though I'm well below the threshold where they would start charging me. Leaflet, on the other hand, is free and open source, and is a popular choice for game related maps.

As it turns out Leaflet also provides a cleaner, simpler API. That was a pleasant surprise.

I have all of the functionality converted over, except for the mouse-over coordinates display. I'll work on that later. For now though, I have everything else in there. The map types and POI marker overlays are all in the layers button in the upper-right, and the player position upload is in the small button under the zoom controls on the upper left.

If anyone still needs to use the mouse-over coordinates, you can still pull up the Google version here (for now): https://edastro.com/galmap/googlemaps.html

The main interactive map is still at the same URL as before:

 
I've switched the interactive map over to use Leaflet instead of the Google Maps API. I did this mainly because Google did something that I was afraid they might do-- They sometimes shut off your access to force you to acknowledge billing conditions, etc, even though I'm well below the threshold where they would start charging me. Leaflet, on the other hand, is free and open source, and is a popular choice for game related maps.

As it turns out Leaflet also provides a cleaner, simpler API. That was a pleasant surprise.

I have all of the functionality converted over, except for the mouse-over coordinates display. I'll work on that later. For now though, I have everything else in there. The map types and POI marker overlays are all in the layers button in the upper-right, and the player position upload is in the small button under the zoom controls on the upper left.

If anyone still needs to use the mouse-over coordinates, you can still pull up the Google version here (for now): https://edastro.com/galmap/googlemaps.html

The main interactive map is still at the same URL as before:

Looks good - the leaflet version feels more responsive, too!
 
I wasn't expecting to get it done today, but I have the mouse-over coordinates working too. So the functionality should all be back to where it was. I moved the coordinate display to the lower left so that it plays nice with browser extensions that add icons/whatever in the lower right.
 
I've switched the interactive map over to use Leaflet instead of the Google Maps API. I did this mainly because Google did something that I was afraid they might do-- They sometimes shut off your access to force you to acknowledge billing conditions, etc, even though I'm well below the threshold where they would start charging me. Leaflet, on the other hand, is free and open source, and is a popular choice for game related maps.
Good move :) … but as a member of the FSF I have to say that ;) … well actually I also want to say that because in my opinion it is the right choice.

And also good work because it looks and feels like before :)
 
Btw. thank you for telling that you are using Leaflet now. I didn't know about this software since its functionality is outside of what I usually do. But I've started using it in a project at work now and thus can deliver much more than I originally thought :)
 
Last edited:
Awesome, glad it's helping! I've been pretty happy with Leaflet so far. The only things I felt it lacked, were an easy way to add simple buttons, and it didn't do a cross-fade when changing base map layers. Both were easily fixed with plugins.
 
I added a spreadsheet for planets with zero distance from arrival. I've only been storing integer numbers for this, and I think that's how I'm getting them all from EDSM anyway, so these are probably all rounded down from very small numbers, like < 0.5 light seconds. Right now the list is only 1029 planets, so it makes me wonder if actually EDSM was rounding up, and these actually got stored as "0" erroneously? Who knows. It's enough of an open question that I thought it was worth providing the list.

 
After the discussions in the data cleanup thread, it got me wondering how often EDSM deletes or merges entries (which would still be deletions as far as I'm concerned). I might have a lot of orphaned data if EDSM does a lot of cleanup. My suspicion has always been that I would gradually accumulate cruft at a slow rate. But now I'm thinking EDSM does a lot more of this than I was aware.

I took one of the recent full EDSM dumps and did a comparison, and wrote a list of IDs for database rows that I have, that no longer correspond to something in the dumps. The results are much larger than I expected. This is taking into account the dates so that I don't include anything newer than the large dump. Here are the numbers:

~/elite$ wc -l removals-*
866,938 removals-planets.txt
327,033 removals-stars.txt
2,641,018 removals-systems.txt
3,834,989 total

(commas added for clarity)

So the question is, do I delete them? Do I keep them, but modify my spreadsheet scripts to ignore those entries? Or just say F-that, and leave it all alone? ;) I'm not sure what I want to do yet.
 
Top Bottom