So the work-arounds seem to be working, and the maps are updating again. I figured I'd write up a "post mortem" just in case anyone is interested in this sort of thing, though the problem is not actually corrected yet. I'm just working around it.
This is the sort of thing that makes me crazy, as a system administrator. I mean, worse than space madness.
But even so, it's also fun and interesting to do this kind of detective work. It simply gets maddening when it's almost always hitting a server or process that you want to quickly restore to service, with no immediately apparent solution. Thankfully it doesn't happen often, but there's always a risk with updating software on a system that's been humming along automatically just fine.
It all started with an OS update (yum update in CentOS) last week. I forgot to check that the images were still updating for several days, but no biggie, they're just images.
Marx mentioned that the nebula maps weren't up to date and seemed to be missing some data. This involved two problems, first that I didn't have the spreadsheets pulling on the right schedule, but secondly that it could no longer read the XLSX file from Google Docs. It complained of some sort of stream error. I figured at first that this might be a problem with Google mangling it or using a file feature that the Perl modules don't understand. I was only using Excel format because I've worked with the Perl module for that before and it contains all of the sheets in one file.
At this point I also noticed that the other maps hadn't been updating due to a PerlMagick (ImageMagick) error, so I figured that was a separate problem, and focused on the spreadsheet first. I switched it to use ODS format instead, and got a nearly identical error. (What?) So to sidestep this first issue, I switched to CSV format (no reliance on Perl modules) and this script started crashing on the image manipulation too.
Now here's where PerlMagick (a wrapper for the ImageMagick libraries) can be irritating. The crash and error message usually happen long after the actual problem has occurred. Like, maybe hours later, in the case of the map-maker script. It would die on executing a Composite or Write command on one of the images, saying that no image was present. Turning on some additional warning messages (that might otherwise be invisible), I discovered that it was actually failing right at the start, when it tried to load PNG image assets. Most of the other operations would happily (and silently) try to perform operations on an empty data set. The warnings indicated that it thought that the PNGs were corrupted on disk. I knew that wasn't true.
So what do all of these problems have in common? They're all perl modules that are failing to read complex data formats and parse them. I figured the most likely culprit would be compression, since I'm not sure what else they could be doing in common that wouldn't also affect everything else running in Perl. As a test, I started switching the scripts to use BMP and TGA formats internally, since they're uncompressed (though TGA has a compression option, it's rarely used). This worked, thus giving me more evidence that it's an compression decoding issue. The command-line ImageMagick still works, and converts back to PNG before uploading the images to the web server.
So what I know is this-- low level compression in Perl is failing, but everything else is still running fine. I've tried rolling back the linux packages that pertain to Perl, ImageMagick, and Zlib, but nothing has solved this yet. In the meantime, the scripts are doing fine working with other formats. So at least I can relax now.
