There was a lot more effort put into those games, mainly because they were so limited. You couldn't hide a bad game behind flashy graphics and memory limitations were such that you really had to find groundbreaking solutions to solve them. Braben famously did this, with Hill, in the original game, generating an entire galaxy, positions, attributes and descriptions using a clever use of Fibonacci numbers to simulate 'randomness' - it is probably, to this day, still more advanced a procedural algorithm than you'll find in many more modern titles out there.
Personally I blame the rise of middle management. Design of games is increasingly the remit of talent-less mid-level cogs that produce nothing of worth.