This is a fantastic thread, and the OP hits the nail on the head with what's wrong with this game.
I too, way back at the Beta stages, thought we were just seeing a vertical slice of the real game - that the Betas were basically Frankengames made up of little bits of features that FDEV wanted tested at the time.
I was surprised and dismayed at the Release on 16th December 2014, when it turned out that yes - what I saw before then is basically what we got on that date.
Anyway, reading through this thread and it triggered an idea.
One of the limiting factors seems to be because ED is based not on a Central Giant Server creating the universe, content, and running everything, but on P2P connections with a smaller server or server farm supposedly running a background simulation.
Would it not be a good idea to turn the 'weakness' of P2P into an advantage?
Think about it - you have thousands of ED game clients running at any one time, more or less on a P2P network. Why not utilise these clients as a distributed computer system? You could basically turn every game client into a node that computes the universe simulation, and indeed, you could in theory create persistent AI/NPC characters via this method.
Sounds like a daunting task, and I've no doubt it would be, but for example, say you wanted a persistent NPC called 'Dread Pirate Roberts' - you'd utilise this theoretical distributed computing system to compute his character's progress. So you'd allocate say 100 or so ED game clients to this task. Any client that drops off (e.g. player has finished for the day and logs off, or just becomes disconnected for whatever reason), and the work unit assigned to them would get reassigned to some other game client. After all work units have been computed, the result is used to update this character's overall status. Any single event, for example, he interdicted a player ship to pirate it, could be updated by a single node (e.g. the node of the player being interdicted), and the result of the event could be fed back to the distributed computation.
You could use this technique for the wider universe as well.
I have no idea how many players are online in ED at any given time, but I'm pretty sure there are enough in any one 24-hour period to be able to produce a pretty powerful distributed computational system, continuously.
Just an idea - I don't know how practical this would be - it certainly seems to work for other distributed computing tasks, and has been for years. Who knows, perhaps this is something that FDEV could be doing groundbreaking work on.