To me it seems the developer team took inspiration from other MMOs, and slapped a gear upgrading system on top of a game that doesn't actually need it. Which seems an enormous waste of resources and development time.
Well, that, and people were demanding from pretty much day 1 for a crafting system of some kind. Which usually means some sort of upgrade system.
Some MMO style games get round this by scaling enemies, but then you end up with silly situations like level 100 spiders and level 1 Dadera, because the game knows at higher levels you have better kit as well and adjusts based on level.
ED suffers a problem due to not everyone having Horizons (especially Mac owners who can't have Horizons) and those who don't want... again, there would have to be some gamified elements to determine whether the person had done any engineering and if so, how much, to make appropriately challenging enemies.
Then on top, we know FD support a charity for gamers with disabilities who would find playing against super AI difficult.
Players also have a wide range of skills. People can scream git gud as often as they want, but everyone has their own skill cap.
Combine this with a shared galaxy and in Open shared enemies (you might end up facing opposition someone else has spawned), and it can be tricky to balance right.
Two tactics can be considered here for players:
1) Understanding you can't and perhaps shouldn't be able to beat all opponents. This is potentially an argument in favour of increased difficulty of AI, but i think we all know where this would lead. The bottom tier players would find themselves locked out of combat, resulting in a massive wave of discontent and negative reviews.
2) Live with the current NPCs and stop flying top tier ships all the time. If you want a challenge, there are challenges to be had. I've got a bit bored with macing most NPCs (some could still give me trouble though) in my bigger Engineered ships, and i've now sold most of them, all of the combat ones, and now only have Cobras for most activities, and its great fun. Really challenging, even in an engineered ship. Low end still easy, but a wing of enemies, even weaker ones, can still provide a challenge.
So, barring FD upping the difficulty, the way to deal with it is stop flying god ships, and give yourself a challenge.
If FD did up the difficulty, it would be annoying for me now, since i don't really want to go back to the big ships.