The AI that was introduced (briefly) in 2.1 was improved, the NPC ships were fitted with engineered weapons, and there were some significant intermittent bugs in those weapons, e.g. rail guns firing as fast as multi-cannons - the NPCs also hounded players carrying any of the new engineering commodities that were required to complete modifications. This resulted in a situation where any player trying to engineer their ships would be repeatedly attacked by engineered NPCs ship, with more challenging AI and, quite often, bugged weapons.
While a vocal subset of the player-base enjoyed it, many players did not - and I expect that Frontier's play-time statistics, before and after the introduction, showed how popular the changes were. The changes were rolled back and the requirement for commodities was removed from engineering.
Regarding a difficulty setting - we all play against the same NPCs, whether or not we play among other players. Noting that there are optional NPC challenges that far exceed the difficulty of run-of-the-mill NPCs.