@Majinvash I think you'd know best what the environment could do to make this challenging for you. Overwhelming system authority numbers? Jump restrictions in systems where you're public enemy number one? I dunno - you tell us.
As to getting people into Open. I'll try to keep to facts.
Some people just wanna play on their own, which is fine. Sometimes I go solo because I want to be left alone. There's no changing that when I'm in that mood.
That said, I also like to play in Open. I do hop back to solo sometimes, and here's why:
1. I've got back to my faction capital and found it crawling with enemies. That's not really fitting with lore or design - a gang of four players should not be able to take over a whole system in this way. I also suspect they'd have more challenge if they had to be more aware of the environment they were in.
2. I've found a wing that I can't beat. If none of my friends are online and I can't possibly take them on my own then I'll skip to solo. I don't want my game to require that I have friends online to get to certain places.
3. An area (CG most likely) is known to be full of PvPers who will destroy any target before an authority or friendly CMDR can respond.
There aren't many reasons there.
Things that would address these concerns for me:
1. Stronger System Authority. It really shouldn't be any other way in secure systems, as is well known by players and FD. I'll leave that there.
2. NPC wingmates? As a mechant I'd gladly pay for an escort. It's just not a thing in game.
3. Kinda the hard one to address. I'd like to be able to see CMDRs or NPC police detect the interdiction and come to help (or join in the piracy if they want). I'd like more confidence that I'm being pirated and not just blown away with no idea why. It needs more flavour than just dying, and it needs more potential outcomes than just dying. IF nothing else the variety would be cool
On a brief broader note, the challenge needs to be the same in both environments and they need to be connected. Increased pirate activity in any play mode should result in more NPC pirates in all modes. Similar for peacekeepers, traders and bounty hunters.
Hope that helps answer your question.