When it comes to Open Mode, I think one thing would solve all the problems without breaking entire game and pleasing both sorts of players:
Give +50% or even +100% credit bonus to every activity done in Open Mode.
There are three reasons why I consider ideas such as this to be extremely poor ones.
1) Unless a player logs in during their local prime time, the way instancing works in this game, they are effectively playing in solo. Even in busy hubs like CGs, Engineer sites, and Shinrarta Dezhra, they're not likely to encounter very many players, let alone PKs. The "danger" of Open varies wildly based on the players location and when they play.
2) I have long maintained that choosing open should be the choice of a player, free of influence like bonuses to playing in Open, or penalties for playing in other modes. Freely choosing Open simply attracts a larger number of players who I consider
fun to play with. I generally gravitate to MMOs similar to Open Mode Elite: Dangerous, and the amount of unfun behaviour (combat logging, player-killing, harassment, cheating, and overall jerkish behavior) is much lower than I'm used to. I still find it incredible to believe that a
significant majority of players freely choose Open, when in my experience games similar to Elite: Dangerous either loose so many players they have to shut down, or inevitably bite the bullet and put in a PvP switch. I attribute this phenomenon to Frontier's brilliant decision to allow players to choose who they
want to play with.
3) Many firewalls block most peer-to-peer ports by default at the highest security settings, requiring the player to go into their firewall settings and grand ED
special permission to open P2P ports. If a player wants the benefits of Open, but still wants that Solo experience, then they have an easy way to do. And, of course, more knowledgeable players can simulate a poor connection to any unknown player, allowing them to effectively have a private group in Open.
I've long maintained that any additional rewards in Open should not be based on simply playing in Open, but the level of player opposition they face, ideally done by allowing players to pledge their support of factions, both minor and major, similar Powerplay. If done correctly, a reward system should encourage more symmetrical PvP, which would attract a greater audience than the current heavily asymmetrical PvP we have outside of prearranged duels. This would require
significant development resources to implement, both the system itself, and to ensure that such a system is difficult to exploit via player collusion.
edit: TLDR; Reward the PvP, not the mode you play in. It's more fun for
everyone that way.