I don't have any particular ideas* to contribute right now, but a rather general thought to be considered:
Any attempt to encourage someone to go into Open Play, who would rather have preferred Solo or Private Group anyway regardless of any risk/reward mechanics, will just shift the current feeling of frustration onto them. At the moment, those who want to play in Open Play but feel discouraged because they can reach better ratios of credit/hour, merits/hour, CG-contribution/hour, in Solo are rightfully frustrated. Most ideas to alleviate it simply shift that very frustration onto those who would prefer not to participate in Open Play regardless.
Fundamentally, what we want, or should want, is for everyone to pick the game mode they prefer for the sole reason that they prefer the principle workings of this mode: meet all kinds of players: Open Play; meet only select people of a specific mindset: Private Group; play it as a single-player game: Solo. We have now people who pick a mode they rather would not but feel the game mechanics or rewards push them into it, from their Open Play to Solo; plus lots of players in Open Play who are frustrated but do not switch to Solo; and players in Open Play who couldn't care less about such considerations.
If you simply reverse that situation, you will find players who would prefer Solo, but switch to Open because the new game mechanics or rewards make them feel like they miss out or suffer from (relatively) reduced rewards etc; you will find players who stay in their preferred Solo more but are disgruntled that they get lesser rewards just because they prefer a different play style; and you will find players who have been playing in Solo all along and don't care whether their rewards could be better in a different mode.
In each scenario, the status quo as well as the reversed status quo I just outlined, the third type of player is irrelevant for our considerations, and what we are left with is possibly trading the frustration of one type of player for the frustration of a different type of player.
I hereby claim that some who have been asking for higher rewards in Open Play either did not consider this yet, or are being rather selfish to simply wish someone else shall suffer that same frustration instead; plus a third group who is under the illusion that any form of "higher reward in Open Play" solution can be balanced in such a way that the frustration vanishes entirely. The best you could hope is a 50:50 split between frustrated people preferring Open Play but still feeling discouraged from it, and frustrated people preferring Solo mode but feeling discouraged from it, too.
*That said, I wish to throw an idea into the ring which I have mentioned before, very early on, and would like to be reconsidered: to minimize the risk of loss from encounters with other players. That could mean, for example, a greatly discounted (90%) rebuy cost when killed by other players, all cargo not jettisoned manually or through hatch breakers insured (at the same discounted rate) as well, and no w&t effect from interdictions by other players. Plus, bounty vouchers, combat bonds, exploration data, missions etc. should not be lost or failed when being killed by another player.
It'd still be some loss, and always some loss of time, but if we could get away from the prospect of devastating losses from PvP, the (justified) fear of losing hours of gameplay effects in a single random encounter (especially when it is not justified or mandated directly by the gameplay, e.g. one power vs a hostile power), then many more players would be willing to take that very risk.
It is for this very reason that in most MMOs, against other players the regular penalty for dying (e.g. money loss) does not apply at all, and even regular equipment wear&tear mechanics are often exempt from any actions done by other players. If, for example, in WoW an opposing faction player kills you anywhere, you have to respawn the same way as if an NPC killed you, but you suffer no loss of money and thus, the most overall effect any other player have on you, is to temporarily hinder you from something (e.g. reaching a location) or slow you down in your quest, but they could never reset you back to an earlier state of less money, equipment, etc.
In terms of the idea of rebalancing risk/reward scenarions on Open Play, such a change would reduce the risk side of the equation, but leaving the reward side untouched. Therefore, a player in Solo mode and a player in Open Play mode reap the same rewards for the same actions, but the extra possibility of hostile player encounters is mostly mitigated to keep things somewhat even.