Respectfully, I don't think you really understand the mindset of people who want a PvE mode. This wouldn't make any difference to my personal choice to stay out of Open, and I doubt it will change anyone else's mind either.
Reasons
I will never initiate combat against another player. I am happy to be bound to this statement by gameplay mode. I don't want other players that have visibility of me to have any options to break that either.
Whatever punishment you devise, there will always be some players to whom it is not a punishment; fines have no impact on the rich, kill-on-sight orders don't affect people who never want to dock with this faction, even a "literally anyone in the galaxy will impound your ship for 7 days" means nothing to a person with 14 kitted out PK ships. You could put a PvP bounty on their head that never diminishes and is an actual meaningful amount to other players, but they'd still just have another ship ready to hop into.
ED is stuffed with insurance mechanisms to ensure that if a player suffers a loss or even commits to a wetwork contract without thinking about consequences, it doesn't reset them to day 1 or have lasting impact - and those mechanisms work for gankers as well as peaceful players. If you did find a punishment that was an actual deterrent, it would become something that gankers would try to bait innocent players into. Wait for someone to be in combat with an NPC and fly through their line of fire. Now the other player took the first shot, is PvP enabled, and is the one who gets punished by the local police for starting the fight.
Nothing short of a fully restricted PvE mode could tempt me into Open. I strongly suspect that the big majority of PvE players feel the same. This feels like something you could put weeks of dev time into and discover only a hundred players ever changed their behaviour, and would I urge to you at least invest in a community poll before committing to it.
You do not of course, have to care at all. It's your game, you don't have to have a PvE Open mode. But if you are interested in getting solo players into Open; do please try to think about it from their perspective, and deliver a system that they want to see, rather than one that they'll continue to ignore.
Respectfully I don't think you've considered the mindset of people who currently play in open and enjoy it (not all of whom are gankers). I fully understand and sympathise with people who don't want to PvP but keeping players with diametrically opposed preferences separated is better than putting them together. Two main points:
1) P2P instancing means that players can only ever be in the same space as a finite, quite small, number of peers. If you added PvP toggles and invulnerability it would be possible to be instanced exclusively with people who you can't interact with in the way you desire.
This goes both ways. As a fan of the risks of Open I don't want to be instanced with invulnerable ships I can't interact with, nor do I want a prospective attacker to have to ask my permission before attacking me. This damages my enjoyment and suspension of disbelief. Equally if you're the only PvE toggled player in an instance of salty PvPers who are frustrated at your presence you're unlikely to get much worthwhile co-op.
The best solution is to make sure you get where you want to be: instanced with like-minded players to casually co-op with no risk, while I get where I want to be: where there's risk. Because of capped instancing it's not beneficial to water down either pool.
2) Elite has competitive systems that extend beyond individual fights. The BGS and Powerplay are (against all odds!) enjoyed by thousands of players and present in-game stakes and context for conflict.
As a player who participates in the BGS I'm aware that often players will be working against my faction in solo mode. This knowledge doesn't fill me with joy, but neither does it actively frustrate me because I can't see those players doing it. If people were able to affect the gameworld's stakes right in front of our eyes while we look on unable to defend our assets that would again negatively affect my suspension of disbelief and lead to a lot of angry chat messages.
As to what should (if anything) be done about it, the thing is there are already fairly decent solutions for us to both get what we want. There are many large, well populated PvE groups out there, including Mobius PvE, wherein you're pretty much guaranteed to get the experience you're looking for. I actually think a good solution would be for Fdev to allow PG groups to publicise within the game so that players know they're out there. For me the best solution is to continue to play in open where the frisson of danger, the slow-burn competition and fighting can continue unabated.
In short, this game attracts players with different preferences but can only handle a few of them being together at a time. Putting like-minded players together and keeping diametrically opposed players separate should be the priority.