I realize comparing ED to EVE annoys some people, but on this topic it seems relevant. For better or worse, EVE has taken a path that does little to discourage players killing or exploiting each other. There can be creativity and innovation in such things. Some call it emergent gameplay. Much of it can be seen a form of griefing and is very upsetting for victims. But, my impression is that it's quite challenging to balance these different types of players.
One aspect of this path is that EVE doesn't try to force players to play the game a certain way. Instead it recognizes the players will often be smarter than the developers in finding out ways to exploit the system and coming up with goals of their own. It seems quite likely that ED will also have players like this. And, the question will be: Do we try hard to stop them or do we find some way to embrace and accept them?
I could definitely see getting to a point where the pvp pirates in ED will get very good at killing other players. They likely won't care about lack of rewards or penalties. This is the case in EVE for the most part. It's very harsh being a pirate in EVE, few rewards. But, the kills themselves are worth everything to the players who do it. It's similar in Dark Souls, to be honest. It's not just EVE. Pvp is mostly about winning the fights and some players will be very good at it. How much do we try to thwart them? Surely, they could trigger lots of ragequit posts from non-pvp players. But, taking the victims side too often can lead to problems too.
Solo play mode is an interesting solution to some of this. Personally, I think ED should keep it and maybe build on it (carefully).
I'd like to see more cooperative play in open mode. I think this is quite hard, but if ED could somehow pull it off, it would be nice.
EVE also has a core value related to risk/reward. And, I could see applying this as some have mentioned for open play mode. There's more risk since you can be ganked, thus there should be more rewards. Then, a player can have a clear choice of safe solo vs. risky open with more rewards.
Personally, I usually play in open play mode, but I've almost never encountered other players. A few times I've switched to solo to dock. Thus, the issue feels somewhat moot to me. I've been choosing to play away from the core systems for various reasons (better trade profits, quicker access to exploration, etc.)