pvp flag system would not work.
You can not segregate an open world sandbox game like that. you can not claim to share an online world when you create glass walls everywhere to divide it up into neat little personal spaces.
Don't go to Lugh and not expect to encounter CMDR's with contrary motives to your own if your goal is purely PvE. To me that is as counter-intuitive as farming Community Goal criteria from Solo mode. It's bad, counter-productive game play design.
It's world immersion breaking to allow players to be so selective as to have a PvP switch that says "nope! I want to be in an interactive sandbox, but I don't want to interact with you".
I'm not hating on the idea of PvE, I'm all for it. However, if you truly are a group of players who value cooperation and communion in gameplay experiences, then band together for your common interest. If a player insists on PvPing with your group... chase them off in the common interest of the greater majority. Isn't that what being a part of a group is all about?
Otherwise, you should really just stick to Solo and Private games as that is what they're there for.
I understand that some players love playing the spoiler... they undermine gameplay of others using means that are really outside of the virtual world itself. They stream-snipe players on TwitchTV, they lurk in forum threads looking for vulnerable targets. They may exploit loopholes and deficiencies in how the game itself works. I get that. Those situations are not what I'm talking about and not what I consider within the scope of sportsmanship that defines proper PvP. Those guys should be kicked in the virtual balls and no one should have to deal with that in any sort of sanctioned way to enjoy the game.
However, the distinction between spoilers and what is proper and responsible PvP should be shouted from the roof tops by every player who genuinely values a healthy environment for PvP.
What Frontier needs to do is, instead of creating further opportunities for players to easily segregate themselves, they should do the opposite and build up the social aspects of the game and reinforce the positive side of sharing an open world. They need to make it easier for players with common motives to find each other and reward positive behavior that comes from sharing an experience instead of encouraging the parasitic nature of combative/competitive player base encounters by default (which is what exists right now). Not just selectively turn it on and off at the purview of individual motives. Frontier needs to look at how to encourage players to help each other and see it as beneficial, so that they have a proper choice that isn't biased towards the only real value that encountering another player has is to prey upon them and compete directly against them.
It's simple. Many games bias a player to player encounter towards an adversarial outcome by putting a big dollar sign over the other players head. You get points/credits/power for being aggressive, nothing at all for being supportive... it's not rocket science to guess the outcome of that in an open world game. Anyone who looks at that formula and can't figure out why all the players do nothing but kill each other in the virtual environment really isn't suited to curating an open world MMO game.
You'd be surprised at how many players lean towards cooperation and supporting each over direct confrontation when provided with the most modest of incentives to do so. Yet... I've played so many online games that completely fail to do so effectively.
So, my basic rule is any solution that encourages more or stronger interaction between players is the right way to go. Anything that further isolates/segregates players to be the wrong path to follow. Reward and bolster the behaviors you want more of from your players (help over hinder each other), don't just disable or turn off choices for the player. A choice to be positive or negative towards another player is only really a choice when there is a balanced cause and effect from an outcome to consider. You will find a player choosing an outcome that risks a negative consequence if there is a chance for positive gain over a choice in which they have nothing to gain or lose from. Such is the case of the decision to attack another player, or simply ignore them. Where is the benefit of ignoring every player you meet? But then what option, other than ignoring a player, exists if not to attack them?
That is the crux at the heart of open play in Elite Dangerous.
I think that:
1) FD should consider not distinguishing between NPC's and CMDR's by default. An action like attacking or scanning a player should reveal that.
2) Anytime a player is interdicted by another player that instance should appear as a USS to other lawful players. Perhaps enhancing it with a Distress Beacon option so players can call for help from other lawful players (like Bounty Hunters) automatically.
3) Players should be able to declare a community group like a faction if they choose to be identified by one. Much like a faction alignment when scanned shows for players and NPC's.
4) Enhance the benefits of being lawful players, not just the consequences of being Wanted. Being wanted in Elite Dangerous has almost no stigma attached to it and thus very little influence on how the virtual environment treats the player. Again, it's about choice and the merits of choosing to avoid a wanted status over accepting a wanted status are pretty non existent.
Finally,
The real problem with the Open mode is the misconception of Open = PvP instead of Open = Social gameplay to begin with. Not every player to player encounter should be categorized as PvP. If that is the reality of your design for the Open World, then something is not balanced effectively enough to merit the title of "Open World Sandbox".