That's not true.
The game is just you in front of the screen and all kind of pixels on it. Whether or not there are other people controlling some of those pixels is utterly irrelevant. Ultimately there are people behind all pixels, including NPC's (people who programmed the AI in that case).
That's interesting approach, but it's hard to achieve that kind of zen attitude

For some reason I doubt even you approach games like that.
Imagine NPC calling your mother a ***** (any degratory noun). It's hard to treat that kind of scripted behaviour personally, but when player does it, it's different and even if you'll be able to distance yourself from that kind of childich behaviour and ignore it, the fact that some person does something aimed personally at you is not the same as just playing the game with predefined rules and behviours. For better or worse, another player in the game is not the same as AI. If that would be true, we would not need other people for anything - computers and their scripted behaviour would suffice.
When I choose to be in Open I don't mind other people initiating PvP against me in the game - but when those people break the fourth wall in some way, whether by saying some things, or acting in some way that does not fit into the game - that's when they actually break the game for me - something a game itself rarely does if it's not seriously bugged. And that's what griefers do - they try to spoil your game in any way they can. Game and it's pixels are fine until a person somewhere does something, knowing that it will affect you in some way. If you can ignore it, then kudos to you, but most people are not that aloof.
That's why I really appreciate Elite modes, so I can avoid those people entirely when needed.
Also, if we ignore ability to use the game in unintended ways to spoil your gameplay, players in game like Elite add unpedictability in sense that they are not scaled to your "level", as opposed to everything else in the game - they can be so far above your league, that encountering them might be almost gamebreaking. Even if you treat them as pixels and part of the game, it can get pretty irritating if the game starts throwing high level bosses at you, that you can't defeat.