PvE doesn't require anyone else, that's the difference. You can do 'distant worlds' whenever you choose.I feel like we already had the very same discussion not too long ago.
It's the same for PvE players that want to play together or have events. It's not a bane exclusive to PvP players, yet they moan about it all the time. PvE players have to go through that all the time too.