Nothing would prevent this except being thrown out of the mode and restricted from re-entry, something that could not happen if the modes were combined. As they say, "Once is an accident. Twice is a coincident. But three times is enemy action", three strikes and you're out, so to speak. Now, the "bully" only has to get anotther account and do it again but after three times they are out again. Eventually either the "bully" is going to get tired of throwing away money or FDev put in a permanent ban. Still, with separated modes the "bully" has to work hard and eventually spend money to achieve their aims whereas in the combined mode anyone can do this any time they like and there is no downside on the part of the "bully".
It seems to me that in a separated mode situation the chances of PVP action accidental or intended would be greatly reduced whereas in the combined mode it would not. And coupled with RM's 'moved to a separate instance timeout feature' would go a long way to easing the concerns.
Still, the thought does occur to me that your proposed solution effectively separates players out into two 'virtual' modes anyway. After all, a difference that makes no difference is no difference. Or in th modern vernacular, if it walks like a duck, looks like a duck and quacks like a duck, then it's probably a duck.