First, if I'm friends with someone in the same instance shared with someone I have blocked, which wins, the priority to instance with a friend or avoid a blocked player?
Unless they've actually done what they said they have and increased the weight of the block, then one friend will trump one block.
Second, what happens if I'm in an instance with a few non-blocked gankers (or anyone), and then a blocked player tries to enter the system? Do they get sent to empty-instance jail, or do I get kicked out of my current instance?
Unless they are friends or winged with someone in your instance, they get sent to a different instance.
Blocking cannot evict either party from an instance, it can only keep them from joining (or rejoining) it.
'open' is generally assumed to be that participatory space where stuff happens but that's really just an illusion, it's randomly sharded all over anyway, you only slightly affect distribution across shards which is negligible even if it was anyhow relevant, which it is not.
Random instancing, or weights corresponding to things outside player control, are not the same as giving players the ability to influence instancing in Open.
It's also not a negligible thing...if it were, then blocking wouldn't work.