If you still chose to build your sand castle in the general public part of the beach, even when you had your own private, completely walled off section available to you - where you'd still be able to hear all the other beachgoers and even interact with them, though they could not have infringed on your castle building - then does it change things at all?
Not really, theres more than enough beach to go round, the people who want to destroy can go over there a bit and do what they like to each other, no reason to come over here specifically just to destroy.
especially as a lot of them complain theyre not allowed on the Private beaches anymore and they can see and hear people having a good time over there so they want to close them down as well