I don't think you are necessarily doing anything wrong to cause this behaviour, but unfortunately I don't think there is a clear way to make it behave as you expect either. I often find that despite changing the vertical snap slider to zero, the game acts as if it is 1m or more. Sometimes unselecting all the things, reselecting them, moving the vertical slider off 0m and then back to 0m helps for me.
This happens to me very randomly and frequently, it's frustrating and I think it's a minor bug, kind of like how if I want to change the snap angle I always have to uncheck and recheck the snap checkbox before I can change it.