Gender probably comes up far more than we traditional binary types think it does, we're just unaware of it because traditional society caters to us. When the world doesn't cater to you, you're probably far more aware of it.
It's like, I'm left-handed, the inconveniences I've faced over my life and in my job (like counting cheques or money) sticks in my mind wouldn't even register with a right-handed person (for whom the world is built).
If non-binary people tell me there's an inconvenience and there are issues, I'm inclined to believe them, just because my experience doesn't render me personally aware of a situation doesn't mean it doesn't exist.