Most people who don't meet and interact with people who look different, will be suspicious about them. Some will feel very uncomfortable, unsafe, etc. The more they get used to living with foreign born neighbors and such, the more those fears and prejudices go away.
Yes, this is true in general - people are naturally more comfortable around an environment they are familiar with; especially notable the older one gets.
But a good cause can be supported also by ignorant people for the wrong reason. One can oppose a culture, ideology or religion out of ignorance-driven fear, but there could also be reasonable reasons for it. That is, someone's prejudice can be someone else's judgement.
Me and my wife are immigrants ourselves; I'm Italian and she's Chinese, I'm a Catholic Christian and she's agnostic, our kids are half blood. I hold conservative positions not because I am closed to the world, but because I have choose so after considering the alternatives.