When mining geo sites, there are four grades of material sources: cobble, crystal fragments, crystal clusters, and needle crystals. Those correspond to the four grades of
raw materials, so needle crystals (and only these) will always drop the rare material for that planet.
Bio sites work the same way, but have different names for the growths - phloem excretions are the space fungus equivalent of needle crystals.
That said, you can absolutely source the rare raw materials from other places. On planet surfaces, rare elements can drop from popping mesosiderites and metallic meteorites IIRC. However, the collection rate for rare elements should be much better by going to a bio or geo site. If you need a large amount, consider flying out to the
Crystalline Shard bio sites.