OK, I think I figured out what's going on. The cops turn hostile if you attack pirates that belong to the same faction.
I don't know if this is intended. Anarchy systems have no law against attacks, but on the other hand, cops in Anarchy systems can belong to a pirate faction. So it might be reasonable for them to protect their pirates. It could be their second duty in addition to scanning for illegal cargo (I think anarchy systems still have laws against a few types of cargo, usually toxic waste.)
But I don't know is this is how it's supposed to be. Only a dev can verify this.