There are just two rules
- you shoot something you don't know a status of = you get wanted
- anything starts shooting you (if you are clear), you can defend yourself even without scanning them
There's a flaw with that though.
OP obviously came across a group of villains who were then sizing him up to be the next victim, leader gets into a fight and a lackey got accidentally shot.
The issue isn't so much that the OP hadn't scanned that ship first - more that ... would the villain have had "report crimes" turned on or not?
If you want the bounty then you should have to scan the ship(s) but if they're blatantly bad-guys about to attack/rob you - it's self defense! And they're hardly likely to report and call the system police to come help/arrest them if you resist are they?!
IIRC you can get interdicted, told to hand over x amount of cargo and if instead you choose to fire on the ship without scanning first -you- are now the bad guy. Errr. nope.
Let's not keep defending a nonsensical mechanic and hope that FD improve upon its binary nature so that it makes a little more sense.