The effect is more noticable when it's an unanswered spec ops spawn. It's not a personal challenge factor, it's an instance factor... spec ops mow down non-spec ops ships as they often operate in a wing.Although I take my time with killing them, they don't seem to turn the tide. At least not if you're quick killing the regular foes. Then your allies can also focus on the spec ops, allowing you to kill them more swiftly.
As a general rule, I'd say "kill the easy targets first, that's the quickest firepower decrease you can get those foes".