As I have yet neither killed nor probed an Orthrus: what would be the best approach should I ever encounter one: kill it, or probe it as much as possible?
Thus far I have given 6 Orthrus Interceptors a volley of 10 Research limpets each; over those 60 limpets I got only 3 samples total, which is abysmal. This run was lucky enough to yield two at once, but this more typical run gave none at all. Between your two alternatives I would say to kill it, however that requires a different vessel; if I am in my research vessel and attempting to summon a proper Interceptor, I now ignore threat 4 Non-human signal sources. Sometimes I drop at higher-threat signals if they are really close, in the hope of getting an Interceptor rather than a few Scouts.
How many samples could you get out of it assuming good conditions (perfect flying, adequate controller etc.)
You can send one volley, including all eight from a Universal multi-limpet, however I consider the expected yield to be well below one sample. There is almost no flight skill required though; you do need to stay out of its direct path, where it will suffice to target it during arrival and position yourself behind it, then follow loosely while avoiding the caustic trail. Mind that it will make a 90-degree turn after concluding its business; as long as you started behind it, you will not be seen that way.
would that number progress the effort more than the kill itself?
My estimate is almost certainly not, given the poor sample yield. If you must hunt Orthrus signals then it should be for a kill.
And regarding the kill, what is a rule of thumb estimate on the required firepower (e.g. X modshards or Y AX Missiles etc.)? Meaning having just enough to kill it before it flees.
No Guardian modules at all; no Shards, no Plasmas, no Gauss, no Shield or Module reinforcements, no FSD booster, and especially no Power plant! That said, you need very little; this example is @CMDR Vulkarius with an unengineered Imperial Cutter. Try a 8A shield with boosters for stabbing the Orthrus shield, hull reinforcements for the caustic damage, Shield cells to burn caustic afterwards, some EAX missiles and a Shutdown neutraliser. That video was made before Weapon stabilisers, as you can see from the bond value, hence only four missiles. The entire encounter fits well within 60 seconds, where you have around 120 seconds available.
you'll need an anti-shutdown field.
No—you can sample it without a Neutraliser. It deploys a shield regardless, but will only use a Shutdown if it sees a Commander directly in front, or if attacked.