They can, especially very competent FA-Off users. But many people aren't able to fly like Isinona (check his youtube if you don't know him) and/or make almost no mistakes. So in a way I can use my tankiness to wait for mistakes that sooner or later will occur. Against one Vulture I have a lot of time.
But I don't only try to react of course. I use a lot of FA-Off, but only in suitable situations as I can't control my ship well enough for 100% FA-Off (I think this is indeed a little disadvantage of using Mouse and Keyboard [or harder to control], but I make that up with accuracy). Situations like i.e. you are behind him and he turns slightly faster, with FA-Off you can slightly change your angle through creating a little gap (because you drift into another direction rather than following him directly). A sudden switch to flying backwards after an FA-Off 180° can create a surprise moment giving you a full broadside or if he is very close the chance to boost in his direction to just destroy his current superior position.
Flying backwards is useful in general because you can easier maintain a certain distance which helps to get him in sights. Think of it this way: if between you and your target is only a 100m gap and he moves 20m upwards and you stay on your position, you need a big change of you own angle to keep your crosshair on him. If this gap is 1000m, it's relative only a slight change which means you only need to adjust your own angle a little bit. This somewhat makes up for your worse agility. (I hope I could make it clear, it's not my native language) Only going forwards usually means that you constantly lessen the gap and increasing the own needed change of the ships angle. But it's situational in which direction you should fly. If he adapts to your backwards flying, it's time to change again.
I don't know if I'm lucky or unlucky, but I haven't met a pilot I never could even shoot on. I'm sure they exist as I'm not the best, but without wanting to be arrogant: I'm a competent pilot who's no easy prey.