Like a lot of things, it is often a case of risk versus benefit. The reality is that what appears safer for flying, is not always the safest practice in actuality. It's kind of like flying an aeroplane really low - obviously that is not as safe as flying well clear of terrain, but it actually would be worth the risk if people were likely to launch SAMs at you, since you'd be masked by the terrain for much of the time. But back to headgear...
Technically, modern day jet fighter pilots would be 'safer' if they all wore pressure suits with very bulky sealed helmets similar to the kind of thing pilots of specialised very high altitude reconnaissance 'spaceplanes' such as the SR-71, MiG-31 and U2 have to wear, rather than sitting in a pressurised cockpit with a much more lightweight 'bonedome' open-face helmet with a drop down visor and clip-on oxygen mask, as is the case with fighter aircraft.
A pressure suit and full helmet would only be 'safer' in the sense that a pilot would be better protected if there was a loss of cockpit pressure. It would not actually be safer in practical operational terms for a fighter pilot, since in a full pressure suit, they would no longer be able to wear thin gloves, which makes operating the controls easier, and similarly, a cumbersome fully-sealed helmet would constrict peripheral vision massively. That would be disastrous for a fighter pilot, because it is the peripheral vision which is more sensitive to movement, so that's the bit of the vision field which allows a fighter pilot to spot and track targets early and to maintain situational awareness in a dogfight. Even with modern jet fighter combat, you can have all the radar systems in the world, but eventually a dogfight comes down to maneuvering skill and using the Mark 1 Eyeball to maintain your SA.
Thus the modern fighter pilot helmet is more about carrying a visor to prevent dazzle and to hold your earphones, mike and oxygen mask whilst also offering a small amount of protection as a bit of a fringe benefit. But if you asked most fighter pilots, I bet you they'd tell you that they'd prefer it if they did not have to wear one, since they are heavy (not good when pulling G) and they restrict vision and movement somewhat. The oxygen masks can chafe too when you wear them for a long time and they invariably stink a bit as well.
Some years ago, those of us who fly civil aircraft in the UK in close proximity to other aircraft - i.e. airshow display pilots, glider pilots, glider tow aircraft pilots, helicopter pilots and that kind of thing - were warned to stop wearing something as seemingly harmless as baseball caps when piloting, or to turn such caps around so the peak faced backwards when piloting (and this was at a time when the baseball cap was 'the' pilot hat, everyone wore them). The warning followed a CAA air accident investigation of a crash which occurred at the airfield I flew from at the time, in which three pilots I knew fairly well (two in one aircraft, one in the other), were killed in a mid-air collision. It had been determined that their wearing of baseball caps had restricted their upward vision sufficiently to contribute to the accident - i.e. one aircraft descended into the other and the pilots in the aircraft below simply had not seen it coming down onto them, this despite having a fully unrestricted 'blown' canopy, similar to the one in the photo of me below.
That photograph was taken at the time of that crash investigation, at the airfield where that fatal accident occurred. You can see that the baseball cap I am wearing is on my head backwards. This is not me attempting to look all 'gangsta', it is as a direct result of that crash investigation, determining that the peak overhang of such headgear severely restricts upward vision and situational awareness and is dangerous. And believe me, when flying from places like that on a busy day, there would be so many aircraft circling around within spitting distance of one another, that it was reminiscent of being in a WW1 dogfight. So much so in fact, that I used to practice leading other aircraft in 60 degree banked turns and imagine I had a gun on my aircraft for a bit of additional entertainment! Yes, all pilots are big kids!
In other words, anything which restricts movement or visibility in a cockpit is a big no-no if you can get away without wearing it.
