There are several big problems with VR.
You essentially needs to make 2 images (left and right eye) and that means that you need significantly more computational power than a single image on a monitor for a given image quality. And. It gets worse. If you're playing on low-spec hardware on a PC you may have to endure the occasional FPS drop or stutter. Most people tolerate this just fine on a monitor, but on a VR headset a drop to 30fps is going to make you want to blow chunks. You need to see a solid, consistent and high frame rate for VR to work. You can achieve that by either dropping visual quality (so it "looks like crud") or requiring a high spec machine (so it is either inaccessible, or "runs like crud" to most customers). Much the same problems exist with headsets - for example I need to have IPD adjustment as I have a large IPD. But only a few headsets offer it, and they're generally expensive. The cost of entry is just too high to mass adoption. But I think this is perhaps the least of VR's problems.
Secondly is the problem that people instinctively don't like strapping a big, plastic box to their head to play video games. You look ridiculous and some people feel vulnerable knowing they don't know what's going on around them. It's inherently anti-social as people nearby can't interact with you whilst you're playing it. And people don't want to share headsets as it feels unhygienic.
You also have the problem of limited space. If you're going to move around in the game you need to move in real world and you need to make sure there isn't anything in the "real" world occupying the same space as something in the "Virtual" world, or problems will result. Since this means you can't move around much a lot of game types are almost impossible in VR or you are reduced to workarounds like "click to teleport". It works well in vehicle games (like elite) where you're as sat down in VR as you are in real life, but increasingly less so if you have to walk or run anywhere using your legs (see...….)
The fact that VR only really works on 2 senses (sight and sound) and ignores every other sense such as smell, taste, pain, temperature, tactile, muscular sense and - most of all - balance means that for a significant number of people it "doesn't work" or makes them feel sick as their vision says they're moving but their balance centres say they're not. The very best VR in the world is not even close to a Holodeck. It relates to Star Trek Holodeck like Stephenson's Rocket relates to a Tesla. IE, it doesn't. It's still just strapping 2 mobile phones to your head, basically.
It is a cool technology. But the problem is that it's expensive to implement and hasn't really got the "take up" people were hoping for it and there are some pretty inherent problems with fooling only some senses on a person as we've evolved to understand that upset balance is because we're eaten something we shouldn't and need to get rid of it.