FIFA is a sports game. You aren't doing any sport though. Unless you are calling gaming a sport but in that case every game would be a sports game. The genre term space sim is being used for games like Elite since at least 25 years (that's when I first heard it) by the gaming media, players and developers.
People are of course right when they say that Elite isn't a realistic simulation, but that's not what the term suggests. You need to remember what most other games looked like when the term was first used. Take a look at Pong, Space Invaders or Pac-Man. Now compare that to Elite and you realise why people called it a space sim. That's where the term comes from and the reason it's still used today, Elite defined the genre way before real simulators even existed.
Some other notable space sims:
Wing Commander: A space sim.
https://en.m.wikipedia.org/wiki/Wing_Commander_(franchise)
Freelancer: A space sim.
https://en.m.wikipedia.org/wiki/Freelancer_(video_game)
Freespace: A space sim.
https://en.m.wikipedia.org/wiki/FreeSpace_2
X: A space sim.
https://en.m.wikipedia.org/wiki/X_(video_game_series)
As you can see none of these games are realistic simulators, but they are all called space sim. That's not because the name describes the game, but because that's how the genre is defined. That's how language works, it isn't always descriptive and the meaning can be something completely different than the words suggest. I could just as well argue that the Super Bowl isn't a sport event but something in your kitchen.