I'm 50/50 on this, I completely agree with the first part, partly less with the rest: to build a game engine from scratch you have to know damn well what you are doing. 100%.
Following from that, the rest becomes relative, to scope, size, etc. I'll take X4 again as an example, if anything because it's the one I have the most direct experience about. It's a recurring question to developers over there too, "why didn't you use something like UE or Unity or $insert_name$, so you could invest your limited resources in actually making the rest of the game".
And from the point of view of a very small developer with limited means, the answer has always been the same: outside of the licensing costs for any commercially distributed product, there is no point in using an off the shelf engine if the time and resources required to bend it to your needs exceed the efforts of making one yourself, tailored around your requirements. So first things first, you need to already know what kind of game you are going to make, how do you plan of developing it, and eventually how do you plan on expanding and supporting it over time. Once you have ticked those boxes, most of the rest comes as part of a programmer's profession. I don't intend to detract from Egosoft's staff expertise in the field, but they surely aren't some superstar coders or "magic germans" by any means, they are "just" programmers as there are many around, with their varying degrees of experience of course. And yet, that handful of guys working from a home office always came up with their proprietary engine, with their last one allowing space legs at a tiny fraction of costs/resource/time of Star Citizen's wrecknought (and I think the same can be applied to Hello Games with their No Man's Sky too).
Definitely rough around the edges, with no fancy "physics" simulation, different gravities, oxygen and life support, planetary surfaces? Sure. No multiplayer requirements? Sure. Far simpler visuals? Sure. More limited in the kind of first person shenanigans it allows? Sure.
Stable and working good for what the games requires, allowing the players to play the game as it was meant to? Sure.
I don't like to cultivate the notion of "incompetent/lazy programmers", be them from CiG, Frontier, Egosoft, wherever, from my outside perspective I don't see it like a field where you are allowed to improvise competence, you are either doing it because you are good at it, or you are doing something else entirely. Botched works come out of messy ideas, bad management and even worse planning, and no amout of state of the art programmers or magic germans will keep the ship afloat if the captain is an iceberg magnet.
where are they now?