The whole thing is moving from one slot machine to the other. Before the engineers it wasn't a big deal. Drive along a planet, hit a node. It wasn't a big deal. Now, it's in our face. You pull this slot machine, to get a token to move on to another. Sure there are doors. Do "X" and it'll open the door to another set of slot machines.
This style of gameplay isn't new or ground breaking. It's been around forever. Most crafting system in video games are based on this. Regretfully if you want to use the engineers . You have to enjoy gambling. I've left games because of these kind of loot/crafting mechanics. The key for Frontier is to now either stand by the current "odds" and let the house (game) win. Or fix the odds on the side of the player.
The engineers in my opinion has always been a obstacle disguised as a upgrade system. Or the beta would have involved more testing of drop rates and loot mechanics than weapon abilities.
It's up to us as a player base to let them know when the odds aren't in our favor. I think there needs to be a huge improvement to how the materials are organized within UI. I'd personally like to be able to see ALL of the upgrades an engineer has without having to fly to the engineer's base. Also, we should be able to "queue" a material search like in Fallout or Skyrim. to help with locating those materials.
Then we leave the rest up to RNG and hope for the best. Happy Gambling CMDRS!