I am a software engineer and have worked on many projects where an app has to retrieve, parse, format and store data. It can be cumbersome but I have lots of experience doing that sort of thing. So a year ago I decided, just for fun and to learn a new web technology I was interested in, that I would build a "Ship Finder" service where a user could weight their favorite game loops (ex: combat 50%, hauling 30%, mining 15%, FPS 5%), specify various parameters (ex: at least 100 cargo space, purchasable in-game < 3 million aUEC, quantum range X and speed Y), etc. The app would analyze the user requirements and assign a score for each ship in the game for each stated requirement, then derive a total score from the subset, and spit back an ordered list of ships that, based upon the numbers, were the best matches for the user.
So I dove into the convoluted, tangled spaghetti mess of data and started parsing. A more wretched hive of scum and villainy, I have never seen. Most of the ship and component data is stored in XML files, and they were the biggest mess imaginable. Often improperly nested a dozen or more levels deep, tags opened but never closed, tons of pointless wrapper tags that held no data and served no purpose but to nest the data deeper. Awfully-named nodes that frequently had similar-but-slightly-different names for different ships even though each wrapped identical data, making it impossible to traverse the data structure one ship to the next. And on and on ad nauseam. Here's an example of what a tiny part of my parsing code looked like:
https://docs.google.com/document/d/1kpW7CYZNTv2ChwFc3aWro9N7E0_1AIoZu2ncKtB3PqQ/edit?usp=drive_link (go easy on me! I'm a Java dev first, PHP second). And that's after all the weapons, shields, coolers, etc, had already been parsed by other classes. I gave up after about 2 weeks when it became clear that nearly every vehicle (well over 100 of them) would require at least some degree of custom code. Writing it would have been a nightmare; maintaining it from release to release completely, utterly and hopelessly impossible.
And I was only trying to read the data, not reverse-engineer that algorithms that back DPS with various weapons, power settings, configurations, etc. That's what Erkul is doing and for the life of me, I cannot fathom how. That is a full-time job, folks. Erkul is a phenomenal masterpiece. To be clear, I'm not knocking the CIG devs either! They're probably all better developers than me and the way in which they're managing this whole house of cards is very impressive, too!
This illustrates the issue even better. Here's the xml that was in-place at the time for the Cutlass Black:
https://drive.google.com/file/d/1Qo-8lUe37CFwbvPARi6_0cpN1YRF5kdZ/view?usp=drive_link