Migrated to new hardware platform
Yesterday I migrated everything behind
https://ardent-industry.com to new hardware (and to a new provider in the process).
This happened without downtime and went from spinning up a new platform to fully migrating the service in an evening and was uneventful.
- The CPU, RAM and storage capacity is significantly increased on the new service. Performance gains are x10.
- Queries that took > 10 seconds are now sub-second. Queries that took seconds are now sub-millisecond.
- The new service is less expensive than the old service. It's still co-located in the UK but is no longer on AWS Europe/London.
Context
The performance of the previous service was generally good, but relied heavily on diligent use of caching and request optimization to manage performance. There were still occasionally slow performing requests that could occasionally take a few seconds and I was limited in terms of options for optimising to mitigate this because of storage constraints (as a rule storage is cheap, but high performance storage is more expensive).
Critically, the indexes for all databases now fit entirely in RAM, which is a huge boost to performance, are the additional CPU cores.
The new platform relies less on service architecture. The initial implementation used AWS services for managing caching, backups, load balancing, failover and denial of service attacks. The new platform is simpler in terms of system design and is cheaper but the trade off is less fault tolerance and more centralised complexity in the system software. I believe it's worth that trade off for the performance gains in typical usage – for example the new platform is fast enough to not need any caching in front of data and so commodity prices are no longer subject to a delay of up to 5 minutes, although this makes it open to be easier to abuse. It was still possible to get similar performance and retain the benefits of the original platform, but then the trade off is cost (and it would be than I'm willing to commit to for a free service).
What's next
Over the last week or so I've been making small improvements to the information displayed and fixing bugs.
Now that the constrains I was working on are lifted, I'm going to start work on adding in station data and exposing options to filter results (e.g. include/exclude carriers, minimum supply/demand volumes, minimum/maximum pricing, etc) and to perform searches and multi-hop routes.
This should start to make the web based interface actually useful and help inform what might be missing from the Admin APIs.
Once some of the trade specific features are in I'll turn towards exploration.
Request for feedback from developers
The unique aspects to this project - that set it apart from INARA and from EDDB - is that it's open source and that it has a public API, so that it can be integrated into other software, just as EDDN is integrated into so many tools when it comes to
sending data.
In that respect I'm hoping it plugs that gap for trade data, and potentially to expand upon what is already possible with the (excellent) EDSM API for exploration.
If you use the API in your software, please do give me a heads up as I am happy to accommodate avoiding breaking changes (and co-ordinating on potentially breaking changes, such as subtle changes in how requests are handled) but I don't know to do that if I don't know if they are in use.
