Discussion Ardent Insight — API & Website

Flash Moonboots

Flash Moonboots

Ardent Insight API & Website​

Last update 16 April 2025

Ardent Insight (ardent-insight.com) is the new name for the API and website previously called Ardent Industry (ardent-industry.com).

This change reflects the evolved focus and functionality of the site, as the scope expands beyond trade data to include other features.

The old URLs will continue to work for both web and API access, but to avoid confusion website visitors will be directed to the new URL automatically. Those using the old URLs for API calls can update the hostname at their leisure. The old domains will still be valid for at least the next year or two.

As we can't edit the titles of posts I've had to create this new thread. To avoid confusion, after today I will stop updating the original thread, and only be posting forum updates in this thread.

www.ardent-insight.com​

Source code for website:
https://github.com/iaincollins/ardent-www

Ardent API​

Documentation and source code for the REST API:
https://github.com/iaincollins/ardent-api

Ardent Collector​

Source code for the service that collects data submitted to EDDN and generates reports:
https://github.com/iaincollins/ardent-collector

Ardent Authentication​

Source code for the CMDR Authentication service:
https://github.com/iaincollins/ardent-auth

About this software​

The API was originally created in part to support further development of new features in the app ICARUS Terminal as there were no APIs and / or open source or open data platforms available for some of the data - although data dumps and APIs from EDSM.net, Spansh, EDDB.io (RIP) and the work of EDCD developers, and the EDDN service, are invaluable.

The scope and functionality of the API and the website has grown since inception, and continues to grow, and further improvements are planned. The hope is that some features from ICARUS Terminal will be able to be implemented on the website in some form and enhanced by combining CMDR data from the FDev APIs with data from EDDN, EDSM and other third party APIs.

The goal is embrace new and existing features including exploration, power play, colonisation and to surfacing interesting locations in the game, and to expand and refine the UI for the site to provide an immersive and functional experience. By providing an open API, other developers can use it to augment the data in their own third party tools to improve the player experience.

The UI of the website and some of the features will be familiar to anyone who seen ICARUS Terminal.

home.jpg
system.jpg
trade.jpg
 
Last edited:

Flash Moonboots

Flash Moonboots
EDSC Info said:
I received an authentication error, despite being approved by Frontier. Any ideas?
Click to expand...
Hmm no I don't know what would cause that.

I've tried signing in and signing out and again and it's okay here. I can see there are a handful of other folks signed in, so I think it's working, but would appreciate any confirmation it's working from other CMDRs.

I'd be interested to see any errors you are getting, to better understand where the problem might be.
 

Flash Moonboots

Flash Moonboots

Update: Improvements to API and Website​

I've made a lot improvements to the API and to the website over the last week - taking full advantage of the long weekend (even if it is only 3 days in Scotland, and not 4 like in England and Wales!).

The API improvements below have not been fully documented yet, I'll be updating the official documentation soon but thought I might as well do a quick write up here.

PS: The roll out of the new features might have cause the API to misbehave and fail to return data for some queries for 2-3 hours yesterday due to an incomplete change being rolled out - sorry about any service interruption!

API improvements​

1. API now supports querying by System Address (64bit integer ID assigned to each system) anywhere the System Name can be used.

In the case of endpoints that take a systemName query parameter, you can now use systemAddress as well (if both are specified, systemAddress is used as it is never ambiguous).

Examples​

2. Querying for a system (by name or by address) will now return a property named disambiguation with an array of other similarly named systems, if there is more than one system with the same name. The API will attempt to match by exact case if the systems are spelt the same, but the case is different. If there are no exact matches, it will still return data for one of them.

Examples​

JavaScript: 
{
  "systemAddress": 85261750986,
  "systemName": "C Velorum",
  "systemX": 844.5625,
  "systemY": -83.1875,
  "systemZ": -35.15625,
  "systemSector": "41087a2dd2d7d0b4",
  "updatedAt": "2022-11-26T00:18:27.000Z",
  "disambiguation": [
    {
      "systemAddress": 84389401298,
      "systemName": "c Velorum",
      "systemX": 299.40625,
      "systemY": -0.3125,
      "systemZ": -7.625,
      "systemSector": "b54462392f48b156",
      "updatedAt": "2022-08-17T09:45:34.000Z"
    }
  ]
}

3. Searching for a system by name will return a Boolean flag ambiguous in the result if there is another system with the same name. You can query the system via the API to get a back a list of the matches for that system.

Examples​

JavaScript: 
[
  {
    "systemAddress": 84389401298,
    "systemName": "c Velorum",
    "systemX": 299.40625,
    "systemY": -0.3125,
    "systemZ": -7.625,
    "systemSector": "b54462392f48b156",
    "updatedAt": "2022-08-17T09:45:34.000Z",
    "ambiguous": true
  },
  {
    "systemAddress": 85261750986,
    "systemName": "C Velorum",
    "systemX": 844.5625,
    "systemY": -83.1875,
    "systemZ": -35.15625,
    "systemSector": "41087a2dd2d7d0b4",
    "updatedAt": "2022-11-26T00:18:27.000Z",
    "ambiguous": true
  }
]

4. You can now query markets for commodities using the Market ID assigned to the station / port / outpost / settlement / megaship (etc), without having to specify the system name and/or station name.
5. API responses for systems, stations and commodities now include the systemAddress property to facilitate querying the API using the new endpoints, to facilitate disambiguation.

6. There are some experimental API endpoints that use the EDSM.NET API to return back information about systems. These are largely just pass through APIs, at least for the moment. Like all the other endpoints, you can now use

Website improvements​

The website has been improved to take advantage of the new features, in the search, system map and system list views, and to the nearest services views, to expand the information displayed and to make it more useful and to better handle edge cases.

There have also been a lot of smaller functional, cosmetic and performance related improvements.
The system view doesn't yet surface all the same data as ICARUS Terminal, but underlying system for mapping out bodies and stations and megaships in a system is vastly superior, and already handles a lot of edge cases where the data is incomplete (or just unusual).

Further improvements are definitely planned! I might turn efforts towards improving the Commodities trade view next - I would like to replace the input elements with better custom inputs and make the UI more flexible.
 

Attachments

  • 1.jpg
    1.jpg
    347.5 KB · Views: 15
  • 2.jpg
    2.jpg
    397.8 KB · Views: 16
  • 3.jpg
    3.jpg
    348.6 KB · Views: 16
  • 4.jpg
    4.jpg
    345.9 KB · Views: 15
Last edited:

Flash Moonboots

Flash Moonboots

New improvements​

  • Cleaned up data from System Colonisation Ships in the Trade DB - although valid markets you can sell to, they are transitory and typically not something most people are actively looking for and the data for them is also messy. As FDev have said they will be making a change to how they appear in the next patch, they may come back, and be treated as potentially transitory in nature (like FleetCarriers and MegaShips).
  • Normalized all Primary and Secondary Economy types for all stations, and created a new endpoint that returns information about how many of each type are known to exist https://api.ardent-industry.com/v1/stats/stations/economies
  • Cleaned up and normalized data for Fleet Carriers - they should all be type FleetCarrier and have null values for things like Economy, Government, Allegiance, etc (previously this was a bit messy)
  • Lots of cosmetic - and some functional - improvements to the System view, with more to come!

For anyone interested, I've been using the Project View in GitHub to track some of the things I'm working on across the various repos:
github.com

Software for Elite Dangerous • iaincollins

This project is intended to help align and plan development of the tool ICARUS Terminal and the website Ardent Industry.
github.com github.com

Feel free to use the issues there to leave comments or to raise bugs and suggest fixes!

Someone has already used it to flag an issue I hadn't spotted which was much appreciated.
 

Flash Moonboots

Flash Moonboots
There have been some small quality of life improvements recently, but with Frontier authentication service offline I was prompted to add some handling for when the game is offline, so instead of just appearing as not signed in, when the Frontier API returns that it is in maintenance mode (e.g. during an update) CMDRs will see this message in place of sign in now:

1746015721584.png


In future it should be possible to gracefully fall back to using whatever existing CMDR data we have if you are already signed in, but as it is for now, it only works when the game itself is online, as the Frontier API for CMDR data also goes offline during maintenance.
 

EDSC Info

EDSC Info
Flash Moonboots said:
Hmm no I don't know what would cause that.

I've tried signing in and signing out and again and it's okay here. I can see there are a handful of other folks signed in, so I think it's working, but would appreciate any confirmation it's working from other CMDRs.

I'd be interested to see any errors you are getting, to better understand where the problem might be.
Click to expand...
A mistake on my part, I was using an account that wasn;t linked to ED
 

Flash Moonboots

Flash Moonboots
EDSC Info said:
A mistake on my part, I was using an account that wasn;t linked to ED
Click to expand...

Ah cool, thanks for letting me know - I didn't actually test for that scenario and so not sure how it handles it!

I might try creating a second account to test it and put something in that knows how to handle that situation.
 

Flash Moonboots

Flash Moonboots

Service Impacted - 2025-05-11​

Quick update as a PSA that there was a service outage this afternoon. Mitigation has been put in place but it could return.

Cause​

Google started maxing out all 10 CPU cores to 100% by hitting the server from a couple of dozen IP addresses at the same time doing esoteric queries, which chewed up the sever. Google regularly crawls the site and that drives traffic but it doesn't usually actually end up doing a Denial of Service Attack by trying to do so many long running queries at once.

EDIT: This ended up being thousands of requests from over 100 different Google managed IP addresses from a various different subnets in a short space of time. Some of the IP addresses Google manage don't have reverse DNS entries but turned out to be net blocks assigned to Google and used for various Google crawlers.

Mitigation​

It's still generating a lot of traffic but I've done some mitigation work as triage to make the site functional again by allowing to lean more on disk I/O than RAM. Only a couple of cores are now being stressed at any one time and the site is reasonably responsive so hopefully this update will hold.

The site may sometimes run a little slower than usual for a while as consequence, until / unless I do some further work to refine how this type of traffic can be better handled without impacting legitimate traffic.

Context​

The entire trade and stations databases fit in memory these days - after both refining the data stored in the database, and increasing the amount of memory on the server - but it's been maxing out memory usage too, which likely caused by unusual queries from Google that are pulling in system data which doesn't also all fit in memory. There isn't a quick and easy fix for that scenario, at least not a cheap one - rate limiting doesn't really help in this case and neither would limiting concurrent requests by IP as they are coming from a range of IP addresses.

Restoring /robots.txt on the API to discourage crawling (which I've now done) and blocking IP ranges is an option I may explore, but is a bit of a crude / blunt instrument.

More RAM on the server would certainly help and is easy but is relatively expensive. Although I'm aware some folks running private sites - i.e. not open source with documented APIs - apparently take in thousands each year from donations (even as much every month as it costs to run Ardent for a year...) but I would rather try and avoid that as it feels a bit like stepping over the line with monetization of Frontier's Intellectual Property. I'm mindful that I also want the service to be cheap enough to be sustainable indefinitely.

I might see what I can do to make the traffic and service status more transparent, like service monitoring page, like EDDN has, as that might surface more issues than just monitoring the services on a fairly ad hoc basis. I'll also have a think about other measures I could do to try and identify characteristics of problematic queries and isolate them; and perhaps even treat them differently from other more typical queries - including looking at ways in which the database could be optimized to better facilitate the sorts of queries that tend to be more intensive.
 
Last edited:

Flash Moonboots

Flash Moonboots

New API + Website release incoming!​

On Thursday, during the scheduled maintenance window that starts at 7 AM UTC - aligned with weekly game maintenance window - I plan to roll out out a major update that I've been testing in pre-production.

This will change how data is stored for commodities, and consequently the database dumps, and will impact the format of API responses. This will undo some denormalization that was previously done to improve performance, and replace it with a slimmer database that uses improved queries to try make the best use of the system resources available.

This is in part to continue to address the impact of the incredible growth of the platform in the wake of the Trailblazers update, and as consequence of the sustained increase in player activity over the last few months, following the recent updates to the game by Frontier.

Ardent will be ticking over 150,000,000 logged systems soon - a fraction of the actual number recorded in-game but still feels like a neat milestone.

What to look forward to:
  • The trade database should be significantly smaller in terms of disk space used for the same amount of data, I expect it to shrink at least 50% in size. It will require less RAM to hold entirely in memory and the new queries should hopefully should provide an additional uplift to performance when under strain, as well as make for a smaller download.
  • Redundant data has been stripped from some API responses which will make them quicker and faster. Examples include internal ID's that were exposed (these will no longer exist on the backend) and returning details for a Station on a Commodity listing when already querying for them by Station (when you probably know what station you are querying for already) and poor quality status flag data that was never really useful.
  • In some cases, additional information that is relevant has been added to other API responses, such as the size of landing pads at the station and what the station economies are and the distance from the station to the main star. This makes it easier to build more powerful tools without having to make a lot of additional queries.
  • There are improvements under the hood to how station and system data is treated to avoid edge cases now that players are able to colonies systems that may share names with other populated systems (either exact matches, or case insensitive matches) to try to provide more robust handling in the future.
  • A new endpoint for searching for stations / settlements / fleet carriers / megaships by name has been added.
  • There will be new /v2 versions of some endpoints. The /v1 versions will redirect to the new /v2 endpoints and the parameters remain the same (but may be expanded on) but the responses will different slightly. As the responses are substantively the same anyone using the v1 endpoints may not even notice anything has changed.
  • None of the URLs on the website have changed! This is all happening under the hood, existing bookmarks will still work!
  • Some of these new features will be leveraged in the update to the website, others will follow soon after. I've recently rolled out some small improvements to the UI and additional improvements will be included in this update, in particular to improve the features on the trade explorer page to make it more useful - including improved UI controls that allow direct text input and more filtering/sorting options.
  • I know there are some performance issues with the website especially on slower systems, and have been working to moderate the impact of animations and overlays.
What you might want to watch out for:
  • This will add some complexity for anyone using the raw database dumps. If you want to download the trade database to use it in your applications, you will also likely want to download the stations database and use them together - although the download size for both databases will now be smaller than the size of the old trade database on its own.
  • The API and website will likely be inaccessible for up to 30 minutes while the database is rebuilt. It's technically possible to avoid an outage in an instance like this, and it's technically possible to main the v1 endpoint behavior, but frankly I don't think it's worth the effort and I don't think anyone really minds. It's a bunch of extra work to maintain the old stuff and do a graceful migration, compared to very little work (if any) to actually migrate to using the new API routes.
  • Performance should be better, pre-production testing has gone well, but there is always the possibility of unforeseen issues. This isn't a commercial / funded project and I don't have a replica test or staging instance running on exactly the same hardware, so we'll have to see how it goes! Worse case I might have to undo some of the changes. I will probably try to fix-forward rather than reverting changes completely though.
I'll put in place a holding page for the update while it's in progress and will post on this thread when the update is complete.

I should have more information in a CHANGELOG for folks using the API about specifics of the changes (what routes, example responses, etc.)

I would love to get the API into an Open API spec at some point soon - if you feel like you could help with that please reach out!
 
You must log in or register to reply here.
Back
Top Bottom