Density APIs Offline

Incident Report for Density

Resolved

We experienced an outage on the public API service between 23:14 UTC on August 15 and 00:30 UTC on August 16 due to an out-of-memory condition, which resulted in a degraded state of the service. The impact was limited to API availability, no customer data was lost. You may need to retry to access the results from this time period. This was a shared service issue, meaning any customer accessing the API during the incident could have been impacted.

As a next step, we will work on:
Improving scaling strategy to better handle spikes.
Adding profiling to better understand memory usage across the different APIs.
Improving Traefik routing so that during degraded states it returns 503 Service Unavailable instead of 404 Resource Not Found.
Posted Aug 15, 2025 - 21:00 EDT