Back to overview
Resolved

Telemetry ingestion errors during a ClickHouse Cloud slowdown

Sep 23, 2026 at 12:07am UTC
Affected services
Telemetry ingestion

Resolved
Sep 24, 2026 at 7:44pm UTC

Between 8:07 PM and 9:12 PM Eastern on September 22 (00:07 to 01:12 UTC on September 23), our telemetry ingestion endpoint intermittently returned errors to SDKs. Config delivery, flag evaluation, and the dashboard were not affected. SDKs continued to serve flags normally throughout.

What happened: our analytics database, ClickHouse Cloud, experienced a metadata-layer slowdown that made writes take 10 to 50 seconds instead of under a second. Every write still completed, but our ingestion service timed out at 30 seconds and reported an error to the SDK for data that had in fact been stored. SDKs treat that error as a failed send and discard the batch. The slow requests also caused our hosting platform to repeatedly start and stop extra ingestion machines, which produced the connection errors our monitoring recorded.

Impact: roughly 5 to 13 percent of the flag evaluation counts and context data reported by SDKs during that hour were not recorded. You may see a small dip in evaluation charts for that window. No stored data was lost or corrupted, and no action is needed on your side.

What we changed: on September 24 we deployed a fix to the ingestion service. It now waits up to 60 seconds for a slow write, retries a failed write up to three times, and tags every write so that a retry of a write that already landed is deduplicated rather than double counted. We also raised the concurrency limits so that slow writes no longer trigger machine churn, and added a dashboard that shows this failure mode directly. We have verified the change in production and believe this class of issue is resolved. We are separately looking at having SDKs hold on to a batch until it is acknowledged, which would close the remaining gap for longer outages.