Previous incidents

Resolved. We released a fix at 21:33 UTC, and the app is reading and writing configuration normally again. This was a minor incident limited to the Quonfig app; config delivery and flag evaluation were not affected at any point. We are adding monitoring so a problem like this is caught before it reaches production.

On October 28 from 09:53 to 10:32 EDT (13:53–14:32 UTC), our secondary config-delivery standby (secondary.quonfig.com / stream.secondary.quonfig.com) returned 503 errors.

The host running the standby lost network connectivity, and our hosting provider's first two attempts to restart it failed. The third replacement instance started normally, warmed every workspace's configuration within seconds, and has been serving normally since.

The primary delivery service (primary.quonfig.com) was heal...

From about 18:09 to 18:12 UTC, some connections to the dashboard and telemetry ingestion failed or timed out at our hosting provider's (Fly.io) network edge in the US East and Frankfurt regions. Other regions were unaffected, and our services themselves stayed healthy. Everything recovered without intervention. SDKs retry on their own, and served configs were not affected.

Resolved. The dashboard recovered at 20:46 UTC after about 10 minutes of unavailability, and has been stable since. Config delivery to SDKs was not interrupted.

Between 07:24 and 07:26 UTC on September 30, our analytics database provider had a brief connectivity problem. During those ~2 minutes, some SDK usage telemetry sent to telemetry.quonfig.com could not be stored. The service recovered automatically, and ingestion has been normal since 07:26 UTC.

Impact: For customers on older versions of SDK, a small amount of usage analytics from that window may be missing: flag evaluation counts, context data, and example contexts. Flag and config delivery ...

On September 29 from 07:29 to 07:54 EDT (11:29-11:54 UTC), our secondary config-delivery standby (secondary.quonfig.com / stream.secondary.quonfig.com) returned 503 errors.

The standby's hosting provider stopped the instance it was running on, and it was restarted on a new host. The restart was normal except for a slow container image pull, which accounted for most of the 25-minute recovery. Once the new instance was up, it was fully warmed with every workspace's configuration and serving no...

On September 24 between 13:57 and 14:07 UTC, a small number of clients connecting to primary.quonfig.com from the Ashburn, Virginia area experienced TLS connection failures. The issue was isolated to a single edge server operated by our hosting provider. Clients routed to that server could not complete a TLS handshake. All other edge servers, including the two others in the same region, served traffic normally throughout.

Impact was limited. Our config delivery service itself was healthy the...

Between 8:07 PM and 9:12 PM Eastern on September 22 (00:07 to 01:12 UTC on September 23), our telemetry ingestion endpoint intermittently returned errors to SDKs. Config delivery, flag evaluation, and the dashboard were not affected. SDKs continued to serve flags normally throughout.

What happened: our analytics database, ClickHouse Cloud, experienced a metadata-layer slowdown that made writes take 10 to 50 seconds instead of under a second. Every write still completed, but our ingestion ser...