# Runaway connection churn filled the database disk and took the platform down > A provider-quality filter switched itself on at its default date and emptied two selection pools, and our hosted proxy service began churning through short-lived connections faster than the database could serve them. For two days connections failed or stalled for many people while database storage grew with the churn, until on Oct 23 a logging fault filled the disk and the platform was down for 16 hours, then 6 more for the storage expansion. That evening the same churn jammed payment reservations, and for about five hours apps could find providers but not send traffic. Started Oct 21 00:00 UTC; ended Oct 25 03:44 UTC, 99 h 44 min. Worst level: Full outage. https://ur.io/outages/2026-10-21 On Oct 21 a provider-quality filter switched itself on at its default date and left almost no providers in two of our selection pools. Our hosted proxy service, trying to replace providers that failed, began creating short-lived connections far faster than the database could serve them, and for two days connections failed or stalled for many people. Every one of those connections also added to the database's storage. On Oct 23 a logging fault on the nearly full disk filled it completely, and the platform was down for 16 hours, then 6 more while we expanded the storage. That evening the same churn tied up payment reservations, and for about five hours apps could find providers but not send traffic. ## Phases | From (UTC) | To (UTC) | Level | What people saw | |---|---|---|---| | Oct 21 00:00 | Oct 21 14:00 | Degraded | Two selection pools were nearly empty; most people were not affected yet. | | Oct 21 14:00 | Oct 21 15:00 | Partial outage | Database queueing began and connections started to fail. | | Oct 21 15:00 | Oct 22 02:08 | Major outage | Apps found providers but connections failed or stalled for many people. | | Oct 22 02:08 | Oct 22 13:45 | Partial outage | Pacing and the postponed filter eased it, but the churn kept going on its own. | | Oct 22 13:45 | Oct 22 14:41 | Degraded | Churn brakes broke the loop. | | Oct 22 14:41 | Oct 23 11:12 | Partial outage | The loop came back under daytime load, and connections stayed unreliable until the disk filled. | | Oct 23 11:12 | Oct 24 03:20 | Full outage | The database server's disk filled and the database stopped. The whole platform was down. | | Oct 24 03:20 | Oct 24 05:29 | Partial outage | Serving again on most servers while the storage was expanded. | | Oct 24 05:29 | Oct 24 11:29 | Full outage | A second, planned full outage so the storage expansion could finish. | | Oct 24 11:29 | Oct 24 16:48 | Partial outage | Back with reduced capacity: six load-balancer groups served nothing after an operating-system upgrade renamed their network interfaces. | | Oct 24 16:48 | Oct 24 19:30 | Degraded | Full capacity was back while backlogs drained. | | Oct 24 19:30 | Oct 24 20:54 | Full outage | Apps could find providers but could not send traffic. | | Oct 24 20:54 | Oct 24 21:30 | Partial outage | Extra data grants restored traffic for about 35 minutes. | | Oct 24 21:30 | Oct 25 00:26 | Major outage | The grants filled up and failures returned until the fix was ready. | | Oct 25 00:26 | Oct 25 01:27 | Partial outage | The fix rolled out in stages. | | Oct 25 01:27 | Oct 25 02:30 | Healthy | Healthy. | | Oct 25 02:30 | Oct 25 03:44 | Partial outage | Slow again: a long-running backup kept the database from cleaning up. | ## Key moments - Oct 21 00:00: A provider-quality filter turns itself on at its default date. It was meant to wait for a decision that never got made. - Oct 21 14:00: Database queueing starts as our hosted proxy service creates and discards connections hundreds of times a second. - Oct 22 02:08: Discovery pacing ships to every proxy, and refused connection attempts fall by about 90%. - Oct 22 04:03: The filter is postponed. The churn keeps going on its own. - Oct 22 13:45: Churn brakes break the loop, until daytime load brings it back at 14:41. - Oct 23 11:12: The database server's disk fills and the database stops. - Oct 24 05:29: Every service stops so the storage expansion can finish. The platform is back at 11:29. - Oct 24 19:30: Apps can find providers but not send traffic. - Oct 25 00:26: The fix for the payment-reservation stall starts rolling out. Service is healthy at 01:27. - Oct 25 03:42: A long-running backup is stopped, and the database cleans up within minutes. ## Who was affected - **People using the apps.** From 14:00 on Oct 21 to the evening of Oct 22, connections failed or stalled for many people even though the app showed providers. The platform was down from 11:12 on Oct 23 to 03:20 on Oct 24, and again from 05:29 to 11:29. From 19:30 on Oct 24 to 01:27 on Oct 25, apps could find providers but could not send traffic. - **Hosted proxy customers.** Hosted proxies were hit first and hardest. From Oct 21 their windows churned through short-lived identities, and most of those identities never connected. - **Providers.** Providers could not connect while the platform was down. - **Data.** We know of no lost data. ## What went wrong ### Chapter 1: A scheduled filter emptied provider pools and set off a connection storm - **A scheduled change went ahead by default.** The filter's start date was a default in code, so it switched on without anyone deciding. Our own measurements had shown it would remove more than 90% of one pool and leave the other with a single provider, and the deadline to decide passed without a decision. - **Churn that kept itself going.** Our hosted proxy service keeps windows of candidate providers, each tried with a newly created identity. With the pools empty most candidates failed, so windows created more. Every identity costs database work; the database slowed, connections took a median of 14 seconds, and windows discarded identities before they could connect, creating still more. The loop kept going after the filter was postponed. - **Storage that only grows.** Every short-lived identity left permanent records: a signed key registration stored twice, evidence objects, and contracts that are kept for months. The database grew by up to 20 GB an hour, and the disk-runway warning we planned after the Oct 12 near-miss had not been built. ### Chapter 2: The database disk filled, then payment reservations jammed - **A logging fault on a nearly full disk.** Once free space fell below the log service's reserve, the service started a new log file about three times a second and never removed old ones, writing about 200 GB in a few hours. The disk filled and the database stopped. - **A storage expansion that couldn't run under load.** The disk controller's online expansion made no progress while the database was serving, so finishing it took a second full outage. - **Payment reservations released slower than they were made.** Our hosted proxy service's network made hundreds of tiny contracts a second, each holding a reservation for up to 24 hours. A background job released them, but it waited about 96 seconds for each turn behind a backlog the same churn had built. The network's grants filled up, its new contracts failed, and two cache nodes saturated. - **A backup that blocked cleanup.** After recovery, a full backup held one view of the database for almost 7 hours, so the database could not clean up old row versions, and the busiest small tables grew to millions of dead rows. ## What we changed ### Chapter 1: A scheduled filter emptied provider pools and set off a connection storm - Done (2026-10-22): Hosted proxy windows pace their search for providers after a pass that finds none. - [urnetwork/connect@8033a89](https://github.com/urnetwork/connect/commit/8033a8987861): Merge discovery window expansion pacing - Done (2026-10-22): Churn brakes: hosted proxies keep their backoff across a successful search, reuse identities no provider has seen, create at most two identities at a time and race one candidate instead of two. - [urnetwork/connect@24561f9](https://github.com/urnetwork/connect/commit/24561f97d730): Merge hosted window churn fixes (429 backoff, discovery identity reuse) - [urnetwork/server@ef3f5e6](https://github.com/urnetwork/server/commit/ef3f5e613e62): Merge client-key evidence churn brakes (content-key only, history expiry, mint cap 2, race 1x) - Done (2026-10-22): Each identity's key registration is stored once instead of twice, and the extra copies expire. - [urnetwork/server@e9f2573](https://github.com/urnetwork/server/commit/e9f257398157): Merge client-key registration read from evidence (step 1 of 3) - [urnetwork/server@b878407](https://github.com/urnetwork/server/commit/b87840700008): Merge client-key registration stop-writing (step 2 of 3) - Done (2026-10-24): The filter's start was postponed, first in configuration and then in code to no earlier than Nov 4. - [urnetwork/server@47776d9](https://github.com/urnetwork/server/commit/47776d9608f2): Postpone modern provider gate until November 4 UTC - Done (2026-10-23): A new alert for runaway identity churn. - [urnetwork/server@822f479](https://github.com/urnetwork/server/commit/822f479f9888): Add the hosted identity churn probe (SIGNALS.md ยง14.8) ### Chapter 2: The database disk filled, then payment reservations jammed - Done (2026-10-24): The database server's storage was expanded. - Done (2026-10-24): Load balancers can be brought up after their network interfaces are renamed. - [urnetwork/warp@1a190de](https://github.com/urnetwork/warp/commit/1a190de039fe): Let --initialize-unset bootstrap load-balancer blocks without selectors - Done (2026-10-25): Payment reservations spread across all of a network's grants, a busy balance releases full batches in one pass, and the release job gets its own share of task time. - [urnetwork/server@aec74d3](https://github.com/urnetwork/server/commit/aec74d3f3d28): Spread a multi-grant payer's Redis admissions across its large free grants - [urnetwork/server@066ba17](https://github.com/urnetwork/server/commit/066ba1759aba): Drain a hot debit balance's full batches within one flush page - [urnetwork/server@f68b7b9](https://github.com/urnetwork/server/commit/f68b7b93fc1c): Give the debit flush and the payment sender claim-share lanes - Done (2026-10-24): The busiest small tables are vacuumed every five minutes, and their bloated indexes are rebuilt. - [urnetwork/server@1f2e5bd](https://github.com/urnetwork/server/commit/1f2e5bde9baf): Vacuum the churn tables every five minutes and rebuild their bloated indexes - Done (2026-10-24): An alert on the storage runway and retention of every partitioned table. - [urnetwork/server@22a52d8](https://github.com/urnetwork/server/commit/22a52d8eeb0c): Watch every partition registry family's runway and retention ## Still open - The filter must refuse to turn itself on while it would empty a selection pool; for now it is only postponed. - An alert well before the database disk reaches the log service's reserve, and on how fast the disk is filling. - Regular database backups that can't hold up cleanup: at a quiet hour with the snapshot age watched, or from a replica. - More task-scheduling throughput, so release work can't fall behind a backlog again. Published 2026-10-25; last updated 2026-10-25.