URnetwork
•••

Postmortem · Oct 21 – 25, 2026

Runaway connection churn filled the database disk and took the platform down

Started
00:00 UTC
Wednesday, Oct 21, 2026
Ended
03:44 UTC
Sunday, Oct 25, 2026
Duration
99 h 44 min
Healthy again in between, Oct 25, 01:27 – Oct 25, 02:30
Worst
Full outage
23 h 32 min in 3 stretches

On Oct 21 a provider-quality filter switched itself on at its default date and left almost no providers in two of our selection pools. Our hosted proxy service, trying to replace providers that failed, began creating short-lived connections far faster than the database could serve them, and for two days connections failed or stalled for many people. Every one of those connections also added to the database's storage. On Oct 23 a logging fault on the nearly full disk filled it completely, and the platform was down for 16 hours, then 6 more while we expanded the storage. That evening the same churn tied up payment reservations, and for about five hours apps could find providers but not send traffic.

TimelineAll times UTC

  • Healthy
  • Degraded
  • Partial outage
  • Major outage
  • Full outage
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
Free space on the database diskReadings from the postmortems; straight lines between them
Start Oct 21, 00:00Chapter 2End Oct 25, 03:44
Free space on the database disk: the readings
Free space on the database disk, as read during the outage
Time (UTC)Free space on the database disk
Oct 21, 00:00390 GB
Oct 21, 13:00334 GB
Oct 21, 15:00306 GB
Oct 21, 23:57145 GB
Oct 22, 00:00345 GB (after an index rebuild and a nightly cleanup)
Oct 22, 05:50249 GB
Oct 22, 06:02467 GB (after old data was dropped)
Oct 23, 06:30about 200 GB (reconstructed afterwards)
Oct 23, 11:120 GB (full; the database stopped)
Oct 23, 20:05129 GB (log files removed)
Oct 23, 21:44360 GB (after a full vacuum)
  • Full outage23 h 32 min
  • Major outage14 h 04 min
  • Partial outage43 h 27 min
  • Degraded17 h 38 min

Phases

Chapter 1A scheduled filter emptied provider pools and set off a connection storm

  1. Oct 21, 00:00 – Oct 21, 14:00Degraded14 hTwo selection pools were nearly empty; most people were not affected yet.
  2. Oct 21, 14:00 – Oct 21, 15:00Partial outage1 hDatabase queueing began and connections started to fail.
  3. Oct 21, 15:00 – Oct 22, 02:08Major outage11 h 08 minApps found providers but connections failed or stalled for many people.
  4. Oct 22, 02:08 – Oct 22, 13:45Partial outage11 h 37 minPacing and the postponed filter eased it, but the churn kept going on its own.
  5. Oct 22, 13:45 – Oct 22, 14:41Degraded56 minChurn brakes broke the loop.
  6. Oct 22, 14:41 – Oct 23, 11:12Partial outage20 h 31 minThe loop came back under daytime load, and connections stayed unreliable until the disk filled.

Chapter 2The database disk filled, then payment reservations jammed

  1. Oct 23, 11:12 – Oct 24, 03:20Full outage16 h 08 minThe database server's disk filled and the database stopped. The whole platform was down.
  2. Oct 24, 03:20 – Oct 24, 05:29Partial outage2 h 09 minServing again on most servers while the storage was expanded.
  3. Oct 24, 05:29 – Oct 24, 11:29Full outage6 hA second, planned full outage so the storage expansion could finish.
  4. Oct 24, 11:29 – Oct 24, 16:48Partial outage5 h 19 minBack with reduced capacity: six load-balancer groups served nothing after an operating-system upgrade renamed their network interfaces.
  5. Oct 24, 16:48 – Oct 24, 19:30Degraded2 h 42 minFull capacity was back while backlogs drained.
  6. Oct 24, 19:30 – Oct 24, 20:54Full outage1 h 24 minApps could find providers but could not send traffic.
  7. Oct 24, 20:54 – Oct 24, 21:30Partial outage36 minExtra data grants restored traffic for about 35 minutes.
  8. Oct 24, 21:30 – Oct 25, 00:26Major outage2 h 56 minThe grants filled up and failures returned until the fix was ready.
  9. Oct 25, 00:26 – Oct 25, 01:27Partial outage1 h 01 minThe fix rolled out in stages.
  10. Oct 25, 01:27 – Oct 25, 02:30Healthy1 h 03 minHealthy.
  11. Oct 25, 02:30 – Oct 25, 03:44Partial outage1 h 14 minSlow again: a long-running backup kept the database from cleaning up.

Key moments

  1. Oct 21, 00:00A provider-quality filter turns itself on at its default date. It was meant to wait for a decision that never got made.
  2. Oct 21, 14:00Database queueing starts as our hosted proxy service creates and discards connections hundreds of times a second.
  3. Oct 22, 02:08Discovery pacing ships to every proxy, and refused connection attempts fall by about 90%.
  4. Oct 22, 04:03The filter is postponed. The churn keeps going on its own.
  5. Oct 22, 13:45Churn brakes break the loop, until daytime load brings it back at 14:41.
  6. Oct 23, 11:12The database server's disk fills and the database stops.
  7. Oct 24, 05:29Every service stops so the storage expansion can finish. The platform is back at 11:29.
  8. Oct 24, 19:30Apps can find providers but not send traffic.
  9. Oct 25, 00:26The fix for the payment-reservation stall starts rolling out. Service is healthy at 01:27.
  10. Oct 25, 03:42A long-running backup is stopped, and the database cleans up within minutes.

Who was affected

People using the apps. From 14:00 on Oct 21 to the evening of Oct 22, connections failed or stalled for many people even though the app showed providers. The platform was down from 11:12 on Oct 23 to 03:20 on Oct 24, and again from 05:29 to 11:29. From 19:30 on Oct 24 to 01:27 on Oct 25, apps could find providers but could not send traffic.

Hosted proxy customers. Hosted proxies were hit first and hardest. From Oct 21 their windows churned through short-lived identities, and most of those identities never connected.

Providers. Providers could not connect while the platform was down.

Data. We know of no lost data.

What went wrong

Chapter 1A scheduled filter emptied provider pools and set off a connection storm

  1. A scheduled change went ahead by default

    The filter's start date was a default in code, so it switched on without anyone deciding. Our own measurements had shown it would remove more than 90% of one pool and leave the other with a single provider, and the deadline to decide passed without a decision.

  2. Churn that kept itself going

    Our hosted proxy service keeps windows of candidate providers, each tried with a newly created identity. With the pools empty most candidates failed, so windows created more. Every identity costs database work; the database slowed, connections took a median of 14 seconds, and windows discarded identities before they could connect, creating still more. The loop kept going after the filter was postponed.

  3. Storage that only grows

    Every short-lived identity left permanent records: a signed key registration stored twice, evidence objects, and contracts that are kept for months. The database grew by up to 20 GB an hour, and the disk-runway warning we planned after the Oct 12 near-miss had not been built.

Chapter 2The database disk filled, then payment reservations jammed

  1. A logging fault on a nearly full disk

    Once free space fell below the log service's reserve, the service started a new log file about three times a second and never removed old ones, writing about 200 GB in a few hours. The disk filled and the database stopped.

  2. A storage expansion that couldn't run under load

    The disk controller's online expansion made no progress while the database was serving, so finishing it took a second full outage.

  3. Payment reservations released slower than they were made

    Our hosted proxy service's network made hundreds of tiny contracts a second, each holding a reservation for up to 24 hours. A background job released them, but it waited about 96 seconds for each turn behind a backlog the same churn had built. The network's grants filled up, its new contracts failed, and two cache nodes saturated.

  4. A backup that blocked cleanup

    After recovery, a full backup held one view of the database for almost 7 hours, so the database could not clean up old row versions, and the busiest small tables grew to millions of dead rows.

What we changed

Each fix links to the public commits that made it. Changes to our servers' configuration are described without links.

Chapter 1A scheduled filter emptied provider pools and set off a connection storm

  • Done
    Hosted proxy windows pace their search for providers after a pass that finds none.
  • Done
    Churn brakes: hosted proxies keep their backoff across a successful search, reuse identities no provider has seen, create at most two identities at a time and race one candidate instead of two.
  • Done
    Each identity's key registration is stored once instead of twice, and the extra copies expire.
  • Done
    The filter's start was postponed, first in configuration and then in code to no earlier than Nov 4.
  • Done
    A new alert for runaway identity churn.

Chapter 2The database disk filled, then payment reservations jammed

  • Done
    The database server's storage was expanded.
  • Done
    Load balancers can be brought up after their network interfaces are renamed.
  • Done
    Payment reservations spread across all of a network's grants, a busy balance releases full batches in one pass, and the release job gets its own share of task time.
  • Done
    The busiest small tables are vacuumed every five minutes, and their bloated indexes are rebuilt.
  • Done
    An alert on the storage runway and retention of every partitioned table.

Still open

  • Open
    The filter must refuse to turn itself on while it would empty a selection pool; for now it is only postponed.
  • Open
    An alert well before the database disk reaches the log service's reserve, and on how fast the disk is filling.
  • Open
    Regular database backups that can't hold up cleanup: at a quiet hour with the snapshot age watched, or from a replica.
  • Open
    More task-scheduling throughput, so release work can't fall behind a backlog again.