URnetwork
•••

Postmortem · Monday, Oct 19, 2026

Stalled database connection pools took providers offline

Started
09:56 UTC
Monday, Oct 19, 2026
Ended
19:12 UTC
Monday, Oct 19, 2026
Duration
9 h 16 min
Healthy again in between, 13:25 – 15:07
Worst
Full outage
2 h 47 min, 10:28 – 13:15

At 09:56 most of the pools that connect our servers to the database lost their connections and could not get new ones. Twenty-five minutes later, a cleanup job read the servers' missed check-ins as proof they were dead and removed them, and the servers shut themselves down. Connected providers fell from 75,000 to 1,800. A timeout change ended the stall at 13:15, and providers were back within ten minutes. The stall returned at 15:07 in a milder form until a patched pooler reached every pool at 19:12.

TimelineAll times UTC

  • Healthy
  • Degraded
  • Partial outage
  • Major outage
  • Full outage
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
Connected providersReadings from the postmortem; straight lines between them
Start 09:56End 19:12
Connected providers: the readings
Connected providers, as read during the outage
Time (UTC)Connected providers
10:26about 75k (held until here)
10:3633.7k
11:461.8k
13:13about 2.1k (near 2.1k until the pools recovered)
13:2572.6k
14:1874.3k
19:2075.3k (72–75k through the second stall)
  • Full outage2 h 47 min
  • Major outage16 min
  • Partial outage26 min
  • Degraded4 h 05 min

Phases

  1. 09:56 – 10:12Partial outage16 min24 of 32 connection pools stalled. Requests that needed the database failed; the other 8 pools carried the load.
  2. 10:12 – 10:28Major outage16 min31 of 32 pools stalled. New connections fell to zero.
  3. 10:28 – 13:15Full outage2 h 47 minServers restarted in a loop after the cleanup job removed them. Connected providers fell from 75,000 to 1,800.
  4. 13:15 – 13:25Partial outage10 minPools recovered in under a minute and providers reconnected: 10,400, then 58,000, then 72,600.
  5. 13:25 – 15:07Healthy1 h 42 minNormal service: 73,000 to 74,000 providers.
  6. 15:07 – 19:12Degraded4 h 05 minPools stalled again after a 2 to 3 second database pause. Providers stayed connected and connections still worked, but the database ran at under a third of its usual throughput.

Key moments

  1. 09:5624 of the 32 database connection pools lose their connections within two seconds and can't get new ones. Seven more follow at 10:12.
  2. 10:21The cleanup job deletes the record of every running server, because none has checked in for two minutes. Each server shuts itself down; replacements fail their startup check and cycle every two to three minutes.
  3. 11:46Low point: 1,800 providers connected, down from 75,000.
  4. ~12:50A team member notices the provider outage. No alert had fired.
  5. 13:14A shorter queue limit is applied to the stalled pools without restarting them. Every pool recovers within a minute, and providers reconnect.
  6. 15:07A 2 to 3 second database pause stalls 31 pools again. Providers stay connected.
  7. ~18:50The drop is noticed on a database throughput graph.
  8. 19:08A patched pooler rolls out to the 32 pools, one at a time. The stall ends at 19:12.

Who was affected

People using the apps. From about 10:13 new connections failed, and connections already open dropped as providers went offline from 10:28. Service came back at 13:15.

Providers. About 97% were disconnected between 10:28 and 13:15. They reconnected on their own once the database recovered; nothing needed to be done on their side.

During the second stall, 15:07 to 19:12. Providers stayed connected and connections worked. We did not measure how many people saw slower connections.

What went wrong

  1. A pooler bug we already knew about

    The pooler hands each freed database connection to the request that has waited longest, without checking that it is still waiting. Our servers give up after 5 seconds, but requests could queue for 10, so every freed connection went to a request that had already given up, and an emptied pool never refilled. We first saw this on Oct 12. A shorter queue limit that works around it was merged on Oct 15, but it was never applied to the database server.

  2. A cleanup job that couldn't tell a dead server from an unreachable database

    Every minute, a job deletes the records of servers that haven't checked in for two minutes, and a server whose record disappears shuts itself down. With the database path failing, every server was late at once, while the cleanup job could still reach the database. It removed them all, and a database problem became a network-wide provider outage.

  3. Nothing alerted us

    No check watched the pools' state or the number of connected providers. A person noticed about three hours after it began, and the second stall went unnoticed for nearly four hours.

What we changed

Each fix links to the public commits that made it. Changes to our servers' configuration are described without links.

  • Done
    Applied the shorter queue limit to all 32 pools and checked it on each one.
  • Done
    Fixed the pooler itself: before it hands over a connection, it now drops requests that have already given up. Rolled out to all 32 pools.
  • Done
    The cleanup job no longer removes servers while the database is unreachable, and a server no longer shuts down just because its record is gone.
  • Done
    New alerts for stalled pools and for a sudden drop in connected providers.
  • Done
    Every check after a production change now includes the core vitals, pools and connected providers, not only the part that changed.
  • Done
    Server logs are kept for longer, so the next trigger can be traced.
  • Done
    Found what causes the short database pauses: a disk-write setting that was merged on Oct 12 but never applied. It is live now.

Still open

Nothing from this outage is still open.