# Stalled database connection pools took providers offline > The pools that connect our servers to the database stalled and could not refill. A cleanup job then mistook the stalled servers for dead ones and removed them, and 97% of providers dropped off the network for almost three hours. Started Oct 19 09:56 UTC; ended Oct 19 19:12 UTC, 9 h 16 min. Worst level: Full outage. https://ur.io/outages/2026-10-19 At 09:56 most of the pools that connect our servers to the database lost their connections and could not get new ones. Twenty-five minutes later, a cleanup job read the servers' missed check-ins as proof they were dead and removed them, and the servers shut themselves down. Connected providers fell from 75,000 to 1,800. A timeout change ended the stall at 13:15, and providers were back within ten minutes. The stall returned at 15:07 in a milder form until a patched pooler reached every pool at 19:12. ## Phases | From (UTC) | To (UTC) | Level | What people saw | |---|---|---|---| | Oct 19 09:56 | Oct 19 10:12 | Partial outage | 24 of 32 connection pools stalled. Requests that needed the database failed; the other 8 pools carried the load. | | Oct 19 10:12 | Oct 19 10:28 | Major outage | 31 of 32 pools stalled. New connections fell to zero. | | Oct 19 10:28 | Oct 19 13:15 | Full outage | Servers restarted in a loop after the cleanup job removed them. Connected providers fell from 75,000 to 1,800. | | Oct 19 13:15 | Oct 19 13:25 | Partial outage | Pools recovered in under a minute and providers reconnected: 10,400, then 58,000, then 72,600. | | Oct 19 13:25 | Oct 19 15:07 | Healthy | Normal service: 73,000 to 74,000 providers. | | Oct 19 15:07 | Oct 19 19:12 | Degraded | Pools stalled again after a 2 to 3 second database pause. Providers stayed connected and connections still worked, but the database ran at under a third of its usual throughput. | ## Key moments - Oct 19 09:56: 24 of the 32 database connection pools lose their connections within two seconds and can't get new ones. Seven more follow at 10:12. - Oct 19 10:21: The cleanup job deletes the record of every running server, because none has checked in for two minutes. Each server shuts itself down; replacements fail their startup check and cycle every two to three minutes. - Oct 19 11:46: Low point: 1,800 providers connected, down from 75,000. - ~Oct 19 12:50: A team member notices the provider outage. No alert had fired. - Oct 19 13:14: A shorter queue limit is applied to the stalled pools without restarting them. Every pool recovers within a minute, and providers reconnect. - Oct 19 15:07: A 2 to 3 second database pause stalls 31 pools again. Providers stay connected. - ~Oct 19 18:50: The drop is noticed on a database throughput graph. - Oct 19 19:08: A patched pooler rolls out to the 32 pools, one at a time. The stall ends at 19:12. ## Who was affected - **People using the apps.** From about 10:13 new connections failed, and connections already open dropped as providers went offline from 10:28. Service came back at 13:15. - **Providers.** About 97% were disconnected between 10:28 and 13:15. They reconnected on their own once the database recovered; nothing needed to be done on their side. - **During the second stall, 15:07 to 19:12.** Providers stayed connected and connections worked. We did not measure how many people saw slower connections. ## What went wrong - **A pooler bug we already knew about.** The pooler hands each freed database connection to the request that has waited longest, without checking that it is still waiting. Our servers give up after 5 seconds, but requests could queue for 10, so every freed connection went to a request that had already given up, and an emptied pool never refilled. We first saw this on Oct 12. A shorter queue limit that works around it was merged on Oct 15, but it was never applied to the database server. - **A cleanup job that couldn't tell a dead server from an unreachable database.** Every minute, a job deletes the records of servers that haven't checked in for two minutes, and a server whose record disappears shuts itself down. With the database path failing, every server was late at once, while the cleanup job could still reach the database. It removed them all, and a database problem became a network-wide provider outage. - **Nothing alerted us.** No check watched the pools' state or the number of connected providers. A person noticed about three hours after it began, and the second stall went unnoticed for nearly four hours. ## What we changed - Done (2026-10-19): Applied the shorter queue limit to all 32 pools and checked it on each one. - Done (2026-10-19): Fixed the pooler itself: before it hands over a connection, it now drops requests that have already given up. Rolled out to all 32 pools. - [urnetwork/pgbouncer@5594f22](https://github.com/urnetwork/pgbouncer/commit/5594f223307a): Drop waiting clients that hung up before giving them a server - Done (2026-10-19): The cleanup job no longer removes servers while the database is unreachable, and a server no longer shuts down just because its record is gone. - [urnetwork/server@53e8585](https://github.com/urnetwork/server/commit/53e858563bd8): Merge keeping live connect handlers through a database-path outage - Done (2026-10-19): New alerts for stalled pools and for a sudden drop in connected providers. - [urnetwork/server@06a346a](https://github.com/urnetwork/server/commit/06a346a2a067): Merge paging on PgBouncer pool starvation and on a collapse of connected providers - Done (2026-10-20): Every check after a production change now includes the core vitals, pools and connected providers, not only the part that changed. - [urnetwork/server@69dc853](https://github.com/urnetwork/server/commit/69dc85327b6a): Merge always running the tier-0 vitals in focused monitor runs - Done (2026-10-20): Server logs are kept for longer, so the next trigger can be traced. - Done (2026-10-20): Found what causes the short database pauses: a disk-write setting that was merged on Oct 12 but never applied. It is live now. ## Still open Nothing from this outage is still open. Published 2026-10-25; last updated 2026-10-25.