# Three days of database overload, and the failures it set off > Our main database ran at its limit for three days, and every timeout added retries and more load. While it was overloaded, four separate failures hit users: a new connection check that disconnected healthy providers, a load-balancer collapse, a key rotation that broke new sign-ins, and a stalled connection pooler. Started Oct 10 00:22 UTC; ended Oct 13 08:41 UTC, 80 h 19 min. Worst level: Full outage. https://ur.io/outages/2026-10-10 From Oct 10 our main database ran at its limit. Each slowdown turned into timeouts, each timeout into retries, and the retries into more load, so the overload fed itself. A new connection check that disconnected providers whenever the database was slow made it worse, and late on Oct 11, with our load balancers collapsing in turns, traffic fell to about a tenth of normal. The database was out of its overload loop by 08:09 on Oct 12, but that evening a key rotation broke new sign-ins and the connection pooler stalled. Two planned database restarts on Oct 13 caused the last disruptions, and service was back to normal at 08:41. ## Phases | From (UTC) | To (UTC) | Level | What people saw | |---|---|---|---| | Oct 10 00:22 | Oct 10 04:13 | Partial outage | Best-available and country provider lists came back empty, and providers began dropping off in bursts. | | Oct 10 04:13 | Oct 11 03:35 | Major outage | Healthy providers were disconnected in bursts whenever the database was slow, and many people could not connect. | | Oct 11 03:35 | Oct 11 16:17 | Partial outage | A fix for the disconnects shipped and providers came back, while the database stayed overloaded. | | Oct 11 16:17 | Oct 11 18:00 | Degraded | Providers were back near 72,000; the database was still saturated. | | Oct 11 18:00 | Oct 11 23:34 | Major outage | Load balancers collapsed in turns, each down about 5 minutes in every 10. | | Oct 11 23:34 | Oct 12 01:17 | Full outage | Traffic fell to about a tenth of normal. | | Oct 12 01:17 | Oct 12 05:24 | Major outage | Load balancers kept collapsing until a fix reached all of them. | | Oct 12 05:24 | Oct 12 08:09 | Partial outage | The database was still overloaded and many connection attempts failed. | | Oct 12 08:09 | Oct 12 17:46 | Degraded | The database left its overload loop and traffic was back to normal, with short stalls. | | Oct 12 17:46 | Oct 12 19:45 | Major outage | No new sign-in or session refresh worked, and hosted proxies timed out on 12 of 20 proxy groups. | | Oct 12 19:45 | Oct 12 20:08 | Full outage | The connection pooler stalled and almost no new connections were accepted. | | Oct 12 20:08 | Oct 12 20:17 | Partial outage | The pooler was restarted and providers came back. | | Oct 12 20:17 | Oct 12 21:10 | Degraded | The remaining proxy groups were restarted one at a time. | | Oct 12 21:10 | Oct 12 21:17 | Partial outage | The pooler stalled again and was fixed within minutes. | | Oct 12 21:17 | Oct 13 06:45 | Degraded | Short database freezes, seconds long, while the fix was prepared. | | Oct 13 06:45 | Oct 13 06:49 | Full outage | A planned database restart failed and was rolled back. | | Oct 13 06:49 | Oct 13 08:08 | Degraded | Providers were back while the restart was planned again. | | Oct 13 08:08 | Oct 13 08:41 | Major outage | The second restart worked, but new database connections started slowly. | ## Key moments - Oct 10 01:57: Providers start being disconnected whenever the overloaded database misses a new 2-second session check. - Oct 10 16:36: Contracts are found not to be closing. The backlog dates back to Sep 28 and is adding load of its own. - Oct 10 22:44: Reports arrive that providers struggle to connect and people can't reach them. - Oct 11 03:26: The disconnect fix ships: an unanswered check is retried instead of ending the connection. - Oct 11 18:00: Two load balancers start collapsing in turns under slow TLS handshakes and retry storms. - Oct 12 02:36: A load-balancer build with a faster key exchange starts rolling out. The collapses stop by 05:24. - Oct 12 08:09: The database is out of its overload loop, and traffic is back at its usual level. - Oct 12 17:46: A key rotation breaks new sign-ins: servers started before it reject every sign-in issued after it. - Oct 12 19:45: The connection pooler stalls and almost no new connections are accepted. It is restarted at 20:06–20:08. - Oct 13 08:08: A second planned database restart succeeds. New database connections start slowly until 08:40, and service is normal by 08:41. ## Who was affected - **People using the apps.** Connections were unreliable for most of three days. At the worst point, late on Oct 11, traffic fell to about a tenth of normal. From 17:46 to 20:13 on Oct 12 almost nobody could sign in or refresh a session; sessions already running kept working until they needed a refresh. - **Providers.** Providers were disconnected about 3 million times on Oct 10 and 11, in bursts. Most reconnected on their own; operators of some third-party miner builds had to restart them by hand. Separately, a bug in recent SDKs re-sent undeliverable return traffic without end, an estimated 10 TB of provider bandwidth from Oct 2 to Oct 9. - **Hosted proxy customers.** From 17:46 on Oct 12, setting up proxy windows timed out on 12 of 20 proxy groups until the groups were restarted that evening. ## What went wrong - **The database fed its own overload.** Several independent sources of load grew while the database was already slow: a backlog of contracts that had stopped closing, connections thrown away and reopened on every timeout, old app versions retrying on fixed timers, and scans that only became expensive once a task queue held 117 million rows. Each timeout created more work, so the load fed itself. - **A connection check that failed closed.** A new check renewed every connection's authorization each minute with a 2-second database budget. On a saturated database the check went unanswered, and an unanswered check closed the connection: about 3 million provider disconnects, each followed by a reconnect that hit the same database. - **Load balancers that collapsed under their own retries.** Our load balancers used a slow key exchange for every new TLS connection. Under load the handshakes got slower, clients redialed harder, and health checks moved each collapsing host's traffic onto the other one. - **A key rotation nobody knew was a key rotation.** Renewing a certificate also changed a key that new sign-ins depend on. Servers started before the change kept the old key and rejected every new sign-in until they were restarted. - **A connection pooler that couldn't refill.** During the restarts a short database stall emptied most connection pools, and the pooler's queue limit was longer than our servers' own timeouts, so every new connection went to a request that had already given up. The same bug came back on Oct 19, and it is fixed at its source there. - **Changes made during the response.** Several of our own changes during the response caused new failures: a cap that made the contract closer skip work, a database-heavy rollout step, restart waves, the certificate renewal, and a cache rebalance that silently dropped proxy subscriptions. ## What we changed - Done (2026-10-11): Connection checks retry instead of disconnecting, a check no longer re-runs for every connection in a network, and only a definite refusal ends a session. - [urnetwork/server@6688331](https://github.com/urnetwork/server/commit/6688331cfd9e): Merge fix/session-lease-unavailable-retry - [urnetwork/server@7bba6cd](https://github.com/urnetwork/server/commit/7bba6cdb75ec): Merge fix/session-listener-fanout - [urnetwork/server@963c4a5](https://github.com/urnetwork/server/commit/963c4a583bf3): Merge keeping admitted transports through unanswered lease rechecks - Done (2026-10-12): Contracts close again: longer scan deadlines, closes batched per payer, progress that survives a timeout, and a background closer that writes at half the rate. - [urnetwork/server@7e380ef](https://github.com/urnetwork/server/commit/7e380ef16d61): Merge the 24 hour startup contract scan deadline - [urnetwork/server@6507361](https://github.com/urnetwork/server/commit/650736196b40): Merge scheduled close owner batches and the per-instance claim bound - [urnetwork/server@e61ed84](https://github.com/urnetwork/server/commit/e61ed8451ef1): Merge the removal of the queued scheduled close claim bound - [urnetwork/server@6e415d8](https://github.com/urnetwork/server/commit/6e415d8dbdc4): Merge deferred expiry rows and session maintenance progress - [urnetwork/server@c0e5f33](https://github.com/urnetwork/server/commit/c0e5f33f896a): Halve the background closer's write concurrency - Done (2026-10-12): Servers keep their database connections when a caller gives up, instead of throwing them away and opening new ones. - [urnetwork/server@bec961c](https://github.com/urnetwork/server/commit/bec961cb64e8): Merge keeping pooled connections through caller stops at BEGIN - [urnetwork/server@037d68c](https://github.com/urnetwork/server/commit/037d68c94119): Merge keeping pooled sessions when a caller stops during a transaction statement - Done (2026-10-12): Retries from old app versions are answered cheaply: one read for a missing contract, remembered refusals, and a one-minute hold on refused connections. - [urnetwork/server@95e8c09](https://github.com/urnetwork/server/commit/95e8c09a13b4): Merge one-read companion origin fallback for non-companion requests - [urnetwork/server@411de53](https://github.com/urnetwork/server/commit/411de537b9f2): Merge remembering definitive credential refusals and closing refused handshakes in band - [urnetwork/server@64dac87](https://github.com/urnetwork/server/commit/64dac87a2fa4): Merge holding a refused old-build websocket for a minute before its authorization close - [urnetwork/server@20266ee](https://github.com/urnetwork/server/commit/20266ee74906): Merge keeping terminal contract route refusals and deciding missing origins first - [urnetwork/server@a204771](https://github.com/urnetwork/server/commit/a20477198a6d): Merge answering recorded contract refusals before the state query and keeping missing-origin refusals 60s - [urnetwork/connect@23b3b0f](https://github.com/urnetwork/connect/commit/23b3b0f352e0): Merge holding a definitively refused credential and reporting it to its owner - Done (2026-10-11): A cache script whose cost grew with the square of the connections behind one address is bounded. - [urnetwork/server@ec43360](https://github.com/urnetwork/server/commit/ec4336060d38): Merge bounding the forward egress lease script to the owner it updates - Done (2026-10-12): Hosted proxies pace their re-dials to an unreachable provider and reuse its identity, and each proxy process registers a bounded number of new clients. - [urnetwork/connect@5f71903](https://github.com/urnetwork/connect/commit/5f71903c796d): Merge pacing re-dials of an unreachable fixed destination and reusing its identity - [urnetwork/connect@ce63fe2](https://github.com/urnetwork/connect/commit/ce63fe2e4c7f): Merge keeping a reused fixed-destination identity's key across re-dials - [urnetwork/server@d496def](https://github.com/urnetwork/server/commit/d496defa1325): Merge bounding hosted derived-client registrations per proxy process - Done (2026-10-12): Queued work is retired with its contracts, task workers start without scanning the whole queue, and leftover index copies from interrupted maintenance are cleaned up. - [urnetwork/server@098dc9f](https://github.com/urnetwork/server/commit/098dc9f80814): Merge retiring queued scheduled closes with their contracts - [urnetwork/server@d409027](https://github.com/urnetwork/server/commit/d40902757371): Merge keeping the retired-task reap out of taskworker startup scheduling - [urnetwork/server@1b567df](https://github.com/urnetwork/server/commit/1b567dfd6861): Merge bounding the taskworker queue metrics refresh on the server - [urnetwork/server@c292be1](https://github.com/urnetwork/server/commit/c292be18df11): Merge cleaning up interrupted reindex copies and guarding the rotation by load and size - Done (2026-10-12): Load balancers use a faster key exchange, and the collapses stopped. - [urnetwork/warp@f30f6fc](https://github.com/urnetwork/warp/commit/f30f6fc2e78c): Merge fix/lb-x25519-curve: LB TLS prefers X25519 key exchange - Done (2026-10-11): A configuration push no longer restarts running servers across the fleet. - [urnetwork/warp@4cd6bf7](https://github.com/urnetwork/warp/commit/4cd6bf7db357): Merge fix/config-restart-no-hold: a restart: false config restarts no running block - Done (2026-10-12): Servers re-read sign-in keys they don't recognize, and a new key waits before it is used, so a rotation can't lock out new sign-ins. - [urnetwork/server@95e2ce2](https://github.com/urnetwork/server/commit/95e2ce2386b5): Merge rereading JWT keys on an unknown kid and letting new signing keys settle - Done (2026-10-13): Proxies subscribe again when a cache rebalance moves their channels, and keep watching after a failed read. - [urnetwork/server@466a006](https://github.com/urnetwork/server/commit/466a006dab2e): Resubscribe sharded pub/sub channels after their slot moves - [urnetwork/server@24feb5f](https://github.com/urnetwork/server/commit/24feb5fb034a): Keep the proxy client watch and reconcile running after a raised change read - Done (2026-10-13): New alerts for connection pools with requests queued and no database connection. - [urnetwork/warp@a90bade](https://github.com/urnetwork/warp/commit/a90badebfa3e): Merge alerting when a PgBouncer pool has clients queued and no server - Done (2026-10-13): Database write and memory settings tuned to end the short freezes, including the restart on Oct 13. - In progress (2026-10-11): Recent SDKs bound how long they re-send undeliverable return traffic. The fix reaches providers with the next app and SDK release. - [urnetwork/connect@e48e4bb](https://github.com/urnetwork/connect/commit/e48e4bb3f78d): Merge bounding retained return retransmission to a silent destination ## Still open - The app and SDK fixes above reach people and providers only with a new release. Until then, older versions keep retrying on fixed timers. Published 2026-10-25; last updated 2026-10-25.