URnetwork
•••

Postmortem · Oct 10 – 13, 2026

Three days of database overload, and the failures it set off

Started
00:22 UTC
Saturday, Oct 10, 2026
Ended
08:41 UTC
Tuesday, Oct 13, 2026
Duration
80 h 19 min
From the first failure to recovery
Worst
Full outage
2 h 10 min in 3 stretches

From Oct 10 our main database ran at its limit. Each slowdown turned into timeouts, each timeout into retries, and the retries into more load, so the overload fed itself. A new connection check that disconnected providers whenever the database was slow made it worse, and late on Oct 11, with our load balancers collapsing in turns, traffic fell to about a tenth of normal. The database was out of its overload loop by 08:09 on Oct 12, but that evening a key rotation broke new sign-ins and the connection pooler stalled. Two planned database restarts on Oct 13 caused the last disruptions, and service was back to normal at 08:41.

TimelineAll times UTC

  • Healthy
  • Degraded
  • Partial outage
  • Major outage
  • Full outage
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
Connected providersReadings from the postmortem; gaps where nothing was read
Start Oct 10, 00:22End Oct 13, 08:41
Connected providers: the readings
Connected providers, as read during the outage
Time (UTC)Connected providers
Oct 10, 01:5288.8k
Oct 11, 02:0045k
Oct 11, 03:5046k
Oct 11, 14:5950k
Oct 11, 16:1771.8k
Oct 12, 01:00about 68k (65–72k until 19:40)
Oct 12, 14:39about 70k
Oct 12, 14:49about 45k (planned fleet restart)
Oct 12, 14:53about 70k
Oct 12, 19:40about 69k
Oct 12, 19:55about 33k
Oct 12, 20:0969.1k
Oct 12, 20:13about 72k
Oct 13, 06:48about 1.9k (the failed restart)
Oct 13, 06:5468k
Oct 13, 07:1266.5k
Oct 13, 07:3769k
Oct 13, 08:25about 32k (slow start after the second restart)
Oct 13, 08:4170k
Oct 13, 11:0074k
Oct 10, 04:13 – Oct 11, 02:00swung between about 15k and 87k
  • Full outage2 h 10 min
  • Major outage35 h 35 min
  • Partial outage19 h 34 min
  • Degraded23 h

Phases

  1. Oct 10, 00:22 – Oct 10, 04:13Partial outage3 h 51 minBest-available and country provider lists came back empty, and providers began dropping off in bursts.
  2. Oct 10, 04:13 – Oct 11, 03:35Major outage23 h 22 minHealthy providers were disconnected in bursts whenever the database was slow, and many people could not connect.
  3. Oct 11, 03:35 – Oct 11, 16:17Partial outage12 h 42 minA fix for the disconnects shipped and providers came back, while the database stayed overloaded.
  4. Oct 11, 16:17 – Oct 11, 18:00Degraded1 h 43 minProviders were back near 72,000; the database was still saturated.
  5. Oct 11, 18:00 – Oct 11, 23:34Major outage5 h 34 minLoad balancers collapsed in turns, each down about 5 minutes in every 10.
  6. Oct 11, 23:34 – Oct 12, 01:17Full outage1 h 43 minTraffic fell to about a tenth of normal.
  7. Oct 12, 01:17 – Oct 12, 05:24Major outage4 h 07 minLoad balancers kept collapsing until a fix reached all of them.
  8. Oct 12, 05:24 – Oct 12, 08:09Partial outage2 h 45 minThe database was still overloaded and many connection attempts failed.
  9. Oct 12, 08:09 – Oct 12, 17:46Degraded9 h 37 minThe database left its overload loop and traffic was back to normal, with short stalls.
  10. Oct 12, 17:46 – Oct 12, 19:45Major outage1 h 59 minNo new sign-in or session refresh worked, and hosted proxies timed out on 12 of 20 proxy groups.
  11. Oct 12, 19:45 – Oct 12, 20:08Full outage23 minThe connection pooler stalled and almost no new connections were accepted.
  12. Oct 12, 20:08 – Oct 12, 20:17Partial outage9 minThe pooler was restarted and providers came back.
  13. Oct 12, 20:17 – Oct 12, 21:10Degraded53 minThe remaining proxy groups were restarted one at a time.
  14. Oct 12, 21:10 – Oct 12, 21:17Partial outage7 minThe pooler stalled again and was fixed within minutes.
  15. Oct 12, 21:17 – Oct 13, 06:45Degraded9 h 28 minShort database freezes, seconds long, while the fix was prepared.
  16. Oct 13, 06:45 – Oct 13, 06:49Full outage4 minA planned database restart failed and was rolled back.
  17. Oct 13, 06:49 – Oct 13, 08:08Degraded1 h 19 minProviders were back while the restart was planned again.
  18. Oct 13, 08:08 – Oct 13, 08:41Major outage33 minThe second restart worked, but new database connections started slowly.

Key moments

  1. Oct 10, 01:57Providers start being disconnected whenever the overloaded database misses a new 2-second session check.
  2. Oct 10, 16:36Contracts are found not to be closing. The backlog dates back to Sep 28 and is adding load of its own.
  3. Oct 10, 22:44Reports arrive that providers struggle to connect and people can't reach them.
  4. Oct 11, 03:26The disconnect fix ships: an unanswered check is retried instead of ending the connection.
  5. Oct 11, 18:00Two load balancers start collapsing in turns under slow TLS handshakes and retry storms.
  6. Oct 12, 02:36A load-balancer build with a faster key exchange starts rolling out. The collapses stop by 05:24.
  7. Oct 12, 08:09The database is out of its overload loop, and traffic is back at its usual level.
  8. Oct 12, 17:46A key rotation breaks new sign-ins: servers started before it reject every sign-in issued after it.
  9. Oct 12, 19:45The connection pooler stalls and almost no new connections are accepted. It is restarted at 20:06–20:08.
  10. Oct 13, 08:08A second planned database restart succeeds. New database connections start slowly until 08:40, and service is normal by 08:41.

Who was affected

People using the apps. Connections were unreliable for most of three days. At the worst point, late on Oct 11, traffic fell to about a tenth of normal. From 17:46 to 20:13 on Oct 12 almost nobody could sign in or refresh a session; sessions already running kept working until they needed a refresh.

Providers. Providers were disconnected about 3 million times on Oct 10 and 11, in bursts. Most reconnected on their own; operators of some third-party miner builds had to restart them by hand. Separately, a bug in recent SDKs re-sent undeliverable return traffic without end, an estimated 10 TB of provider bandwidth from Oct 2 to Oct 9.

Hosted proxy customers. From 17:46 on Oct 12, setting up proxy windows timed out on 12 of 20 proxy groups until the groups were restarted that evening.

What went wrong

  1. The database fed its own overload

    Several independent sources of load grew while the database was already slow: a backlog of contracts that had stopped closing, connections thrown away and reopened on every timeout, old app versions retrying on fixed timers, and scans that only became expensive once a task queue held 117 million rows. Each timeout created more work, so the load fed itself.

  2. A connection check that failed closed

    A new check renewed every connection's authorization each minute with a 2-second database budget. On a saturated database the check went unanswered, and an unanswered check closed the connection: about 3 million provider disconnects, each followed by a reconnect that hit the same database.

  3. Load balancers that collapsed under their own retries

    Our load balancers used a slow key exchange for every new TLS connection. Under load the handshakes got slower, clients redialed harder, and health checks moved each collapsing host's traffic onto the other one.

  4. A key rotation nobody knew was a key rotation

    Renewing a certificate also changed a key that new sign-ins depend on. Servers started before the change kept the old key and rejected every new sign-in until they were restarted.

  5. A connection pooler that couldn't refill

    During the restarts a short database stall emptied most connection pools, and the pooler's queue limit was longer than our servers' own timeouts, so every new connection went to a request that had already given up. The same bug came back on Oct 19, and it is fixed at its source there.

  6. Changes made during the response

    Several of our own changes during the response caused new failures: a cap that made the contract closer skip work, a database-heavy rollout step, restart waves, the certificate renewal, and a cache rebalance that silently dropped proxy subscriptions.

What we changed

Each fix links to the public commits that made it. Changes to our servers' configuration are described without links.

  • Done
    Connection checks retry instead of disconnecting, a check no longer re-runs for every connection in a network, and only a definite refusal ends a session.
  • Done
    Contracts close again: longer scan deadlines, closes batched per payer, progress that survives a timeout, and a background closer that writes at half the rate.
  • Done
    Servers keep their database connections when a caller gives up, instead of throwing them away and opening new ones.
  • Done
    Retries from old app versions are answered cheaply: one read for a missing contract, remembered refusals, and a one-minute hold on refused connections.
  • Done
    A cache script whose cost grew with the square of the connections behind one address is bounded.
  • Done
    Hosted proxies pace their re-dials to an unreachable provider and reuse its identity, and each proxy process registers a bounded number of new clients.
  • Done
    Queued work is retired with its contracts, task workers start without scanning the whole queue, and leftover index copies from interrupted maintenance are cleaned up.
  • Done
    Load balancers use a faster key exchange, and the collapses stopped.
  • Done
    A configuration push no longer restarts running servers across the fleet.
  • Done
    Servers re-read sign-in keys they don't recognize, and a new key waits before it is used, so a rotation can't lock out new sign-ins.
  • Done
    Proxies subscribe again when a cache rebalance moves their channels, and keep watching after a failed read.
  • Done
    New alerts for connection pools with requests queued and no database connection.
  • Done
    Database write and memory settings tuned to end the short freezes, including the restart on Oct 13.
  • In progress
    Recent SDKs bound how long they re-send undeliverable return traffic. The fix reaches providers with the next app and SDK release.

Still open

  • Open
    The app and SDK fixes above reach people and providers only with a new release. Until then, older versions keep retrying on fixed timers.