POSTMORTEMS

Ausfälle

URnetwork wächst schnell, und Wachstum findet die Grenzen eines Systems. Wenn wir an eine stoßen, schreiben wir es hier auf: wann es begann und endete, was du gesehen hast, was schiefging und was wir geändert haben.

Letzte 3 Monate

Jede Zeile ist ein Monat, die Tage stehen untereinander. Höhe und Farbe zeigen beide, wie stark der Dienst betroffen war.

Okt. 2026
Sept. 2026
Aug. 2026
  • Normalbetrieb
  • Beeinträchtigt
  • Teilausfall
  • Schwerer Ausfall
  • Totalausfall

Oktober 2026

3 Ausfälle

21.–25. Okt. 2026

99 Std. 44 Min.

Runaway connection churn filled the database disk and took the platform down

A provider-quality filter switched itself on at its default date and emptied two selection pools, and our hosted proxy service began churning through short-lived connections faster than the database could serve them. For two days connections failed or stalled for many people while database storage grew with the churn, until on Oct 23 a logging fault filled the disk and the platform was down for 16 hours, then 6 more for the storage expansion. That evening the same churn jammed payment reservations, and for about five hours apps could find providers but not send traffic.

Begonnen 21. Okt., 00:00 UTCBeendet 25. Okt., 03:44 UTC
  • Totalausfall23 Std. 32 Min.
  • Schwerer Ausfall14 Std. 04 Min.
  • Teilausfall43 Std. 27 Min.
  • Beeinträchtigt17 Std. 38 Min.
Postmortem lesen

19. Okt. 2026

9 Std. 16 Min.

Stalled database connection pools took providers offline

The pools that connect our servers to the database stalled and could not refill. A cleanup job then mistook the stalled servers for dead ones and removed them, and 97% of providers dropped off the network for almost three hours.

Begonnen 19. Okt., 09:56 UTCBeendet 19. Okt., 19:12 UTC
  • Totalausfall2 Std. 47 Min.
  • Schwerer Ausfall16 Min.
  • Teilausfall26 Min.
  • Beeinträchtigt4 Std. 05 Min.
Postmortem lesen

10.–13. Okt. 2026

80 Std. 19 Min.

Three days of database overload, and the failures it set off

Our main database ran at its limit for three days, and every timeout added retries and more load. While it was overloaded, four separate failures hit users: a new connection check that disconnected healthy providers, a load-balancer collapse, a key rotation that broke new sign-ins, and a stalled connection pooler.

Begonnen 10. Okt., 00:22 UTCBeendet 13. Okt., 08:41 UTC
  • Totalausfall2 Std. 10 Min.
  • Schwerer Ausfall35 Std. 35 Min.
  • Teilausfall19 Std. 34 Min.
  • Beeinträchtigt23 Std.
Postmortem lesen