Zurich 14:42:57
UTC 13:42:57
Open incidents P10 P21 P30 P41 Devices down 0/40 10 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:42:12 acc-brn-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/9, changed state to down 14:42:06 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/11, changed state to up 14:40:08 rtr-lug-1 %CELLWAN-2-NC_EVENT: Cellular0/2/0: Network change event, Radio Access Technology: LTE, Band: 3, RSRP: -97 dBm 14:39:44 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4) 14:00:01 INC-1004 expired: SLA breached 13:47:01 INC-1005 opened: Bern: network alarm, no user impact so far (P3) 13:46:00 INC-1003 expired: SLA breached 13:38:26 Chaos monkey off (every ~300s, max 2) 13:38:25 Chaos monkey on (every ~300s, max 2)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P2 INC-1002 · Easy · reported by Mira Kälin (Monitoring)

Bern: Intranet unavailable, devices not answering

Resolved

  1. Mira Kälin (Monitoring) Report 13:14:47

    Alertmanager went red at 13:14: dc-leaf-1 and srv-dns-1 are down from our point of view. Several users in Bern confirm problems with Intranet. Can the NOC take a look?

  2. hannes Attempt correct 13:35:23

    B: Dispatch a technician to Bern: dc-leaf-1 hardware/power fault

  3. Mira Kälin (Monitoring) Reporter reply 13:35:24

    Our technician found dc-leaf-1 completely dark: both power feeds lost. Breaker reset, it booted, everything is reachable again. Great call! - Mira

Post-mortem

Root cause
dc-leaf-1 lost power (both feeds).
Fault
Power loss
Right answer
B: Dispatch a technician to Bern: dc-leaf-1 hardware/power fault
Solved by
hannes in 20m 36s

dc-leaf-1 lost power at 13:14:46. Devices only reachable through it went silent as well: srv-dns-1. Their logs never reached the collector.

Causal chain

  1. dc-leaf-1 lost power (both feeds).
  2. dc-leaf-1 went dark; its neighbours logged their links and adjacencies to it as down.
  3. Everything only reachable through it became unreachable from srv-mon: srv-dns-1. They were healthy, but their metrics stopped and their logs never arrived.
  4. What was noticed: 2 devices stopped answering and Intranet failed for some users.

Logs that never reached srv-mon: dc-leaf-1, srv-dns-1

Why the other options were wrong

  • Reboot rtr-bsl-1 remotely: it showed a CPU spike at the same time Wrong: it assumed a reboot of rtr-bsl-1 would clear it. rtr-bsl-1's CPU spike was routine background noise (a scheduled job), unrelated. The cause: dc-leaf-1 lost power (both feeds).
  • Reboot dc-leaf-1 remotely: it stopped answering Wrong: it assumed a reboot of dc-leaf-1 would clear it. dc-leaf-1 went dark, but not for that reason. The cause: dc-leaf-1 lost power (both feeds).
  • Dispatch a technician to Bern: srv-dns-1 hardware/power fault Wrong: it assumed srv-dns-1 had a hardware or power fault. srv-dns-1 only looked down: it sits behind the failure, so pings, scrapes and its logs could not get through. It was healthy throughout. The cause: dc-leaf-1 lost power (both feeds).

Timeline

  1. 13:14:47 INC-1002 opened: Bern: Intranet unavailable, devices not answering (Mira Kälin (Monitoring))
  2. 13:35:23 hannes chose B: correct (+250)
  3. 13:35:24 Resolved by hannes; fault repaired

Inbox

2