Zurich 15:04:08
UTC 14:04:08
Open incidents P10 P20 P32 P40 Devices down 1/40 2 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
15:03:28 sw-bsl-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to up 15:02:36 sw-bsl-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 15:02:23 sw-zrh-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up 15:01:51 sw-zrh-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down 15:00:56 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to down 14:57:00 INC-1009 opened: Bern: WiFi slow in one area (P3) 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P2 INC-1002 · Easy · reported by Mira Kälin (Monitoring)

Bern: Intranet unavailable, devices not answering

Resolved

  1. Mira Kälin (Monitoring) Report 13:14:47

    Alertmanager went red at 13:14: dc-leaf-1 and srv-dns-1 are down from our point of view. Several users in Bern confirm problems with Intranet. Can the NOC take a look?

  2. hannes Attempt correct 13:35:23

    B: Dispatch a technician to Bern: dc-leaf-1 hardware/power fault

  3. Mira Kälin (Monitoring) Reporter reply 13:35:24

    Our technician found dc-leaf-1 completely dark: both power feeds lost. Breaker reset, it booted, everything is reachable again. Great call! - Mira

Post-mortem

Root cause
dc-leaf-1 lost power (both feeds).
Fault
Power loss
Right answer
B: Dispatch a technician to Bern: dc-leaf-1 hardware/power fault
Solved by
hannes in 20m 36s

dc-leaf-1 lost power at 13:14:46. Devices only reachable through it went silent as well: srv-dns-1. Their logs never reached the collector.

Causal chain

  1. dc-leaf-1 lost power (both feeds).
  2. dc-leaf-1 went dark; its neighbours logged their links and adjacencies to it as down.
  3. Everything only reachable through it became unreachable from srv-mon: srv-dns-1. They were healthy, but their metrics stopped and their logs never arrived.
  4. What was noticed: 2 devices stopped answering and Intranet failed for some users.

Logs that never reached srv-mon: dc-leaf-1, srv-dns-1

Why the other options were wrong

  • Reboot rtr-bsl-1 remotely: it showed a CPU spike at the same time Wrong: it assumed a reboot of rtr-bsl-1 would clear it. rtr-bsl-1's CPU spike was routine background noise (a scheduled job), unrelated. The cause: dc-leaf-1 lost power (both feeds).
  • Reboot dc-leaf-1 remotely: it stopped answering Wrong: it assumed a reboot of dc-leaf-1 would clear it. dc-leaf-1 went dark, but not for that reason. The cause: dc-leaf-1 lost power (both feeds).
  • Dispatch a technician to Bern: srv-dns-1 hardware/power fault Wrong: it assumed srv-dns-1 had a hardware or power fault. srv-dns-1 only looked down: it sits behind the failure, so pings, scrapes and its logs could not get through. It was healthy throughout. The cause: dc-leaf-1 lost power (both feeds).

Timeline

  1. 13:14:47 INC-1002 opened: Bern: Intranet unavailable, devices not answering (Mira Kälin (Monitoring))
  2. 13:35:23 hannes chose B: correct (+250)
  3. 13:35:24 Resolved by hannes; fault repaired

Inbox

5