Zurich 14:36:37
UTC 13:36:37
Open incidents P10 P21 P30 P41 Devices down 0/40 10 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:36:28 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/11, changed state to down 14:36:10 acc-brn-2 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 14:34:35 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up 14:34:18 sw-zrh-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up 14:34:07 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to down 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4) 14:00:01 INC-1004 expired: SLA breached 13:47:01 INC-1005 opened: Bern: network alarm, no user impact so far (P3) 13:46:00 INC-1003 expired: SLA breached 13:38:26 Chaos monkey off (every ~300s, max 2)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P2 INC-1003 · Easy · reported by Luca Brunner (Service Desk)

Bern: Intranet unavailable, devices not answering

SLA breached

  1. Luca Brunner (Service Desk) Report 13:16:00

    Several users in Bern can't use Intranet since around 13:16. It looks like only some areas are affected. Forwarded from the service desk, users keep calling. Can you have a look?

  2. NOC-POC System 13:46:00

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
dc-leaf-4 lost power (both feeds).
Fault
Power loss
Right answer
A: Dispatch a technician to Bern: dc-leaf-4 hardware/power fault
Solved by
Nobody: SLA breached

dc-leaf-4 lost power at 13:15:59. Devices only reachable through it went silent as well: srv-db. Their logs never reached the collector.

Causal chain

  1. dc-leaf-4 lost power (both feeds).
  2. dc-leaf-4 went dark; its neighbours logged their links and adjacencies to it as down.
  3. Everything only reachable through it became unreachable from srv-mon: srv-db. They were healthy, but their metrics stopped and their logs never arrived.
  4. What was noticed: 2 devices stopped answering and Intranet failed for some users.

Logs that never reached srv-mon: dc-leaf-4, srv-db

Why the other options were wrong

  • Roll back the last configuration change on dc-leaf-4 Wrong: it assumed a configuration change on dc-leaf-4 broke it. dc-leaf-4 went dark, but not for that reason. The cause: dc-leaf-4 lost power (both feeds).
  • Dispatch a technician to Bern: srv-db hardware/power fault Wrong: it assumed srv-db had a hardware or power fault. srv-db only looked down: it sits behind the failure, so pings, scrapes and its logs could not get through. It was healthy throughout. The cause: dc-leaf-4 lost power (both feeds).
  • Dispatch a technician to Bern: dc-leaf-3 hardware/power fault Wrong: it assumed dc-leaf-3 had a hardware or power fault. dc-leaf-3 is the redundancy partner of dc-leaf-4: it carried the traffic and was healthy. The cause: dc-leaf-4 lost power (both feeds).

Timeline

  1. 13:16:00 INC-1003 opened: Bern: Intranet unavailable, devices not answering (Luca Brunner (Service Desk))
  2. 13:46:00 SLA breached; fault repaired

Inbox

7