Zurich 14:35:37
UTC 13:35:37
Open incidents P10 P21 P30 P41 Devices down 0/40 10 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:34:35 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up 14:34:18 sw-zrh-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up 14:34:07 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to down 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4) 14:00:01 INC-1004 expired: SLA breached 13:47:01 INC-1005 opened: Bern: network alarm, no user impact so far (P3) 13:46:00 INC-1003 expired: SLA breached 13:38:26 Chaos monkey off (every ~300s, max 2) 13:38:25 Chaos monkey on (every ~300s, max 2) 13:36:00 INC-1004 opened: Many failed logins on the network devices (P2)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P3 INC-1005 · Medium · reported by Mira Kälin (Monitoring)

Bern: network alarm, no user impact so far

SLA breached

  1. Mira Kälin (Monitoring) Report 13:47:01

    At 13:47 the alerts for Bern fired: links went down and the traffic moved to the backup paths. Traffic still flows, nobody called yet. Can the NOC take a look?

  2. NOC-POC System 14:24:00

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
jdoe's configuration change on core-brn-2 administratively shut its uplink.
Fault
Bad config change (wrong uplink shut)
Right answer
D: Roll back the last configuration change on core-brn-2
Solved by
Nobody: SLA breached

At 13:47:00 jdoe changed the configuration of core-brn-2 and shut Hu1/0/51 (DOWNLINK dc-spine-1 Eth1/54) instead of an unused port. Traffic moved to the redundant uplink; the admin-down interface is the clue.

Causal chain

  1. jdoe's configuration change on core-brn-2 administratively shut its uplink.
  2. core-brn-2--dc-spine-1 stopped carrying traffic; the devices at both ends logged the loss (if they could reach srv-mon).
  3. Redundancy carried the traffic after convergence: nothing else became unreachable.
  4. What was noticed: monitoring alarms only; users kept working on the redundant path.

Why the other options were wrong

  • Replace the fan tray of core-brn-2 before it overheats Wrong: it assumed the fans of core-brn-2 had failed. Right place, wrong fault: core-brn-2 had no such problem. The cause: jdoe's configuration change on core-brn-2 administratively shut its uplink.
  • Replace the fan tray of core-brn-1 before it overheats Wrong: it assumed the fans of core-brn-1 had failed. core-brn-1 was healthy: no alarm, its sensors and power supplies were normal. The cause: jdoe's configuration change on core-brn-2 administratively shut its uplink.
  • Roll back the last configuration change on core-brn-1 Wrong: it assumed a configuration change on core-brn-1 broke it. core-brn-1 was healthy: no alarm, its sensors and power supplies were normal. The cause: jdoe's configuration change on core-brn-2 administratively shut its uplink.

Timeline

  1. 13:47:01 INC-1005 opened: Bern: network alarm, no user impact so far (Mira Kälin (Monitoring))
  2. 14:24:00 SLA breached; fault repaired

Inbox

7