Zurich 14:56:36
UTC 13:56:36
Open incidents P10 P20 P31 P40 Devices down 1/40 1 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:54:38 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to down 14:54:38 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up 14:54:30 rtr-zrh-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 14:54:21 sw-zrh-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to down 14:53:42 sw-bsl-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 14:53:18 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to up 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P3 INC-1005 · Medium · reported by Mira Kälin (Monitoring)

Bern: network alarm, no user impact so far

SLA breached

  1. Mira Kälin (Monitoring) Report 13:47:01

    At 13:47 the alerts for Bern fired: links went down and the traffic moved to the backup paths. Traffic still flows, nobody called yet. Can the NOC take a look?

  2. NOC-POC System 14:24:00

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
jdoe's configuration change on core-brn-2 administratively shut its uplink.
Fault
Bad config change (wrong uplink shut)
Right answer
D: Roll back the last configuration change on core-brn-2
Solved by
Nobody: SLA breached

At 13:47:00 jdoe changed the configuration of core-brn-2 and shut Hu1/0/51 (DOWNLINK dc-spine-1 Eth1/54) instead of an unused port. Traffic moved to the redundant uplink; the admin-down interface is the clue.

Causal chain

  1. jdoe's configuration change on core-brn-2 administratively shut its uplink.
  2. core-brn-2--dc-spine-1 stopped carrying traffic; the devices at both ends logged the loss (if they could reach srv-mon).
  3. Redundancy carried the traffic after convergence: nothing else became unreachable.
  4. What was noticed: monitoring alarms only; users kept working on the redundant path.

Why the other options were wrong

  • Replace the fan tray of core-brn-2 before it overheats Wrong: it assumed the fans of core-brn-2 had failed. Right place, wrong fault: core-brn-2 had no such problem. The cause: jdoe's configuration change on core-brn-2 administratively shut its uplink.
  • Replace the fan tray of core-brn-1 before it overheats Wrong: it assumed the fans of core-brn-1 had failed. core-brn-1 was healthy: no alarm, its sensors and power supplies were normal. The cause: jdoe's configuration change on core-brn-2 administratively shut its uplink.
  • Roll back the last configuration change on core-brn-1 Wrong: it assumed a configuration change on core-brn-1 broke it. core-brn-1 was healthy: no alarm, its sensors and power supplies were normal. The cause: jdoe's configuration change on core-brn-2 administratively shut its uplink.

Timeline

  1. 13:47:01 INC-1005 opened: Bern: network alarm, no user impact so far (Mira Kälin (Monitoring))
  2. 14:24:00 SLA breached; fault repaired

Inbox

0
  • No incidents match the filters.