Zurich 15:40:12
UTC 14:40:12
Open incidents P10 P20 P30 P40 Devices down 0/40 Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
15:39:24 acc-brn-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to up 15:38:41 sw-bsl-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to up 15:38:28 sw-bsl-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up 15:38:00 rtr-lug-1 %LINK-3-UPDOWN: Interface GigabitEthernet0/1/0, changed state to up 15:37:48 acc-brn-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 15:37:20 acc-brn-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to down 15:38:00 INC-1008 expired: SLA breached 15:24:00 INC-1009 expired: SLA breached 14:57:00 INC-1009 opened: Bern: WiFi slow in one area (P3) 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P3 INC-1008 · Easy · reported by Mira Kälin (Monitoring)

Lugano: network alarm, no user impact so far

SLA breached

  1. Mira Kälin (Monitoring) Report 14:53:00

    Since 14:53 we see alarms in Lugano: a link alarm on sw-lug-1. Users don't complain so far, but please check before we lose the redundancy for good. Can the NOC take a look?

  2. NOC-POC System 15:38:00

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
jdoe's configuration change on sw-lug-1 administratively shut its uplink.
Fault
Bad config change (wrong uplink shut)
Right answer
D: Roll back the last configuration change on sw-lug-1
Solved by
Nobody: SLA breached

At 14:52:59 jdoe changed the configuration of sw-lug-1 and shut Gi1/1/1 (UPLINK rtr-lug-1 Gi0/1/0) instead of an unused port. The change cut off sw-lug-1.

Causal chain

  1. jdoe's configuration change on sw-lug-1 administratively shut its uplink.
  2. rtr-lug-1--sw-lug-1 stopped carrying traffic; the devices at both ends logged the loss (if they could reach srv-mon).
  3. Redundancy carried the traffic after convergence: nothing else became unreachable.
  4. What was noticed: monitoring alarms only; users kept working on the redundant path.

Why the other options were wrong

  • Reboot dist-brn-1 remotely: it showed a CPU spike at the same time Wrong: it assumed a reboot of dist-brn-1 would clear it. dist-brn-1's CPU spike was routine background noise (a scheduled job), unrelated. The cause: jdoe's configuration change on sw-lug-1 administratively shut its uplink.
  • Open a ticket with the MPLS provider for the Lugano access circuit Wrong: it assumed the MPLS provider had an outage. the MPLS provider had no incident: their circuit was clean and all their counters green. The cause: jdoe's configuration change on sw-lug-1 administratively shut its uplink.
  • Reload sw-lug-1 and upgrade it to the fixed release (memory leak bug) Wrong: it assumed sw-lug-1 suffered from a software bug. sw-lug-1 was fine in that respect. The cause: jdoe's configuration change on sw-lug-1 administratively shut its uplink.

Timeline

  1. 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (Mira Kälin (Monitoring))
  2. 15:38:00 SLA breached; fault repaired

Inbox

0
  • No incidents match the filters.