Zurich 14:55:46
UTC 13:55:46
Open incidents P10 P20 P31 P40 Devices down 1/40 1 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:54:38 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to down 14:54:38 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up 14:54:30 rtr-zrh-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 14:54:21 sw-zrh-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to down 14:53:42 sw-bsl-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 14:53:18 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to up 14:53:00 rtr-lug-1 %LINK-3-UPDOWN: Interface GigabitEthernet0/1/0, changed state to down 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P4 INC-1006 · Medium · reported by Samuel Rey (Monitoring)

Bern: hardware alarm, no user impact

SLA breached

  1. Samuel Rey (Monitoring) Report 14:01:00

    One of the Bern devices raised an environment alarm at 14:01. No impact visible so far. Can the NOC take a look?

  2. NOC-POC System 14:49:01

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
A power supply of core-brn-2 failed; the second one carried the load.
Fault
PSU failure
Right answer
D: Replace the failed power supply of core-brn-2
Solved by
Nobody: SLA breached

PS2 of core-brn-2 failed at 14:01:00; the remaining supply carries the load.

Causal chain

  1. A power supply of core-brn-2 failed; the second one carried the load.
  2. What was noticed: a hardware alarm while the device kept running.

Why the other options were wrong

  • Replace the failed power supply of core-brn-1 Wrong: it assumed a power supply of core-brn-1 had failed. core-brn-1 was healthy: no alarm, its sensors and power supplies were normal. The cause: A power supply of core-brn-2 failed; the second one carried the load.
  • Roll back the last configuration change on core-brn-1 Wrong: it assumed a configuration change on core-brn-1 broke it. core-brn-1 was healthy: no alarm, its sensors and power supplies were normal. The cause: A power supply of core-brn-2 failed; the second one carried the load.
  • Roll back the last configuration change on core-brn-2 Wrong: it assumed a configuration change on core-brn-2 broke it. Right place, wrong fault: core-brn-2 had no such problem. The cause: A power supply of core-brn-2 failed; the second one carried the load.

Timeline

  1. 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (Samuel Rey (Monitoring))
  2. 14:49:01 SLA breached; fault repaired

Inbox

5