Zurich 15:17:38
UTC 14:17:38
Open incidents P10 P20 P32 P40 Devices down 1/40 2 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
15:17:09 acc-brn-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to up 15:16:52 core-brn-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 15:15:38 rtr-lug-1 %CELLWAN-2-NC_EVENT: Cellular0/2/0: Network change event, Radio Access Technology: LTE, Band: 3, RSRP: -97 dBm 15:14:12 acc-brn-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up 14:57:00 INC-1009 opened: Bern: WiFi slow in one area (P3) 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4) 14:00:01 INC-1004 expired: SLA breached

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P2 INC-1007 · Hard · reported by Luca Brunner (Service Desk)

Bern: network almost unusable

SLA breached

  1. Luca Brunner (Service Desk) Report 14:25:00

    From about 14:25 everything in the Bern campus crawls: logins time out, printers vanish. The data centre services seem fine from outside. My first guess would be the application servers. Forwarded from the service desk, users keep calling. Can you have a look?

  2. NOC-POC System 14:43:00

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
Two ports of acc-brn-2 were bridged by a desk switch with BPDU guard off: a layer-2 loop and broadcast storm.
Fault
STP loop / broadcast storm
Right answer
C: Shut the looped access ports on acc-brn-2 and enable BPDU guard
Solved by
Nobody: SLA breached

At 14:24:59 a loop was created on acc-brn-2 (two user ports bridged by a desk switch, BPDU guard off). A broadcast storm saturated the campus links, the distribution switches' CPU went to 99 % and campus devices saw heavy loss.

Causal chain

  1. Two ports of acc-brn-2 were bridged by a desk switch with BPDU guard off: a layer-2 loop and broadcast storm.
  2. What was noticed: everything became extremely slow.

Why the other options were wrong

  • No action: planned maintenance per CHG-20457 (Swisscom fibre maintenance on the Bern access ring (provider notice)) Wrong: it assumed the planned maintenance CHG-20457 explained it. CHG-20457 is scheduled (08.11. 00:00 UTC) and concerns inet-rtr-brn-1 and isp-swisscom: no maintenance was running at the time. The cause: Two ports of acc-brn-2 were bridged by a desk switch with BPDU guard off: a layer-2 loop and broadcast storm.
  • Open a ticket with Swisscom for the BGP session on inet-rtr-brn-1 Real but harmless: Swisscom's BGP router towards inet-rtr-brn-1 failed (provider side); the link itself stayed up. Redundancy covered it and it did not cause what Bern reported; fix it in a normal change.
  • Reload acc-brn-2 and upgrade it to the fixed release (memory leak bug) Wrong: it assumed acc-brn-2 suffered from a software bug. acc-brn-2 was fine in that respect. The cause: Two ports of acc-brn-2 were bridged by a desk switch with BPDU guard off: a layer-2 loop and broadcast storm.

Timeline

  1. 14:25:00 INC-1007 opened: Bern: network almost unusable (Luca Brunner (Service Desk))
  2. 14:43:00 SLA breached; fault repaired

Inbox

1