Zurich 15:29:06
UTC 14:29:06
Open incidents P10 P20 P31 P40 Devices down 1/40 1 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
15:28:58 acc-brn-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 15:27:38 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down 15:27:31 sw-zrh-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to up 15:27:15 acc-brn-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up 15:26:58 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down 15:26:50 sw-zrh-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down 15:26:39 acc-brn-2 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up 15:24:00 INC-1009 expired: SLA breached 14:57:00 INC-1009 opened: Bern: WiFi slow in one area (P3) 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P3 INC-1009 · Hard · reported by Luca Brunner (Service Desk)

Bern: WiFi slow in one area

SLA breached

  1. Luca Brunner (Service Desk) Report 14:57:00

    Some users in Bern complain that WiFi is unusable in their area since around 14:57, worse during the busy hours. Cable users don't notice anything. My first guess would be the application servers. Forwarded from the service desk, users keep calling. Can you have a look?

  2. NOC-POC System 15:24:00

    SLA breached: nobody solved this incident in time. The fault was repaired.

Post-mortem

Root cause
acc-brn-2 Gi1/0/24 ran at half duplex against a full-duplex peer: late collisions and load-dependent loss.
Fault
Duplex mismatch
Right answer
D: Set acc-brn-2 Gi1/0/24 back to auto-negotiation (duplex mismatch)
Solved by
Nobody: SLA breached

A duplex mismatch appeared on acc-brn-2 Gi1/0/24 at 14:56:59 (port forced to half duplex after a configuration change, peer at full duplex): errors on the half-duplex side and load-dependent packet loss, no outage.

Causal chain

  1. acc-brn-2 Gi1/0/24 ran at half duplex against a full-duplex peer: late collisions and load-dependent loss.
  2. What was noticed: slow, lossy connections in one area that got worse with load.

Why the other options were wrong

  • Set acc-brn-2 Gi1/1/2 back to auto-negotiation (duplex mismatch) Wrong: it assumed acc-brn-2 had a duplex mismatch. Right place, wrong fault: acc-brn-2 had no such problem. The cause: acc-brn-2 Gi1/0/24 ran at half duplex against a full-duplex peer: late collisions and load-dependent loss.
  • Send the cabling team: fibre cut between acc-brn-2 and dist-brn-2 Wrong: it assumed the cable acc-brn-2--dist-brn-2 was cut. acc-brn-2 Gi1/1/2 (between acc-brn-2 and dist-brn-2) was clean: normal light levels, no errors, no flaps. The cause: acc-brn-2 Gi1/0/24 ran at half duplex against a full-duplex peer: late collisions and load-dependent loss.
  • Send the cabling team: fibre cut between ap-brn-2 and acc-brn-2 Wrong: it assumed the cable ap-brn-2--acc-brn-2 was cut. acc-brn-2 Gi1/0/24 (between ap-brn-2 and acc-brn-2) was clean: normal light levels, no errors, no flaps. The cause: acc-brn-2 Gi1/0/24 ran at half duplex against a full-duplex peer: late collisions and load-dependent loss.

Timeline

  1. 14:57:00 INC-1009 opened: Bern: WiFi slow in one area (Luca Brunner (Service Desk))
  2. 15:24:00 SLA breached; fault repaired

Inbox

4