Incidents
The shared ticket inbox of the NOC shift.
P2 INC-1003 · Easy · reported by Luca Brunner (Service Desk)
Bern: Intranet unavailable, devices not answering
SLA breached
-
Luca Brunner (Service Desk) Report 13:16:00
Several users in Bern can't use Intranet since around 13:16. It looks like only some areas are affected. Forwarded from the service desk, users keep calling. Can you have a look?
-
NOC-POC System 13:46:00
SLA breached: nobody solved this incident in time. The fault was repaired.
Post-mortem
- Root cause
- dc-leaf-4 lost power (both feeds).
- Fault
- Power loss
- Right answer
- A: Dispatch a technician to Bern: dc-leaf-4 hardware/power fault
- Solved by
- Nobody: SLA breached
dc-leaf-4 lost power at 13:15:59. Devices only reachable through it went silent as well: srv-db. Their logs never reached the collector.
Causal chain
- dc-leaf-4 lost power (both feeds).
- dc-leaf-4 went dark; its neighbours logged their links and adjacencies to it as down.
- Everything only reachable through it became unreachable from srv-mon: srv-db. They were healthy, but their metrics stopped and their logs never arrived.
- What was noticed: 2 devices stopped answering and Intranet failed for some users.
Logs that never reached srv-mon: dc-leaf-4, srv-db
Why the other options were wrong
- Roll back the last configuration change on dc-leaf-4 Wrong: it assumed a configuration change on dc-leaf-4 broke it. dc-leaf-4 went dark, but not for that reason. The cause: dc-leaf-4 lost power (both feeds).
- Dispatch a technician to Bern: srv-db hardware/power fault Wrong: it assumed srv-db had a hardware or power fault. srv-db only looked down: it sits behind the failure, so pings, scrapes and its logs could not get through. It was healthy throughout. The cause: dc-leaf-4 lost power (both feeds).
- Dispatch a technician to Bern: dc-leaf-3 hardware/power fault Wrong: it assumed dc-leaf-3 had a hardware or power fault. dc-leaf-3 is the redundancy partner of dc-leaf-4: it carried the traffic and was healthy. The cause: dc-leaf-4 lost power (both feeds).
Timeline
- 13:16:00 INC-1003 opened: Bern: Intranet unavailable, devices not answering (Luca Brunner (Service Desk))
- 13:46:00 SLA breached; fault repaired