Zurich 14:40:56
UTC 13:40:56
Open incidents P10 P21 P30 P41 Devices down 0/40 10 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:40:08 rtr-lug-1 %CELLWAN-2-NC_EVENT: Cellular0/2/0: Network change event, Radio Access Technology: LTE, Band: 3, RSRP: -97 dBm 14:39:44 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down 14:38:12 inet-rtr-brn-1 %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 14:25:00 INC-1007 opened: Bern: network almost unusable (P2) 14:24:00 INC-1005 expired: SLA breached 14:01:00 INC-1006 opened: Bern: hardware alarm, no user impact (P4) 14:00:01 INC-1004 expired: SLA breached 13:47:01 INC-1005 opened: Bern: network alarm, no user impact so far (P3) 13:46:00 INC-1003 expired: SLA breached 13:38:26 Chaos monkey off (every ~300s, max 2) 13:38:25 Chaos monkey on (every ~300s, max 2) 13:36:00 INC-1004 opened: Many failed logins on the network devices (P2)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P3 INC-1001 · Medium · reported by Samuel Rey (Monitoring)

Metrics and logs missing from some routers

Resolved

  1. Samuel Rey (Monitoring) Report 13:11:35

    Since 13:11 we get no SNMP metrics and no syslog from some of our routers. The scrape alerts say the targets are down, but ping still answers and users don't complain. Can the NOC take a look?

  2. hannes Attempt correct 13:13:36

    A: Revert the last policy change on fw-brn-1 (policy 12, MGMT-to-NOC)

  3. Samuel Rey (Monitoring) Reporter reply 13:13:36

    Policy 12 had been set to deny by mistake. Back to accept, the metrics are back. Thanks! - Samuel

Post-mortem

Root cause
jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
Fault
Firewall policy blocks SNMP/syslog
Right answer
A: Revert the last policy change on fw-brn-1 (policy 12, MGMT-to-NOC)
Solved by
hannes in 2m 01s

At 13:11:34 jdoe edited firewall policy 12 (MGMT-to-NOC) on fw-brn-1 and changed its action to deny. SNMP and syslog from the devices behind the firewall (internet edge routers) stopped reaching srv-mon while ping and user traffic kept working.

Causal chain

  1. jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
  2. What was noticed: metrics and logs vanished while ping and users were fine.

Logs that never reached srv-mon: inet-rtr-brn-1, inet-rtr-brn-2

Why the other options were wrong

  • Dispatch a technician to Bern: fw-brn-1 hardware/power fault Wrong: it assumed fw-brn-1 had a hardware or power fault. fw-brn-1 was fine in that respect. The cause: jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
  • No action: planned maintenance per CHG-20470 (LTE SIM swap on rtr-lug-1) Wrong: it assumed the planned maintenance CHG-20470 explained it. CHG-20470 is cancelled (01.11. 07:00 UTC) and concerns rtr-lug-1: no maintenance was running at the time. The cause: jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
  • Reboot dc-spine-1 remotely: it showed a CPU spike at the same time Wrong: it assumed a reboot of dc-spine-1 would clear it. dc-spine-1's CPU spike was routine background noise (a scheduled job), unrelated. The cause: jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.

Timeline

  1. 13:11:35 INC-1001 opened: Metrics and logs missing from some routers (Samuel Rey (Monitoring))
  2. 13:13:36 hannes chose A: correct (+292)
  3. 13:13:36 Resolved by hannes; fault repaired

Inbox

2