Zurich 14:59:48
UTC 13:59:48
Open incidents P10 P20 P32 P40 Devices down 1/40 2 degr. Active alerts …
Syslog / s 0.0
MTTR today 11m 19s
live
Live
14:59:38 sw-zrh-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to down 14:59:26 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up 14:59:25 acc-brn-3 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down 14:59:05 sw-gva-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to up 14:58:29 acc-brn-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down 14:58:00 inet-rtr-brn-2 %BGP-3-NOTIFICATION: sent to neighbor 198.51.100.1 4/0 (hold time expired) 0 bytes 14:57:29 acc-brn-1 %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up 14:57:00 INC-1009 opened: Bern: WiFi slow in one area (P3) 14:53:00 INC-1008 opened: Lugano: network alarm, no user impact so far (P3) 14:49:01 INC-1006 expired: SLA breached 14:43:00 INC-1007 expired: SLA breached 14:25:00 INC-1007 opened: Bern: network almost unusable (P2)

Incidents

The shared ticket inbox of the NOC shift.

Sign in to play

P3 INC-1001 · Medium · reported by Samuel Rey (Monitoring)

Metrics and logs missing from some routers

Resolved

  1. Samuel Rey (Monitoring) Report 13:11:35

    Since 13:11 we get no SNMP metrics and no syslog from some of our routers. The scrape alerts say the targets are down, but ping still answers and users don't complain. Can the NOC take a look?

  2. hannes Attempt correct 13:13:36

    A: Revert the last policy change on fw-brn-1 (policy 12, MGMT-to-NOC)

  3. Samuel Rey (Monitoring) Reporter reply 13:13:36

    Policy 12 had been set to deny by mistake. Back to accept, the metrics are back. Thanks! - Samuel

Post-mortem

Root cause
jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
Fault
Firewall policy blocks SNMP/syslog
Right answer
A: Revert the last policy change on fw-brn-1 (policy 12, MGMT-to-NOC)
Solved by
hannes in 2m 01s

At 13:11:34 jdoe edited firewall policy 12 (MGMT-to-NOC) on fw-brn-1 and changed its action to deny. SNMP and syslog from the devices behind the firewall (internet edge routers) stopped reaching srv-mon while ping and user traffic kept working.

Causal chain

  1. jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
  2. What was noticed: metrics and logs vanished while ping and users were fine.

Logs that never reached srv-mon: inet-rtr-brn-1, inet-rtr-brn-2

Why the other options were wrong

  • Dispatch a technician to Bern: fw-brn-1 hardware/power fault Wrong: it assumed fw-brn-1 had a hardware or power fault. fw-brn-1 was fine in that respect. The cause: jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
  • No action: planned maintenance per CHG-20470 (LTE SIM swap on rtr-lug-1) Wrong: it assumed the planned maintenance CHG-20470 explained it. CHG-20470 is cancelled (01.11. 07:00 UTC) and concerns rtr-lug-1: no maintenance was running at the time. The cause: jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.
  • Reboot dc-spine-1 remotely: it showed a CPU spike at the same time Wrong: it assumed a reboot of dc-spine-1 would clear it. dc-spine-1's CPU spike was routine background noise (a scheduled job), unrelated. The cause: jdoe changed firewall policy 12 on fw-brn-1 to deny: SNMP and syslog were dropped in transit, user traffic was fine.

Timeline

  1. 13:11:35 INC-1001 opened: Metrics and logs missing from some routers (Samuel Rey (Monitoring))
  2. 13:13:36 hannes chose A: correct (+292)
  3. 13:13:36 Resolved by hannes; fault repaired

Inbox

5