Incidents
The shared ticket inbox of the NOC shift.
P2 INC-1004 · Medium · reported by Samuel Rey (Monitoring)
Many failed logins on the network devices
-
Samuel Rey (Monitoring) Report 13:36:00
While going through the logs at 13:36 we noticed a lot of failed SSH logins on routers, switches and the firewalls, every few seconds and still going on. Nobody from the network team is logging in. No alert fired. Can the NOC take a look?
-
NOC-POC System 14:00:01
SLA breached: nobody solved this incident in time. The fault was repaired.
Post-mortem
- Root cause
- A compromised workstation in Bern (10.10.10.57) tried default logins on the network devices over SSH; the failures were logged at warning level only.
- Fault
- SSH login brute force
- Right answer
- C: Block 10.10.10.57 at the firewall and lock the account admin
- Solved by
- Nobody: SLA breached
From 13:36:00 a compromised workstation in BRN (10.10.10.57) tried default credentials (admin, cisco, root, administrator, netadmin) over SSH on 24 network devices, one attempt every 2 s. Every attempt failed and was logged at warning level; no alert fired.
Causal chain
- A compromised workstation in Bern (10.10.10.57) tried default logins on the network devices over SSH; the failures were logged at warning level only.
- What was noticed: a stream of failed logins on the network devices, no alarm.
Why the other options were wrong
- Roll back the last configuration change on inet-rtr-brn-1 Wrong: it assumed a configuration change on inet-rtr-brn-1 broke it. inet-rtr-brn-1 was not changed; it only lost its path. The cause: A compromised workstation in Bern (10.10.10.57) tried default logins on the network devices over SSH; the failures were logged at warning level only.
- Block 10.10.20.40 at the firewall and lock the account neops Wrong: it assumed 10.10.20.40 was attacking the devices. 10.10.20.40 only logged in successfully as neops, as every day; the failed logins came from another address. The cause: A compromised workstation in Bern (10.10.10.57) tried default logins on the network devices over SSH; the failures were logged at warning level only.
- Block 10.10.20.15 at the firewall and lock the account jdoe Wrong: it assumed 10.10.20.15 was attacking the devices. 10.10.20.15 only logged in successfully as jdoe, as every day; the failed logins came from another address. The cause: A compromised workstation in Bern (10.10.10.57) tried default logins on the network devices over SSH; the failures were logged at warning level only.
Timeline
- 13:36:00 INC-1004 opened: Many failed logins on the network devices (Samuel Rey (Monitoring))
- 14:00:01 SLA breached; fault repaired