It is 3:00 AM. Your phone buzzes on the nightstand, jolting you awake. Adrenaline spikes as you fumble for the screen, expecting a data breach or a production outage. You squint at the notification: “High CPU usage on dev-environment-worker-node-4.” You sigh, knowing that this particular node always spikes during the nightly backup. It resolves itself in five minutes. You swipe the alert away and try to go back to sleep, but the damage is done.
The next time the phone buzzes at 3:00 AM, you might not check it quite as fast. By the third time, you might just mute the channel entirely.
This is the reality of “alert fatigue,” and it is the silent killer of effective security operations. When everything is urgent, nothing is urgent. For security teams, the goal isn’t just to detect threats; it is to design a signaling system that engineering teams respect and trust. If your on-call rotation feels like a punishment rather than a safeguard, your defenses are already compromised.
Here is how to dismantle the wall of noise and build an alerting strategy that engineers actually listen to.
The “Actionability” Litmus Test
The golden rule of on-call alerting is simple: If a human cannot do anything about it, a human should not be woken up for it.
Too often, security alerts are informational rather than actionable. A scanner might flag a vulnerability that has no fix available, or an intrusion detection system might log a failed login attempt from a known scanner. These are data points, not incidents.
To fix this, every alert configuration must pass the “Actionability Litmus Test.” Before enabling a notification, ask three questions:
- Does this require immediate intervention to prevent damage?
- Is there a clear, documented path to resolution (a runbook)?
- What happens if we wait until morning?
If the answer to the first question is “no,” it is not a P0 alert. It is a ticket. If the answer to the second question is “no,” you are setting your engineers up for failure.
According to the principles laid out in Google’s Site Reliability Engineering (SRE) books, paging a human should only happen when a service level objective (SLO) is threatened. Security teams should adopt this mindset. Alert on the symptom (e.g., “Data is leaving the network to a suspicious IP”) rather than the cause (e.g., “Firewall rule 403 triggers”). Causes are for debugging; symptoms are for alerting.
Separating Signal from Noise with the Right Tools
Modern infrastructure generates terrabytes of logs. Trying to manually sift through this stream is impossible, and piping raw logs into Slack or PagerDuty is a recipe for disaster. You need an intermediary layer—a decision engine that ingests data and outputs decisions.
This is where choosing the right security monitoring tools becomes critical. The best tools today don’t just forward alerts; they correlate them. They understand that 50 failed login attempts followed by a successful root login and a sudden change in IAM permissions isn’t three separate alerts—it is one narrative of an attack.
Effective tooling should allow you to:
- Deduplicate: Group similar alerts into a single notification so an engineer isn’t bombarded with 100 pings for one issue.
- Suppress: Automatically silence alerts during known maintenance windows or deployments.
- Route: Send infrastructure alerts to the SRE team and application vulnerabilities to the product team, ensuring the right eyes see the right problem.
The Hierarchy of Urgency
Not all security signals are created equal. Treating a low-risk policy violation with the same urgency as an active exfiltration attempt dilutes the importance of the real threats. You need a tiered hierarchy.
- P1 (Critical / Wake Up): Imminent or active threat to customer data or production availability. Example: “Ransomware detected,” “Public S3 bucket access detected on sensitive data,” “Root account login without MFA.”
- P2 (High / Next Business Day): Serious issue, but not currently being exploited or causing damage. Example: “Critical CVE found in production image,” “Suspicious new admin user created (pending verification).”
- P3 (Info / Weekly Review): Compliance drifts or housekeeping items. Example: “MFA not enabled on non-prod user,” “Key rotation due in 7 days.”
By rigorously enforcing these tiers, you make a promise to your engineers: If we page you, it matters. This restores trust. When the pager goes off, they know it’s not a drill.
Continuous Tuning: The “Alert Review” Ritual
An alerting system is not a “set it and forget it” mechanism. It is a living organism that needs pruning.
Introduce a weekly or bi-weekly “Alert Review” meeting. Look at every alert that fired during the previous on-call shift. Analyze the “false positive” rate. If a specific rule triggered ten times and every time the engineer marked it as “Safe” or “No Action Needed,” that rule is broken.
You have two options:
- Tune it: Adjust the threshold. Maybe 80% CPU isn’t a security risk, but 99% is.
- Delete it: If it provides no value, kill it.
Research from Atlassian regarding incident management highlights that alert fatigue leads to longer response times and higher turnover. Engineers burn out when they feel helpless against a barrage of noise. By actively deleting useless alerts, you demonstrate that you value their time and sanity.
Conclusion: Empathy as a Security Strategy
Designing on-call signals isn’t just technical work; it is cultural work. It requires empathy for the human being on the other end of the pager.
When you design alerts that work—signals that are rare, actionable, and rich with context—you transform your security team from a source of annoyance into a source of protection. Engineers stop muting the channel. They start engaging with the data. And ultimately, that engagement is the only thing that keeps your organization secure when the real threat arrives at 3:00 AM.
