Monitoring and Alerting: Avoiding Alert Fatigue with Actionable Thresholds

Editorial Team · · 3 min read

Alert Fatigue Is a Real Operations Problem

I once inherited a monitoring setup that fired 200+ alerts per day. The on-call engineer had muted their PagerDuty notifications. That's not a people problem — it's a systems design failure.

When every alert demands immediate attention and most of them are false positives or non-actionable, humans do what humans do: they stop paying attention. The fix isn't more discipline. It's fewer, better alerts.

The Difference Between Monitoring and Alerting

Monitoring is passive — dashboards, graphs, and logs that you look at when you're investigating. It should be comprehensive. Record everything you might need.

Alerting is active — it interrupts someone. It should be minimal. Only fire when a human needs to act right now.

The mistake most teams make is treating them as the same system. They build monitoring dashboards, then add alerts to every graph. CPU above 80%? Alert. Memory above 70%? Alert. Disk above 60%? Alert. Response time above 500ms? Alert.

None of those thresholds mean anything in isolation. CPU at 85% might be perfectly fine if response times are normal and error rates are flat. You're not alerting on problems — you're alerting on symptoms that may or may not indicate problems.

SLOs and Error Budgets: Alert on What Matters

Service Level Objectives give you meaningful thresholds. Instead of "CPU is high," you alert on "we're burning through our error budget too fast."

A practical SLO framework:

  • Availability SLO: 99.9% of requests return a non-5xx response (allows ~43 minutes of downtime per month)
  • Latency SLO: 95% of requests complete in under 300ms, 99% in under 1 second
  • Correctness SLO: 99.99% of data mutations are eventually consistent within 5 seconds

Your error budget is the inverse of the SLO. With 99.9% availability, you've got a 0.1% error budget — roughly 43 minutes per month or 8.7 hours per year.

Alert when the burn rate threatens to exhaust the budget:

# Prometheus alerting rule: fast burn rate
- alert: HighErrorBurnRate
  expr: |
    (
      sum(rate(http_requests_total{code=~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
    ) > 14.4 * 0.001
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error rate will exhaust monthly budget in 1 hour at current rate"

The 14.4 multiplier means: at this burn rate, you'll consume your entire monthly error budget in roughly 1 hour. That's worth waking someone up. A 1x burn rate means you'll exhaust the budget exactly at month's end — worth a Slack notification during business hours, not a page.

Multi-Window Burn Rate Alerts

Google's SRE book recommends multi-window alerts to reduce false positives. Check both a short window (fast detection) and a long window (confirms it's not a transient spike):

- alert: HighErrorBurnRate
  expr: |
    (
      sum(rate(http_requests_total{code=~"5.."}[1h]))
      / sum(rate(http_requests_total[1h]))
    ) > 14.4 * 0.001
    AND
    (
      sum(rate(http_requests_total{code=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m]))
    ) > 14.4 * 0.001
  for: 2m
  labels:
    severity: critical

Both the 1-hour and 5-minute windows must exceed the threshold. This eliminates the single-spike-at-minute-zero false positive that plagues simple rate-based alerts.

Alert Severity Tiers

Not every alert is a page. Define clear tiers:

  • P1 / Critical: Page the on-call. Customer-facing impact right now. SLO burn rate > 14x. Examples: site down, data loss in progress, security breach.
  • P2 / Warning: Slack notification to the team channel. Degraded but functional. SLO burn rate 6-14x. Examples: elevated error rate on one endpoint, replica lag > 30s.
  • P3 / Info: Ticket created automatically. No immediate impact but needs attention this week. Examples: disk projected to fill in 7 days, certificate expiring in 14 days.

If more than 20% of your P1 alerts don't require immediate action, they should be downgraded. Review alert severity monthly based on actual response data.

Practical Alert Design Rules

Every alert should pass this checklist before going live:

  • Is it actionable? — Can the on-call engineer do something about it right now? If the answer is "wait and see," it's a dashboard metric, not an alert.
  • Does the runbook exist? — Every alert should link to a runbook with diagnosis steps and remediation actions. "Investigate high CPU" is not a runbook.
  • Is the threshold based on data? — "CPU above 80%" is arbitrary. Base thresholds on historical baselines — alert when the value is 3+ standard deviations from the trailing 4-week average.
  • Does it fire during normal operations? — If an alert fires regularly and gets acknowledged without action, delete it.

Monitoring the Monitoring

Your alerting pipeline is itself a system that can fail. Prometheus can't scrape. AlertManager's config has a syntax error. The PagerDuty integration token expired.

Dead man's switch alerts solve this: a heartbeat alert that fires continuously and is expected to always be active. If it stops firing, something is wrong with the alerting pipeline itself. PagerDuty and Opsgenie both support this pattern natively.

Run quarterly alert fire drills. Deliberately trigger an alert condition and verify it reaches the right person within the expected time. I've seen alerting setups that looked correct in configuration but had broken routing rules that sent everything to a deactivated user.

The goal isn't zero alerts — it's zero ignored alerts. Every notification should prompt either action or a conscious decision to improve the alert itself.