Alert fatigue: how it develops and the rules that cut noise, from actionable pages to weekly alert reviews

Alert fatigue is what happens when responders receive so many alerts that do not need action that they start ignoring, muting or slowly acknowledging all of them, including the ones that matter. You fix it by paging only for actionable problems, separating severities by delivery channel, collapsing duplicates into one notification, routing each alert to one owner and reviewing every alert after each on-call week. This guide covers how the noise builds up and the concrete rules that remove it.

What is alert fatigue and how does it develop?

Nobody designs a noisy alerting setup. It grows one reasonable decision at a time. After an outage, someone adds an alert so it never happens unnoticed again. A new service gets monitors with thresholds copied from an older one. A disk warning fires at 80 percent because that number felt safe two years ago.

Each alert makes sense on the day it is created. The problem is that alerts are added after incidents and almost never removed after quiet weeks. Over months, the ratio shifts: a growing share of notifications describe things that are slightly unusual but not broken.

Several patterns accelerate the drift:

• Cascades: one database outage triggers twenty API checks, three latency alerts and a queue backlog warning, all at once.
• Flapping: a check hovers around its threshold and sends down, up, down, up within minutes.
• Broadcast channels: every alert goes to one shared channel where everyone sees it and nobody owns it.
• Same urgency for everything: a certificate expiring in 25 days arrives the same way as a checkout returning 500 errors.

The human result is predictable. People learn that most alerts resolve themselves, so they wait five minutes before looking. That habit is harmless for noise and expensive for the one real outage hidden in it.

Signs your team already has alert fatigue

You rarely get a clear moment where fatigue starts. Look for these behaviors instead:

• Alerts get acknowledged and then auto-resolve without anyone doing anything.
• The alerts channel is muted by most of the team, sometimes by the on-call person.
• People describe specific monitors as "known noisy" and explain new colleagues which ones to ignore.
• Real incidents are first reported by customers or support, even though an alert fired earlier.
• On-call handovers include phrases like "it was the usual stuff".

If two or more of these sound familiar, the alerting system has stopped being a reliable signal and needs pruning, not more alerts.

Rule 1: page only for actionable alerts

The most effective single rule: an alert that interrupts a human must describe a problem that is affecting users, or will soon, and that the recipient can act on right now. Everything else belongs in a ticket, a dashboard or a digest.

Before an alert is allowed to page, it should pass three questions:

1. Is a user affected now, or will one be soon if nobody acts?
2. Can the person receiving it do something concrete, today, with the access they have?
3. Does it need to happen now, or would tomorrow morning be fine?

Some typical conversions:

• CPU at 90 percent on one node: not a page by itself. Page on the symptom it causes, such as rising response times or failing health checks.
• Disk at 80 percent: a ticket. Disk projected to be full within a few hours at the current write rate: a page.
• SSL certificate expiring in 21 days: a ticket for the owner. Certificate expiring in 2 days with renewal failing: a page.
• A cron job that missed one run of an hourly sync: chat notification. The same job missing runs for half a day while invoices depend on it: a page.

Alerting on symptoms that users notice, such as failed HTTP checks, error rates and latency, instead of every internal cause keeps the paging set small and meaningful. Causes still matter, but they are what the responder looks at after being paged.

Rule 2: define severity levels with different delivery

Severity only helps if it changes how an alert is delivered. Three levels are enough for most small teams. Write them down so nobody has to guess when creating a new monitor:

critical
  meaning:   users affected now, or will be within the hour
  delivery:  push/SMS/phone to the on-call person, any hour
  escalate:  to secondary after 10 min without acknowledgment

warning
  meaning:   degradation or risk, a human should look today
  delivery:  team chat channel, working hours only
  escalate:  none, reviewed at standup if still open

info
  meaning:   context, no action required
  delivery:  dashboard or daily digest, never a notification

The 10-minute escalation is an example; pick a value that matches how quickly your service needs a response. What matters is that critical is rare. If critical alerts fire several times per week and most turn out to be minor, the definition is too loose, not the team too slow.

A useful test: if you would not accept being woken at 3 a.m. for this alert, it is not critical. Downgrade it.

Rule 3: deduplicate and group related alerts

One incident should produce one notification thread, not dozens. Deduplication works on several levels:

• Per monitor: notify on state change only. A check that stays down sends one alert, an optional reminder at a fixed interval, and one recovery message. It does not resend every check cycle.
• Per dependency: if the database is down, the twenty endpoints behind it are a consequence. Alert on the database check and suppress or group the dependent ones.
• Against flapping: require a check to fail several consecutive times, or from several locations, before changing state. Require a few consecutive successes before declaring recovery.

Flapping and single-location failures overlap heavily with false positives. If those are your main source of noise, the guide on reducing false positive alerts with multi-region checks, retries and realistic timeouts goes deeper into the tuning. Note the difference: a false positive is an alert that is wrong. Alert fatigue also comes from alerts that are technically correct but useless, which is why the actionability rule matters as much as threshold tuning.

Rule 4: route each alert to the right person

An alert sent to everyone is owned by no one. Every monitor needs an owner, either a team or a role, and every critical alert needs exactly one person who is expected to respond first.

• Assign each monitor to a service owner. The payments team gets payment alerts, not the frontend team.
• Send critical alerts to whoever is on call according to a schedule, not to a shared channel.
• Respect off-hours: warnings outside working hours wait until morning.
• Escalate when nobody acknowledges, so a missed phone does not mean a missed outage.

A shared channel can still receive a copy for visibility, but it should never be the only path for something urgent. For how to split responsibility in a team of two to eight people, including overrides and what should page at night, see on-call rotation for small teams.

How to reduce alert fatigue with a weekly alert review

Rules decay unless someone enforces them. The cheapest enforcement is a short review at the end of every on-call week, done by the person handing over, ideally with the next person on call present. Thirty minutes is usually enough.

1. Export or list every alert that fired during the week, including those resolved automatically.
2. Classify each one: actionable (someone did something useful), real but not urgent, false positive or duplicate.
3. Decide one change per non-actionable alert: delete it, downgrade the severity, tune the threshold or duration, merge it into a dependency group, or open a ticket to fix the underlying cause.
4. Check the night pages separately. Every alert that woke someone and was not critical gets fixed before the next week starts.
5. Write the changes into the handover notes so the next responder knows what changed.

A simple log keeps the review honest:

Week 2024-W18, on call: alex

alert                        fired  class         action
api-latency-p95 > 800ms      14     not urgent    warning, working hours
db-primary down              1      actionable    keep
worker-queue backlog         9      duplicate     group under db-primary
ssl expiry < 30d (staging)   7      not urgent    ticket to owner, info
checkout 5xx rate            2      actionable    keep

night pages: 6 (4 not critical, fixed)

Track two numbers over time: total pages per week and pages outside working hours. You do not need a target from anyone else; you need the trend to go down while real incidents are still caught. If a real incident was missed, the review also asks which alert should have fired and adds it, so pruning does not become blind deletion.

Treat deleting an alert as a normal outcome. A monitor that has never led to action in months is documentation of a past worry, not protection.

Applying these rules in m0nitor

If you use m0nitor, the rules above map onto a few features. Alert workflows follow an if-this-then-that pattern on monitor events: a workflow can notify one or more channels, open an incident or call a webhook. That lets you send critical monitors to a paging channel and route warnings to a team chat, instead of sending every event everywhere.

Available channels include email, Slack, Discord, Microsoft Teams, Mattermost, Telegram, Pushover, SMS, PagerDuty and ilert. On-call schedules support rotations, overrides and do-not-disturb hours, so pages reach the person on duty rather than the whole team. The same separation works with any tool that lets you route by severity and schedule.

FAQ

What causes alert fatigue?

Alert fatigue is caused by a high share of alerts that do not require action: thresholds that are too sensitive, alerts on causes instead of user-facing symptoms, duplicate alerts from one incident, flapping checks and alerts broadcast to people who do not own the problem. Alerts are added after incidents but rarely removed, so noise accumulates over time.

How many alerts per on-call shift are too many?

There is no universal number. A practical test is whether the on-call person can investigate every page properly and still get their normal work done. If pages are routinely acknowledged without action, or if night pages are anything other than rare, the volume is too high for your team regardless of the count.

Is alert fatigue the same as false positives?

No. False positives are alerts that report a problem that does not exist, and they are one cause of alert fatigue. Correct alerts can cause fatigue too, when they describe real but unimportant conditions, arrive at the wrong time or reach the wrong person. Fixing both requires threshold tuning and stricter rules for what is allowed to page.

Should warnings ever wake someone up at night?

No. If a condition is serious enough to wake someone, it is critical by definition and should be labeled that way. Warnings should go to a chat channel or ticket during working hours. If a warning keeps turning into an outage overnight, promote that specific case to critical with a tighter condition instead of paging on all warnings.