Blog
False alarms train your team to ignore pages. Here is how to cut them with multi-region confirmation, retries, timeouts based on real latency and split thresholds.
If you want to know how to reduce false positive alerts, start with three changes: confirm every failure from more than one region, retry before you notify anyone, and set timeouts based on your service's measured latency instead of a default. Then split each check into a warning threshold that goes to chat and a critical threshold that pages a human. Most noisy monitoring setups improve sharply with these four changes alone.
This guide goes through the causes of false alarms one by one, shows a concrete configuration you can adapt, and explains how to check afterwards that the noise is actually gone.
A false positive is an alert for a problem that users never experienced. One of them is a minor annoyance. A steady trickle changes behaviour: people mute channels, acknowledge pages without looking, and start assuming the next alert is also nothing. That is alert fatigue, and it is how a real outage ends up sitting unacknowledged for twenty minutes.
The goal is not fewer alerts at any cost, but alerts where every single one is worth waking someone up for. Every technique below trades a few seconds of detection time for a large gain in trust. That trade is almost always worth it, as long as you know how many seconds you are giving up.
Before tuning anything, find out which kind of false alarm you have. The fixes differ.
A check that runs from one location tests your service and the entire network path to it. A congested peering link, a DNS resolver hiccup at the probe's provider or a short routing change will look exactly like your site being down. Nothing was wrong with your service, but the alert fired anyway.
A quick way to tell the difference during an alert is to test the URL from several locations at once, for example with a free network check that probes your site from locations worldwide. If only one region fails, you are looking at a path problem, not an outage.
Many monitoring setups use a timeout of a few seconds by default. That works for a static page behind a CDN. It does not work for an API endpoint that does a database query, a cold-started serverless function or a page that is slow on the first request after a deploy. The check times out, the service was fine, and someone gets paged.
Flapping is a check that switches between up and down repeatedly within a short window. Typical causes are a response time hovering right at the threshold, one unhealthy instance behind a load balancer that receives every third request, or a health endpoint that depends on a slow downstream service. Each state change produces a notification, so one borderline condition turns into a dozen alerts.
Keyword checks on content that changes (a banner, a localized string, an A/B test variant) fail when the page is perfectly healthy. Health endpoints that report "down" when an optional dependency is slow cause the same problem. If an alert regularly fires without user impact, question what the check actually asserts.
Work through these steps in order. Each one removes a category of noise, and later steps are easier to tune once the earlier ones are in place.
Here is an illustrative configuration that combines the first five steps. The syntax is generic, so map it to whatever tool you use:
check: https://api.example.com/health
interval: 60s
timeout: 10s
regions: [eu-west, us-east, ap-southeast]
confirm: down in at least 2 of 3 regions
retries: 2, 15s apart
recover_after: 2 consecutive successes
warning: response_time over 1500ms for 3 checks -> chat channel
critical: confirmed down -> page on-callMulti-region confirmation is the single most effective fix for network-related noise. Instead of trusting one probe, the monitor checks from several regions and changes state only when enough of them agree. The monitoring glossary defines this as multi-region consensus.
Practical rules that hold up well:
If your users are concentrated in one area, weight your regions accordingly. A failure from the region that serves most of your traffic deserves more attention than one from a region with no customers.
Retries catch the blips that multi-region checks miss, such as a single dropped connection or a request that hit a restarting container. The idea is simple: when a check fails, run it again a few seconds later before changing state.
Retries add detection delay, so calculate it instead of guessing. With a 60-second interval and two retries 15 seconds apart:
For most SaaS services, 90 seconds is fine. If you have committed to a tight response target, shorten the interval rather than dropping retries. A 30-second interval with one retry after 10 seconds gives a worst case of about 40 seconds and still filters single-request failures.
Avoid long retry chains. Five retries spread over several minutes mostly hide real outages that happen to be intermittent.
A timeout should express "this is too slow to count as working", not "this is slower than usual". Slowness belongs in a warning threshold, which comes next.
Measure what your endpoint actually does. Run this a few dozen times, ideally from more than one machine and at different times of day:
for i in $(seq 1 30); do
curl -o /dev/null -s \
-w '%{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total}\n' \
https://api.example.com/health
sleep 2
doneThe columns show DNS lookup, TCP connect, time to first byte and total time in seconds. Look at the slowest results, not the average. Then set the timeout well above the worst value you saw, with room for slow moments that are still acceptable to users. As a starting point, many teams use around 10 seconds for HTML pages and API endpoints and only go lower when they have measured a stable, fast service.
Two details that cause surprise timeouts:
time_namelookup is occasionally high, the problem is resolution, not your app. Check your DNS provider and TTLs before blaming the service.A single threshold forces a bad choice: set it low and you get paged for slowness that users barely notice, set it high and you miss degradation until it becomes an outage. Two thresholds solve this.
Examples that work as starting points:
Response time warning: over 1500ms for 3 consecutive checks
critical: timeout or confirmed down
Disk usage warning: over 80% critical: over 95%
SSL expiry warning: 21 days critical: 7 days
Error rate warning: over 1% for 5 minutes
critical: over 5% for 5 minutesNote the "for 3 consecutive checks" and "for 5 minutes" conditions. A threshold without a duration flaps on every spike. Requiring the condition to hold for a period is the cheapest flapping protection there is.
Routing is where this pays off. With alert workflows that react differently to each monitor event, a warning can post to a Slack or Teams channel while a critical event pages the on-call person and opens an incident.
Even with durations on thresholds, a value that hovers around a boundary can keep toggling. Two techniques fix this.
Hysteresis means using different thresholds for entering and leaving a state. Alert when response time exceeds 1500 ms for three checks, but clear the alert only when it drops below 1000 ms for three checks. The gap between the two values absorbs the noise.
Recovery confirmation applies the same idea to up/down checks. Require two or three consecutive successful checks before marking the monitor as up. Otherwise a service that fails every other request sends "down", "up", "down" notifications all afternoon.
If a check still changes state many times within a short window, for example more than four times in 30 minutes, treat flapping as its own condition. Send one notification saying the check is unstable and suppress the individual state changes until it settles. One message about instability is useful. Twelve up/down messages are not.
Tuning without measuring tends to drift back to noisy defaults. For two weeks after the changes, label every alert that fires:
For each noise alert, write down which cause from the list above it matches: single region, timeout, flapping or wrong assertion. Patterns show up quickly, usually one or two checks produce most of the noise. Fix those, then repeat the review once a quarter or after major infrastructure changes.
Also look in the other direction. If an incident was reported by a customer before any alert fired, your tuning went too far for that check. Tighten it again.
If you use m0nitor, the building blocks above map to existing features. Monitors check websites, APIs and servers on a fixed interval from distributed probes in several regions. Workflows can notify a channel, create an incident or call a webhook depending on the event, which is how you separate warning and critical routing. Alerts can go to email, Slack, Discord, Microsoft Teams, Mattermost, Telegram, Pushover, SMS, PagerDuty and ilert, and on-call schedules with overrides and do-not-disturb hours make sure critical pages reach the person actually on duty. The same principles apply with any other tool.
One or two retries at a short interval, typically 10 to 15 seconds apart, filter most single-request failures. More retries mainly delay detection of real outages. Calculate the worst-case delay as interval plus retry time and make sure it fits your response targets.
At least three, so that a majority decision is possible. Alert when two out of three agree. With only two regions, a disagreement leaves you guessing, and requiring every region to fail hides regional outages.
Usually not. Slowness belongs in a warning threshold routed to a chat channel. Page only when the service times out or is confirmed down, or when slowness is severe and sustained enough that users clearly cannot work.
Yes, mostly through jobs whose run time varies. Give heartbeat monitors a grace period that covers the longest normal run, as described in our guide on heartbeat monitoring for cron jobs, instead of alerting the moment the expected ping is late.