Blog

How to reduce MTTR: split it into detection, acknowledgement, diagnosis and recovery, then fix each phase

One MTTR number hides where the time goes. Split it into detection, acknowledgement, diagnosis and recovery, then cut each phase with alerts, routing, runbooks and rollbacks.

To reduce MTTR, stop treating it as one number. Split every incident into four phases: time to detect, time to acknowledge, time to diagnose and time to recover. Then attack the largest phase with its specific lever: fast confirmed alerts for detection, on-call routing for acknowledgement, runbooks for diagnosis and tested rollbacks for recovery.

This guide walks through each phase, shows the math and commands that matter, and ends with a simple way to measure whether your changes actually work.

What MTTR measures and why one number hides the problem

MTTR usually means mean time to recovery (sometimes repair or resolve): the average time from the moment a service breaks until users are served normally again. The monitoring glossary lists it next to related terms like SLO and error budget, and the definitions matter, because teams that mix up "recovered" and "root cause fixed" end up comparing numbers that measure different things.

Pick one definition and write it down. For most small teams the useful one is: start of user impact until user impact ends. The permanent fix can come later and belongs in the postmortem, not in MTTR.

The problem with a single average is that it tells you nothing about where the time went. A 40-minute incident could be 25 minutes of nobody noticing and 15 minutes of fixing, or 2 minutes of noticing and 38 minutes of guessing. The fixes for those two incidents have nothing in common. Reducing MTTR is not one project; it is four separate problems, and you should only work on the one that is currently the biggest.

The four phases, with the timestamp that closes each one:

  • Detection: impact starts until the first alert fires.
  • Acknowledgement: alert fires until a human says "I have it".
  • Diagnosis: acknowledged until someone knows what action will restore service.
  • Recovery: action chosen until users see normal behaviour again.

How to reduce MTTR step by step

  1. Agree on the MTTR definition and the five timestamps: started, detected, acknowledged, mitigation chosen, recovered.
  2. Reconstruct those timestamps for your last five to ten incidents from alert logs, chat history and deploy logs.
  3. Compute the duration of each phase per incident and find the phase that dominates.
  4. Apply the lever for that phase (sections below) and change one thing at a time.
  5. Record the timestamps for every new incident and compare the phase durations after a quarter.

Step 2 is where most teams discover something surprising. It is common to assume diagnosis is the slow part and then find that the first alert arrived long after customers had already written to support.

Phase 1: reduce detection time with fast, confirmed alerts

Detection time is mostly arithmetic. The worst case is roughly:

worst-case detection = check interval
                     + (failures required - 1) x retry interval
                     + request timeout

Two configurations make the difference concrete:

  • 5-minute interval, alert after 3 consecutive failures, 30 s timeout: up to about 15.5 minutes before anyone is notified.
  • 1-minute interval, failure confirmed by an immediate recheck from a second region, 10 s timeout: around 1 to 2 minutes.

The second setup is not only faster, it is also less noisy, because confirmation from another location filters out a single flaky probe or network path. That is the trade-off to optimise: shorter intervals and fewer sequential retries, but confirmation across locations instead of over time. The details on retries, timeouts and split thresholds are in the guide on reducing false positive alerts with multi-region checks.

Three more detection gaps to close:

  • Check what users use, not just the homepage. A login endpoint or API health route with a body check catches failures that a cached marketing page hides.
  • Watch jobs that have no URL. Cron jobs, queue workers and backups fail silently. A heartbeat that alerts when an expected ping is missing turns "we noticed three days later" into minutes.
  • Alert on degradation, not only on down. A response-time threshold (for example, alert when p95 latency stays above 2 s for 5 minutes) often fires before the hard failure.

Phase 2: shorten acknowledgement with on-call routing

An alert that goes to a shared channel where ten people see it is often acknowledged by nobody. Acknowledgement time drops when exactly one person is responsible at any moment and knows it.

The levers here are organisational more than technical:

  • Route critical alerts to the current on-call person through a channel that wakes them up (push, SMS or phone), not only to a chat channel.
  • Require an explicit acknowledgement, so the system knows whether a human is on it.
  • Escalate automatically if nobody acknowledges within a fixed time. Five minutes for critical alerts is a reasonable starting point; adjust it to your SLO.
  • Keep the schedule current. Overrides for holidays and sick days matter more than the rotation model itself.

If you have not set this up yet, look at tooling for on-call schedules with overrides and do-not-disturb hours, and keep the escalation chain short. A practical test: pick a random time, fire a test alert, and measure how long acknowledgement takes. Do it once during the day and once in the evening.

Also check what reaches the pager. If on-call people get paged for things that do not need action, they start acknowledging slowly. Removing noise is an acknowledgement-time fix.

Phase 3: cut diagnosis time with better alerts and runbooks

Diagnosis is the phase where experience matters most and where the person on call at 3 a.m. may not be the person who knows the system best. Two things close that gap: alerts that carry context, and runbooks that turn tribal knowledge into steps.

Put the first five minutes of investigation into the alert

A good alert answers the questions the responder would otherwise look up:

  • What failed: monitor name, URL, check type.
  • How it failed: status code, timeout, TLS error, body mismatch.
  • Where: which regions see the failure, and whether all of them do.
  • Since when: first failure timestamp.
  • What changed: last deploy or config change, if you can attach it.
  • Where to start: a link to the runbook for this monitor.

"api.example.com: HTTP 502 from 3 of 3 regions since 14:02 UTC, last deploy 13:55" points almost directly at the action. "Monitor DOWN" does not.

Write short runbooks per alert, not a wiki

A runbook should fit on one screen and start with the most common cause. Here is a template:

Runbook: API health check failing
Owner: platform on-call

1. Confirm scope
   curl -sS -o /dev/null -w "%{http_code} %{time_total}\n" https://api.example.com/health
   Failing from all regions? -> our side. One region? -> network/CDN.

2. Recent change?
   Deploy in the last 60 min -> roll back first (step 4), investigate later.

3. Dependencies
   Database:  check connection count and replication lag
   Upstream:  check proxy error log for "upstream timed out" / "connection refused"

4. Mitigate
   kubectl rollout undo deployment/api
   kubectl rollout status deployment/api

5. Still failing after 15 min -> escalate to backend lead, post status update.

The rule in step 2 is the most valuable line in most runbooks: if something changed recently, undo it before debugging. Every postmortem should end with either a new runbook or a correction to an existing one; the blameless incident postmortem template has a section for exactly that kind of follow-up action.

Phase 4: speed up recovery with tested rollbacks

Recovery is fast when the action that restores service is boring, rehearsed and takes one command. It is slow when it requires writing a fix under pressure.

Separate mitigation from repair. Mitigation restores service; repair fixes the cause. MTTR only needs mitigation. Common mitigations, roughly from fastest to slowest:

  • Turn off a feature flag.
  • Roll back to the previous release (kubectl rollout undo, redeploying the previous image tag, or reverting a commit and letting CI deploy).
  • Fail over to a replica or secondary region.
  • Scale out or restart a stuck process.
  • Block abusive traffic at the edge.

Make rollbacks safe in advance:

  • Database migrations must be backward compatible. Use expand and contract: add columns first, deploy code that uses them, remove old columns in a later release. Otherwise a rollback breaks on schema mismatch.
  • Keep the previous artifact available and pinned by tag, not by latest.
  • Practise the rollback on a normal day. A rollback that has never been run is a guess.

Finally, verify recovery with the same checks that detected the failure. The incident is over when external monitoring is green from all regions, not when the deploy pipeline reports success.

How to measure MTTR progress

You need the five timestamps per incident, stored somewhere queryable. A spreadsheet works; so does a table:

SELECT
  percentile_cont(0.5) WITHIN GROUP (ORDER BY detected_at - started_at)      AS median_detect,
  percentile_cont(0.5) WITHIN GROUP (ORDER BY acked_at - detected_at)        AS median_ack,
  percentile_cont(0.5) WITHIN GROUP (ORDER BY mitigation_at - acked_at)      AS median_diagnose,
  percentile_cont(0.5) WITHIN GROUP (ORDER BY recovered_at - mitigation_at)  AS median_recover
FROM incidents
WHERE started_at >= now() - interval '90 days';

A few rules keep the numbers honest:

  • Use the median, and look at the slowest incident separately. Small teams have few incidents, and one long outage distorts the mean.
  • Take started_at from monitoring data or logs, not from when someone noticed. Otherwise detection time is always zero.
  • Compare quarter over quarter per phase. A flat total MTTR can hide a real improvement in detection offset by one hard diagnosis.
  • Read the incidents, not just the chart. With five incidents a quarter, the story behind each one tells you more than the median.

Tooling for detection, routing and the incident timeline

Any setup that gives you multi-region checks, on-call routing and timestamped incident records can support this process. In m0nitor, monitors check from distributed probes in several regions, on-call schedules route alerts to the current responder, and incidents keep a timeline of updates that you can use as the source for your phase timestamps. Alert workflows can react to a monitor event by notifying a channel, creating an incident or calling a webhook, for example to attach deploy information or open a ticket automatically.

FAQ

What is a good MTTR?

There is no universal target. Work backwards from your SLO: if your monthly error budget allows around 43 minutes of downtime (99.9%), a single incident with a 40-minute MTTR consumes almost all of it. Your target is whatever lets typical incidents fit inside the budget with room to spare.

What is the difference between MTTR, MTTD and MTTA?

MTTD (mean time to detect) and MTTA (mean time to acknowledge) are the first two phases of MTTR. Tracking them separately shows whether slow recovery comes from noticing late, responding late or fixing slowly.

Does a shorter check interval reduce MTTR?

It reduces detection time, which is part of MTTR. Going from 5-minute to 1-minute checks can save several minutes per incident, but only if alerts are confirmed quickly and someone acknowledges them. If diagnosis takes an hour, shorter intervals barely move the total.

Should MTTR end when service is restored or when the root cause is fixed?

For customer impact, end it when service is restored. Track the permanent fix separately, as a follow-up action from the postmortem. Mixing the two makes MTTR punish teams for doing careful root-cause work after mitigation.

Product

  • Pricing
  • Features
  • Integrations
  • Security
  • Free network check

Features

  • Uptime monitoring
  • Status pages
  • Alert workflows
  • Incident management
  • Dependencies
  • AI assistant
  • Global infrastructure
  • On-call schedules
  • Enterprise SSO

Use cases

  • Websites for free
  • Cron jobs
  • SSL certificates
  • Databases
  • IP blacklists
  • GDPR compliance

Resources

  • What is a status page?
  • Documentation
  • API
  • MCP server
  • Monitoring glossary
  • Free tools
  • SLA calculator
  • SSL checker
  • DNS lookup
  • Blog
  • Changelog
  • About
  • Contact

Compare

  • Free uptime monitoring
  • UptimeRobot
  • Better Stack
  • Pingdom
  • StatusCake
  • Uptime Kuma
  • Hyperping
  • Atlassian Statuspage
  • Instatus
  • Cachet
  • Freshping