Blog
One MTTR number hides where the time goes. Split it into detection, acknowledgement, diagnosis and recovery, then cut each phase with alerts, routing, runbooks and rollbacks.
To reduce MTTR, stop treating it as one number. Split every incident into four phases: time to detect, time to acknowledge, time to diagnose and time to recover. Then attack the largest phase with its specific lever: fast confirmed alerts for detection, on-call routing for acknowledgement, runbooks for diagnosis and tested rollbacks for recovery.
This guide walks through each phase, shows the math and commands that matter, and ends with a simple way to measure whether your changes actually work.
MTTR usually means mean time to recovery (sometimes repair or resolve): the average time from the moment a service breaks until users are served normally again. The monitoring glossary lists it next to related terms like SLO and error budget, and the definitions matter, because teams that mix up "recovered" and "root cause fixed" end up comparing numbers that measure different things.
Pick one definition and write it down. For most small teams the useful one is: start of user impact until user impact ends. The permanent fix can come later and belongs in the postmortem, not in MTTR.
The problem with a single average is that it tells you nothing about where the time went. A 40-minute incident could be 25 minutes of nobody noticing and 15 minutes of fixing, or 2 minutes of noticing and 38 minutes of guessing. The fixes for those two incidents have nothing in common. Reducing MTTR is not one project; it is four separate problems, and you should only work on the one that is currently the biggest.
The four phases, with the timestamp that closes each one:
Step 2 is where most teams discover something surprising. It is common to assume diagnosis is the slow part and then find that the first alert arrived long after customers had already written to support.
Detection time is mostly arithmetic. The worst case is roughly:
worst-case detection = check interval
+ (failures required - 1) x retry interval
+ request timeoutTwo configurations make the difference concrete:
The second setup is not only faster, it is also less noisy, because confirmation from another location filters out a single flaky probe or network path. That is the trade-off to optimise: shorter intervals and fewer sequential retries, but confirmation across locations instead of over time. The details on retries, timeouts and split thresholds are in the guide on reducing false positive alerts with multi-region checks.
Three more detection gaps to close:
An alert that goes to a shared channel where ten people see it is often acknowledged by nobody. Acknowledgement time drops when exactly one person is responsible at any moment and knows it.
The levers here are organisational more than technical:
If you have not set this up yet, look at tooling for on-call schedules with overrides and do-not-disturb hours, and keep the escalation chain short. A practical test: pick a random time, fire a test alert, and measure how long acknowledgement takes. Do it once during the day and once in the evening.
Also check what reaches the pager. If on-call people get paged for things that do not need action, they start acknowledging slowly. Removing noise is an acknowledgement-time fix.
Diagnosis is the phase where experience matters most and where the person on call at 3 a.m. may not be the person who knows the system best. Two things close that gap: alerts that carry context, and runbooks that turn tribal knowledge into steps.
A good alert answers the questions the responder would otherwise look up:
"api.example.com: HTTP 502 from 3 of 3 regions since 14:02 UTC, last deploy 13:55" points almost directly at the action. "Monitor DOWN" does not.
A runbook should fit on one screen and start with the most common cause. Here is a template:
Runbook: API health check failing
Owner: platform on-call
1. Confirm scope
curl -sS -o /dev/null -w "%{http_code} %{time_total}\n" https://api.example.com/health
Failing from all regions? -> our side. One region? -> network/CDN.
2. Recent change?
Deploy in the last 60 min -> roll back first (step 4), investigate later.
3. Dependencies
Database: check connection count and replication lag
Upstream: check proxy error log for "upstream timed out" / "connection refused"
4. Mitigate
kubectl rollout undo deployment/api
kubectl rollout status deployment/api
5. Still failing after 15 min -> escalate to backend lead, post status update.The rule in step 2 is the most valuable line in most runbooks: if something changed recently, undo it before debugging. Every postmortem should end with either a new runbook or a correction to an existing one; the blameless incident postmortem template has a section for exactly that kind of follow-up action.
Recovery is fast when the action that restores service is boring, rehearsed and takes one command. It is slow when it requires writing a fix under pressure.
Separate mitigation from repair. Mitigation restores service; repair fixes the cause. MTTR only needs mitigation. Common mitigations, roughly from fastest to slowest:
kubectl rollout undo, redeploying the previous image tag, or reverting a commit and letting CI deploy).Make rollbacks safe in advance:
latest.Finally, verify recovery with the same checks that detected the failure. The incident is over when external monitoring is green from all regions, not when the deploy pipeline reports success.
You need the five timestamps per incident, stored somewhere queryable. A spreadsheet works; so does a table:
SELECT
percentile_cont(0.5) WITHIN GROUP (ORDER BY detected_at - started_at) AS median_detect,
percentile_cont(0.5) WITHIN GROUP (ORDER BY acked_at - detected_at) AS median_ack,
percentile_cont(0.5) WITHIN GROUP (ORDER BY mitigation_at - acked_at) AS median_diagnose,
percentile_cont(0.5) WITHIN GROUP (ORDER BY recovered_at - mitigation_at) AS median_recover
FROM incidents
WHERE started_at >= now() - interval '90 days';A few rules keep the numbers honest:
started_at from monitoring data or logs, not from when someone noticed. Otherwise detection time is always zero.Any setup that gives you multi-region checks, on-call routing and timestamped incident records can support this process. In m0nitor, monitors check from distributed probes in several regions, on-call schedules route alerts to the current responder, and incidents keep a timeline of updates that you can use as the source for your phase timestamps. Alert workflows can react to a monitor event by notifying a channel, creating an incident or calling a webhook, for example to attach deploy information or open a ticket automatically.
There is no universal target. Work backwards from your SLO: if your monthly error budget allows around 43 minutes of downtime (99.9%), a single incident with a 40-minute MTTR consumes almost all of it. Your target is whatever lets typical incidents fit inside the budget with room to spare.
MTTD (mean time to detect) and MTTA (mean time to acknowledge) are the first two phases of MTTR. Tracking them separately shows whether slow recovery comes from noticing late, responding late or fixing slowly.
It reduces detection time, which is part of MTTR. Going from 5-minute to 1-minute checks can save several minutes per incident, but only if alerts are confirmed quickly and someone acknowledges them. If diagnosis takes an hour, shorter intervals barely move the total.
For customer impact, end it when service is restored. Track the permanent fix separately, as a follow-up action from the postmortem. Mixing the two makes MTTR punish teams for doing careful root-cause work after mitigation.