Definition
Retry or confirmation is the practice of re-checking an endpoint after a failed check, often from another vantage point, before declaring it down. It guards against false positives caused by momentary network blips, brief timeouts, or transient errors.
Rather than alerting on the very first failure, a monitor with confirmation requires the failure to persist across a retry (or several) — trading a few seconds of detection delay for far fewer false alarms.
Why It Matters
False alarms are corrosive. If your monitor cries wolf on every transient hiccup, your team starts ignoring alerts — and then misses the real outage. Confirmation keeps alerts trustworthy by ensuring they represent genuine, sustained problems, which is the whole point of monitoring.
How It Works
When a check fails, the monitor immediately re-checks (sometimes from a different location). If the retry also fails, the endpoint is confirmed down and alerts fire; if the retry succeeds, the blip is ignored. The same logic helps confirm recovery quickly once the endpoint is back.
Real-World Example
A brief network glitch makes one check time out. With confirmation, the monitor immediately re-checks; the endpoint responds normally, so no alert fires. Later, a real outage causes the check and its retry to both fail, and only then does the team get alerted — a true positive.
Best Practices
- Confirm failures with a retry before alerting to cut false positives
- Keep the confirmation fast so detection delay stays small
- Re-check failing monitors frequently to catch recovery quickly
- Balance sensitivity (fast alerts) against noise (false alarms)
- Treat repeated confirmed failures as genuine incidents
Common Mistakes
- Alerting on a single failed check and generating noise
- Adding so many retries that real outages are detected late
- Ignoring alerts because past false positives eroded trust
- Not re-checking down monitors, so recovery is noticed slowly
- Treating every transient blip as a full incident
In Monitoristic
When a monitor fails, Monitoristic re-checks it every 60 seconds. That rapid re-checking helps confirm whether an outage is real and detects recovery quickly, rather than relying on a single check in either direction.