A customer wrote in last week with a fair complaint. One of their sites had been down for about three hours. relay19 had noticed within minutes and posted to Slack, exactly once. By the time anyone looked, that message was forty scrolls up in a busy channel, and the recovery notice had not arrived yet because there was nothing to recover from.
One message per incident was a deliberate choice. It was also the wrong one for the middle of a long outage, so we changed it.
What happens now
While a monitor stays down, relay19 re-sends a still-down notice on a fixed schedule measured from the moment the incident opened:
15m 30m 45m 1h 2h 4h 8h 24h then dailyFour reminders in the first hour, then one every doubling. Each one carries the same cause as the original alert plus how long the site has been down, so a reminder arriving cold in a channel still makes sense on its own.
The recovery notice is unchanged. The first passing check resolves the incident and stops the reminders in the same step.
Why the interval decays
A fixed interval gets the first hour wrong or the tenth hour wrong. Every fifteen minutes is right when the outage is fresh and people are arguing about whether it is real. By hour six it is noise, and noise gets muted, which is how a channel ends up ignoring the one alert that mattered.
A decaying schedule follows the shape of the outage instead. Early on, the reminders are dense because that is when attention is cheap and the question is still “is someone on this?”. Later, they thin out because the answer is known and what remains is a long wait for a fix. Daily after the first day is enough to make sure a forgotten monitor does not stay forgotten.
Why there is no setting
We looked at a per-monitor interval, a per-channel schedule, and a snooze button, and shipped none of them. The schedule above is the one we would recommend anyway, and a knob invites every team to pick a fixed interval and rediscover the problem in the paragraph above. If it turns out to be wrong for a real workload, we would rather change the schedule for everyone than hand out the option to be wrong individually.
One exception is built in. PagerDuty channels never receive reminders. That tool already holds the alert, escalates it on its own timetable, and pages the next person when the first one does not respond. Re-triggering it every fifteen minutes would only add duplicate noise to a rota that is already doing the job. Chat, email, Telegram, ntfy and webhook channels get the full schedule.
The small print
Reminders are per incident, and a multi-region monitor holds one incident per failing region. That matches how the original alerts already work: if the site is down from two vantages, you were told twice at the start, and you are reminded twice as well. Disabled monitors are skipped, and an incident that is resolved between a reminder being scheduled and being sent is left alone.
Webhook receivers see the new event as incident.reminder, with the same
body as incident.opened. If your integration switches on the event name
and drops anything it does not recognise, nothing changes for you until you
decide it should.
Nothing to configure, nothing to enable. The next long outage will just be harder to miss.