Urgent alerts need a concrete action

An urgent notification should interrupt because a useful action is needed now. If all warnings have the same priority, the on-call person must still do the real classification. Google's SRE book describes how frequent interruptions can lead people to skim or ignore relevant alerts too.

10 messages are not 10 incidents

In a constructed case, 4 instances report the same project-list failure. There are also 6 CPU warnings that need no immediate intervention at that particular moment. The calculation 4 + 6 = 10 counts messages, not independent incidents. The team therefore designs this distribution:

This distribution is an editorial example, not a measured operating result. The CPU warnings remain under observation. Their investigation simply follows a different notification route.

Fewer interruptions need preserved context

Alertmanager supports grouping and routing to configured recipients. Similar alerts become a shared notification, while the affected instances must remain identifiable. A shared name alone is insufficient when different regions or owners need independent treatment.

Fewer urgent messages demonstrate neither a lower error rate nor faster recovery. Hiding warnings or losing a notification also produces fewer messages for the on-call person. Quality must therefore also be checked against missed problems and a functioning notification path.

When an early resource signal is urgent

The recommendation does not mean always waiting for visible user failures. Prometheus names capacity constraints as a reason for human intervention before an outage occurs. An imminent critical storage shortage or missing required redundancy, for example, can make timely action urgent.

The assessment changes when delay makes substantial harm likely and the team being contacted can act effectively now. Then an early resource signal belongs on the urgent route too. What matters is the concrete response expected and whether it can happen in time; the category CPU, application or infrastructure alone does not decide that.

Sources

• Google SRE — Monitoring Distributed Systems

• Prometheus — Alerting best practices

• Prometheus — Alertmanager