Write alert messages with evidence and a response path

A useful alert message shows what is wrong with an application and which initial response path is planned. Its recipient should understand it without having to guess the service from an unfamiliar server name.

From the user task to the signal

Start with an important task, such as loading a project list in a customer portal. For interactive services, Prometheus recommends user-visible errors and suitable views for investigating causes, among other signals. A utilisation value can provide context; it does not automatically explain whether the project list works.

Define which requests and responses enter the measurement. Repeated calls are additional requests, not automatically additional affected people. An error rate therefore needs a clear denominator.

A worked alert example

The following software team, measurements and rules are fictional. This is not an ASES incident, a general threshold or a support commitment. All recorded requests for the named operation form the denominator here. The chosen rule evaluates a 5-minute window; it requires no additional holding period.

Here both the minimum request count and the error threshold have been exceeded. The 20 % describes requests within this window. It proves neither a corresponding share of affected users nor a particular cause.

Prepare the initial response

The runbook is a short guide to the intended diagnostic path. First, the responsible person acknowledges ownership and checks whether the problem is ongoing. Acknowledgement does not mean the fault is fixed. The view's preselected filters must carry the same service, operation and time information.

Next, narrow down affected instances, versions and dependencies. A release at 09:08 would be an event to investigate in this example, not proof of causation. A rollback requires a suitable, previously checked path; the alert does not authorise arbitrary restarts or changes.

Check the notification path too

Alertmanager can group similar alerts and route them to configured recipients. Affected instances must remain visible. Independent problems must not collapse into one merely because their names are similar.

Use a clearly marked test notification to check reception, acknowledgement and the planned backup path too. Prometheus explicitly calls for monitoring the monitoring system itself. Later normal measurements may end the alert state; they replace neither checking the user task nor further cause analysis. Business-level errors returned as success or slow responses are not covered by this simple server-error rule alone.

Sources

• Google SRE — Monitoring Distributed Systems

• Prometheus — Alerting best practices

• Prometheus — Alertmanager

• GitLab — Incident Diagnosis in a Symptom-based World