This lesson's interactive features are locked, please to keep using them

Actionable Alerts

The previous lesson used a noisy 401 alert to prove the alerting pipeline works.

Real alerts should be actionable. If a rule fires, an operator should know there's likely a system issue worth investigating.

For example, in most web apps a spike in 5xx responses (there is likely an issue in our service) is much scarier than a spike in 4xx responses (usually an issue in the client's request).

Avoid Alert Fatigue

An alert that fires constantly gets ignored. Avoiding "alert fatigue" by choosing signals that reflect real incidents is a core part of a good alerting system. A practical first cut is:

alert on 500 spikes
do not alert on normal user mistakes (like bad credentials)

Assignment

Update your alert rule so it focuses on actionable server failures (500 responses), and rename it to "500 Error Spike":
```
sum(rate(http_requests_total{status="500"}[5m])) * 60
```
Keep the threshold at > 3.
Spam 25 "frodo"/"badPassword" 401 requests in Linko.
Click "Run queries" to confirm the series is not firing from the 401 requests.
Spam 25 "saruman"/"malformedPassword" 500 requests in Linko.
Click "Run queries" again to confirm the series is firing from the 500 requests.
Save the alert rule, and stay on the Alert page.

If Grafana hasn't updated yet after the 500 requests, wait about 15 seconds and click "Run queries" again.

Run and submit the CLI tests from the root of the Linko repo.