Alerting

We'll start with a deliberately noisy alert to prove the full flow works:

• query a metric
• evaluate a threshold
• see the rule transition from Normal to Pending to Firing

Create a noisy alert that fires when the HTTP 401 response rate exceeds 3 requests/minute (calculated over the last 5 minutes).

1. Ensure your Linko app, Prometheus, and Grafana are running.
2. In Grafana, go to Alerting -> Alert rules -> New alert rule.
3. Create a new rule called "401 Login Failure Spike" using this query (average number of 401's per minute for the last 5 minutes):

   sum(rate(http_requests_total{status="401"}[5m])) * 60
   

4. Configure the condition to fire when the value is above 3.
5. Try to log in with "frodo"/"wrongPassword" at least 25 times on the Linko app, triggering 401 responses.
6. Click "Run queries" in Grafana. You should see "Series 1" transition to a Firing state.
7. Add the Alert to a new folder called "Linko Alerts"
8. Add the Alert to a new evaluation group called "Linko Group".
   • Set the "Evaluation interval" (how often the rule checks the condition) to 10 seconds
   • Set the "pending period" (the duration the rule has to be in a bad state before it transitions to "firing") to None
   • The "keep firing for" (the duration the rule will stay in "firing" before it goes back to "normal") should also be None
9. Set the "Contact point" for the notifications to "empty" (we won't actually send notifications, but in the real world this is where you'd set up Email, Slack, PagerDuty, etc.)
10. Save the alert rule, and stay on the Alert page.
11. Make another burst of failed login attempts, then wait about 15 seconds and refresh the alert rule page. You should see it transition to Firing again.

Run and submit the CLI tests from the root of the Linko repo.