Alerts in Grafana: From Data to Action

Alert rules · Contact points · Routing · Alert design strategy

Learning Objectives

• ✓ Configure Grafana alert rules using the Unified Alerting framework
• ✓ Define meaningful alert thresholds based on business context — not technical defaults
• ✓ Route alerts to notification channels (email or Slack webhook)
• ✓ Explain the alert state machine: Normal → Pending → Firing → Resolved
• ✓ Avoid common alerting mistakes: fatigue, missing context, wrong audience

Session Timeline

Time	Segment	Activity
0–10 min	Recap	L15 dashboards — what are you watching and why?
10–25 min	Why Alerts Matter	The "sleeping in front of a dashboard" problem
25–40 min	Grafana Alert Architecture	Rules, contact points, notification policies, state machine
40–60 min	Lab Part 1	Create alert rules from the transaction-risk dashboard
60–70 min	Lab Part 2	Configure email / Slack contact point + routing
70–80 min	Alert Design Framework	Thresholds, routing, escalation, fatigue
80–88 min	Case Studies	Zepto & HDFC Securities production alerting
88–90 min	Module 4 Wrap-Up	Course arc + Final Project preview

Section 1: Why Alerts Matter

A dashboard is only useful if someone is looking at it. In a 24-hour streaming system, no one watches 24 hours a day. Alerts let the data come to you — instead of you going to the data.

Without alerts:
  10:47 PM  Flagged transaction rate crosses the operating threshold
  10:47 PM  Nobody sees it
  11:30 PM  Risk manager checks Grafana manually — too late

With alerts:
  10:47 PM  Grafana alert fires → Slack message in #fraud-ops
  10:48 PM  Manager opens the review queue and assigns an analyst

Tool	Purpose	Acts On
Logging	Record everything that happened	Engineers debugging after an event
Monitoring	Show current state continuously	Teams watching dashboards
Alerting	Notify when action is needed	Decision-makers who need to respond

Section 2: Grafana Alert Architecture

Three Components

1. Alert Rules — Conditions evaluated on a schedule. Each rule has a query and fires when the condition is met for a defined duration.
2. Contact Points — Where notifications go. Supported: Email, Slack, PagerDuty, Opsgenie, Teams, Webhook, Telegram, and 20+ more.
3. Notification Policies — Routing rules: which alert labels go to which contact point.

The Alert State Machine

The "For" Duration — Why It Matters

Without a "For" duration, a single bad data point triggers an alert. With For: 2m, the condition must be true for 2 continuous minutes before the alert fires.

Without "For":
  Check 1: Fraud rate = 3.8% → FIRES immediately
  Check 2: Fraud rate = 3.4% → Resolves — short-lived spike

With For: 2m:
  Check 1: Fraud rate = 3.8% → enters PENDING state
  2 min later: still above 3.5% → FIRES — sustained risk, worth investigating.

Section 3: Lab Part 1 — Create Alert Rules

Alert 1: "High Flagged Transaction Rate"

Navigate: Left sidebar → Alerting (bell icon) → Alert Rules → New alert rule

Alert 2: "High-Value Flagged Transaction"

1. Create a query that counts MySQL rows where is_fraud = 'true' and amount > 50000.
2. Condition: IS ABOVE 0 | Evaluate every: 1m | For: 1m.
3. Labels: severity=critical, team=fraud-ops.

SELECT COUNT(*) AS value
FROM transactions
WHERE is_fraud = 'true'
  AND amount > 50000;

Section 4: Lab Part 2 — Contact Points

Option A: Email

Requires SMTP configuration in Docker. Add environment variables to the Grafana run command:

-e GF_SMTP_ENABLED=true
-e GF_SMTP_HOST=smtp.gmail.com:587
-e GF_SMTP_USER=your@gmail.com
-e GF_SMTP_PASSWORD=your-app-password

Then: Alerting → Contact points → New → Type: Email → Add your address → Test

Option B: Slack Webhook (Recommended for demos)

1. Go to api.slack.com/apps → Create New App → Incoming Webhooks → Activate
2. Add New Webhook to Workspace → select channel #grafana-alerts
3. Copy webhook URL: https://hooks.slack.com/services/T.../B.../...
4. Grafana: Alerting → Contact points → New → Type: Slack → Paste URL → Test

Notification Routing by Label

Alerting → Notification policies → Edit root policy → Add matchers:

• severity = critical → Slack
• severity = warning → Email
• Default (catch-all) → Email

Expected Slack Alert Message

FIRING: Transaction Risk — High Fraud Rate
Severity: warning | Team: fraud-ops
Flagged transaction rate exceeds threshold
Current value: 3.8%
Starts at: 11:47:00
Dashboard: http://localhost:3000/d/...

Section 5: Alert Design Framework

Three Questions for Every Alert Rule

1. Who receives this alert and what will they do? If you can't answer in one sentence, the alert is not ready.
2. What is the cost of a false positive? Each false positive erodes trust. After 10, people ignore alerts.
3. What is the cost of a false negative? A missed alert on a financial system can mean regulatory exposure or monetary loss.

Notification Cost	Example	Threshold Approach
Low (Slack message, easy to dismiss)	Single high-value flagged case	Sensitive threshold may be acceptable
Medium (email + calendar)	Fraud rate above operating threshold	Threshold needs care, use "For: 2m"
High (PagerDuty, on-call woken up)	System outage, fraud spike	Very precise threshold, long "For" duration

Avoiding Alert Fatigue

Alert fatigue = when so many alerts fire that operators start ignoring them — including the ones that matter.

• Set For: 2m or longer on non-critical rules — filters out single-tick blips
• Delete any alert you can't write a runbook for
• Use severity labels and separate channels (warnings vs critical)
• Review and tune thresholds monthly — conditions change as data patterns change

Section 6: Case Studies

Case A: Zepto — Dark Store Operations

Zepto operates ~350 dark stores. Their alerting rules:

• order_acceptance_rate < 90% for 3 minutes → Slack to store ops manager
• avg_delivery_time > 35 min for 5 minutes → Escalation to city head
• inventory_stockout_count > 20 SKUs → Automated reorder + alert

Discussion: Why is the "For" duration 3 minutes for order acceptance but 5 minutes for delivery time? What is different about the business consequence of each?

Case B: HDFC Securities — Trading Platform

• price_feed_latency > 500ms → PagerDuty (on-call engineer, 24/7)
• order_rejection_rate > 2% → Critical Slack + SMS to CTO
• active_users_drop > 30% in 5 min → Infrastructure alert (possible outage)

Discussion: The order rejection rate alert goes to the CTO, not an engineer. Why? What does this tell you about how alerting functions as a business communication tool, not just a technical one?

Module 4 Summary

Module 1 (L1-L5):    UNDERSTAND — What is streaming? How does Kafka work?
Module 2 (L6-L10):   BUILD — Producer, consumer, persistence, alert routing
Module 3 (L11-L13):  SCALE — Windowing, Spark, Flink — enterprise patterns
Module 4 (L14-L16):  VISUALISE — Grafana dashboards + alert routing
Module 5 (L17-L18):  DECIDE — Multi-source pricing, explainability, governance
Module 6 (L19-L20):  PRESENT — Final project and architecture review

Lecture 16 Extension: See the Full Pipeline Live

The alert rules above evaluate data already stored in MySQL. The extension reconnects every course component so students can watch the system operate: the producer publishes, two Kafka consumers persist transactions and alert events, and Grafana refreshes from both tables every five seconds.

Key Takeaways