Module 4 · Lecture 16

Alerts in Grafana: From Data to Action

Alert rules · Contact points · Routing · Alert design strategy

⏱ 90 minutes 📍 Prerequisites: Lecture 15 (dashboards built) 🔧 Lab: Alert rules + Slack/email routing

Learning Objectives
Session Timeline
Why Alerts Matter
Grafana Alert Architecture
Lab Part 1 — Create Alert Rules
Lab Part 2 — Configure Contact Points
Alert Design Framework
Case Studies
Module 4 Summary
Key Takeaways

Learning Objectives

✓ Configure Grafana alert rules using the Unified Alerting framework
✓ Define meaningful alert thresholds based on business context — not technical defaults
✓ Route alerts to notification channels (email or Slack webhook)
✓ Explain the alert state machine: Normal → Pending → Firing → Resolved
✓ Avoid common alerting mistakes: fatigue, missing context, wrong audience

Session Timeline

Time	Segment	Activity
0–10 min	Recap	L15 dashboards — what are you watching and why?
10–25 min	Why Alerts Matter	The "sleeping in front of a dashboard" problem
25–40 min	Grafana Alert Architecture	Rules, contact points, notification policies, state machine
40–60 min	Lab Part 1	Create alert rules on Open Alerts count panel
60–70 min	Lab Part 2	Configure email / Slack contact point + routing
70–80 min	Alert Design Framework	Thresholds, routing, escalation, fatigue
80–88 min	Case Studies	Zepto & HDFC Securities production alerting
88–90 min	Module 4 Wrap-Up	Course arc + Final Project preview

Section 1: Why Alerts Matter

A dashboard is only useful if someone is looking at it. In a 24-hour streaming system, no one watches 24 hours a day. Alerts let the data come to you — instead of you going to the data.

Without alerts:
  10:47 PM  INFY price drops 6% — alert fires in stock_alerts
  10:47 PM  Nobody sees it
  11:30 PM  Risk manager checks Grafana manually — too late

With alerts:
  10:47 PM  Grafana alert fires → Slack message in #risk-alerts
  10:48 PM  Manager reviews position, takes action

Tool	Purpose	Acts On
Logging	Record everything that happened	Engineers debugging after an event
Monitoring	Show current state continuously	Teams watching dashboards
Alerting	Notify when action is needed	Decision-makers who need to respond

Key principle: The goal of an alert is to trigger a human action. If an alert fires and nobody knows what to do, the alert is not useful — it is noise. Every alert rule should have a corresponding runbook: who gets notified and what they do.

Section 2: Grafana Alert Architecture

Three Components

Alert Rules — Conditions evaluated on a schedule. Each rule has a query and fires when the condition is met for a defined duration.
Contact Points — Where notifications go. Supported: Email, Slack, PagerDuty, Opsgenie, Teams, Webhook, Telegram, and 20+ more.
Notification Policies — Routing rules: which alert labels go to which contact point.

The Alert State Machine

NORMAL

→ condition met →

PENDING

→ "For" duration elapsed →

FIRING

→ condition clears →

RESOLVED

The "For" Duration — Why It Matters

Without a "For" duration, a single bad data point triggers an alert. With For: 2m, the condition must be true for 2 continuous minutes before the alert fires.

Without "For":
  Tick 1: Rs 1390 → FIRES immediately (bad tick? spike? genuine drop?)
  Tick 2: Rs 1410 → Resolves — false alarm. Engineers start ignoring alerts.

With For: 2m:
  Tick 1: Rs 1390 → enters PENDING state
  2 min later: still Rs 1385 → FIRES — genuine sustained move, worth acting on.

Section 3: Lab Part 1 — Create Alert Rules

Alert 1: "Too Many Open Alerts"

Navigate: Left sidebar → Alerting (bell icon) → Alert Rules → New alert rule

Name: Stock Alerts — High Open Count | Data source: MongoDB

Query: stock_alerts collection, match status=open, count

Condition: IS ABOVE 5 | Evaluate every: 1m | For: 2m

Labels: severity=warning, team=risk

Annotations: Summary: "Open alert count has exceeded threshold" | Description: "There are {{ $value }} unreviewed stock alerts."

Alert 2: "Symbol Price Anomaly" (from panel)

Open the INFY Price panel → Edit → Alert tab → Create alert rule from this panel
Condition: IS BELOW 1400 (absolute floor) | For: 1m
Labels: severity=critical, symbol=INFY

Threshold design discussion: Should the threshold be absolute (Rs 1,400) or relative (5% from yesterday's close)? Absolute is simpler to configure. Relative is more accurate but requires the query to compute the deviation. The consumer_alerts.py from Lecture 9 already handles relative alerting at the Kafka layer — Grafana can handle the absolute floor as a safety net.

Section 4: Lab Part 2 — Contact Points

Option A: Email

Requires SMTP configuration in Docker. Add environment variables to the Grafana run command:

-e GF_SMTP_ENABLED=true
-e GF_SMTP_HOST=smtp.gmail.com:587
-e GF_SMTP_USER=your@gmail.com
-e GF_SMTP_PASSWORD=your-app-password

Then: Alerting → Contact points → New → Type: Email → Add your address → Test

Option B: Slack Webhook (Recommended for demos)

Go to api.slack.com/apps → Create New App → Incoming Webhooks → Activate
Add New Webhook to Workspace → select channel #grafana-alerts
Copy webhook URL: https://hooks.slack.com/services/T.../B.../...
Grafana: Alerting → Contact points → New → Type: Slack → Paste URL → Test

Notification Routing by Label

Alerting → Notification policies → Edit root policy → Add matchers:

severity = critical → Slack
severity = warning → Email
Default (catch-all) → Email

Expected Slack Alert Message

FIRING: Stock Alerts — High Open Count
Severity: warning | Team: risk
Open alert count has exceeded threshold
Current value: 7
Starts at: 11:47:00
Dashboard: http://localhost:3000/d/...

Section 5: Alert Design Framework

Three Questions for Every Alert Rule

Who receives this alert and what will they do? If you can't answer in one sentence, the alert is not ready.
What is the cost of a false positive? Each false positive erodes trust. After 10, people ignore alerts.
What is the cost of a false negative? A missed alert on a financial system can mean regulatory exposure or monetary loss.

Notification Cost	Example	Threshold Approach
Low (Slack message, easy to dismiss)	Warning — 5+ open alerts	Sensitive threshold OK
Medium (email + calendar)	Price below floor	Threshold needs care, use "For: 2m"
High (PagerDuty, on-call woken up)	System outage, fraud spike	Very precise threshold, long "For" duration

Avoiding Alert Fatigue

Alert fatigue = when so many alerts fire that operators start ignoring them — including the ones that matter.

Set For: 2m or longer on non-critical rules — filters out single-tick blips
Delete any alert you can't write a runbook for
Use severity labels and separate channels (warnings vs critical)
Review and tune thresholds monthly — conditions change as data patterns change

Section 6: Case Studies

Case A: Zepto — Dark Store Operations

Zepto operates ~350 dark stores. Their alerting rules:

order_acceptance_rate < 90% for 3 minutes → Slack to store ops manager
avg_delivery_time > 35 min for 5 minutes → Escalation to city head
inventory_stockout_count > 20 SKUs → Automated reorder + alert

Discussion: Why is the "For" duration 3 minutes for order acceptance but 5 minutes for delivery time? What is different about the business consequence of each?

Case B: HDFC Securities — Trading Platform

price_feed_latency > 500ms → PagerDuty (on-call engineer, 24/7)
order_rejection_rate > 2% → Critical Slack + SMS to CTO
active_users_drop > 30% in 5 min → Infrastructure alert (possible outage)

Discussion: The order rejection rate alert goes to the CTO, not an engineer. Why? What does this tell you about how alerting functions as a business communication tool, not just a technical one?

Module 4 Summary

What Module 4 Delivered

	L14: Grafana Basics	L15: Dashboard Build	L16: Alerts
Concept	Data sources, panels, dashboards	Dashboard design + live multi-panel build	Alert rules, contact points, routing
Built	First panel: INFY time series	Stock monitor + transactions dashboards	Alert rules with Slack/email routing
Business skill	Understand the visual layer	Design a dashboard around a business question	Design an alerting strategy that creates action

Module 1 (L1-L5):    UNDERSTAND — What is streaming? How does Kafka work?
Module 2 (L6-L10):   BUILD — Producer, consumer, MongoDB, alert routing
Module 3 (L11-L13):  SCALE — Windowing, Spark, Flink — enterprise patterns
Module 4 (L14-L16):  VISUALISE — Grafana dashboards + alert routing
Module 5 (L17-L20):  PRESENT — Atlas Charts, final project, real architectures

Key Takeaways

✓ Key Takeaways — Lecture 16

Alerts close the feedback loop — data flows in, decisions flow out. Without alerts, dashboards require 24/7 human watching.
Three components: Alert Rules (conditions) → Notification Policies (routing) → Contact Points (delivery)
The "For" duration prevents noise — require a condition to persist for 1–2 min before firing
Every alert needs a runbook — who receives it, what they do. If you can't write this, the alert shouldn't exist.
Alert fatigue kills alerting systems — fewer, well-calibrated, high-signal alerts outperform many low-quality ones
Threshold design is a business decision — it depends on cost of false positives vs cost of missing a real event

Coming Up — Module 5: Atlas Charts & Final Project

L17: MongoDB Atlas Charts — native charting tool, no separate install required.
L18–L20: Final Project — architecture review, pipeline critique, presentations.
Start thinking: What business problem are you solving? Who is your end user? What alert would notify a manager without watching the screen?

Alerts in Grafana: From Data to Action

Contents

Learning Objectives

Session Timeline

Section 1: Why Alerts Matter

Section 2: Grafana Alert Architecture

Three Components

The Alert State Machine

The "For" Duration — Why It Matters

Section 3: Lab Part 1 — Create Alert Rules

Alert 1: "Too Many Open Alerts"

Alert 2: "Symbol Price Anomaly" (from panel)

Section 4: Lab Part 2 — Contact Points

Option A: Email

Option B: Slack Webhook (Recommended for demos)

Notification Routing by Label

Expected Slack Alert Message

Section 5: Alert Design Framework

Three Questions for Every Alert Rule

Avoiding Alert Fatigue

Section 6: Case Studies

Case A: Zepto — Dark Store Operations

Case B: HDFC Securities — Trading Platform

Module 4 Summary

What Module 4 Delivered

Key Takeaways

✓ Key Takeaways — Lecture 16

Coming Up — Module 5: Atlas Charts & Final Project