Smarter Consumers:
Aggregations & Alert Routing

Where We Are & What We Build

In Lecture 8 we saved every tick to MongoDB stock_prices. Today we make the consumer smarter — it now computes statistics on batches of prices and automatically flags significant moves.

Today's Build — Three Steps

Why Aggregate in the Consumer?

Raw tick data is noisy. A single bad tick — a data quality issue, a momentary spike — can trigger false alerts. Aggregating over a small window gives you a more stable signal.

The mean of each completed batch becomes the new prev_close for the next batch. This rolling reference prevents alerts from chasing price after a large move.

Buffer & Compute Stats

We accumulate incoming prices in a per-symbol list. When the buffer reaches BUFFER_SIZE, we compute stats and reset.

from collections import defaultdict
import statistics

BUFFER_SIZE = 5   # compute stats after every 5 ticks per symbol

# State: one buffer and one prev_close entry per symbol
price_buffer = defaultdict(list)  # {"INFY": [1410, 1415, 1408, 1420, 1412]}
prev_close   = {}                  # {"INFY": 1413.0}  — updated each batch

def compute_stats(prices):
    return {
        "count": len(prices),
        "sum":   round(sum(prices), 2),
        "mean":  round(statistics.mean(prices), 2),
        "min":   round(min(prices), 2),
        "max":   round(max(prices), 2),
    }

# Inside the consumer loop:
price_buffer[symbol].append(price)

if len(price_buffer[symbol]) >= BUFFER_SIZE:
    stats = compute_stats(price_buffer[symbol])
    print(f"sum={stats['sum']}  mean={stats['mean']}  min={stats['min']}  max={stats['max']}")
    price_buffer[symbol] = []   # reset buffer

Sample Output — Step 1 Only

From Stats to Alerts — the 5% Rule

After every completed batch, compare the batch's min and max against prev_close — the mean from the previous batch.

ALERT_PCT = 5.0   # alert threshold in percent

def check_alert(symbol, stats):
    if symbol not in prev_close:
        return []  # first batch — no reference yet

    pc      = prev_close[symbol]
    reasons = []

    min_dev = abs(stats["min"] - pc) / pc * 100
    max_dev = abs(stats["max"] - pc) / pc * 100

    if min_dev > ALERT_PCT:
        reasons.append(f"MIN_DEV {stats['min']:.2f} ({-min_dev:.1f}% vs close {pc:.2f})")
    if max_dev > ALERT_PCT:
        reasons.append(f"MAX_DEV {stats['max']:.2f} (+{max_dev:.1f}% vs close {pc:.2f})")

    return reasons  # empty list = no alert

# After batch is full:
reasons = check_alert(symbol, stats)
prev_close[symbol] = stats["mean"]  # update reference for next batch

Two Collections, One Consumer

The routing logic is a single if. Same consumer, same Kafka topic — two very different use cases served.

# Every message → stock_prices (unchanged from L8)
prices_col.insert_one({**data, "_saved_at": datetime.utcnow().isoformat()})

# When buffer full — route alert to stock_alerts if triggered
if reasons:
    alerts_col.insert_one({
        "symbol":        symbol,
        "stats":         stats,          # full sum/mean/min/max of the batch
        "prev_close":    prev_close.get(symbol),
        "alert_reasons": reasons,
        "alert_at":      datetime.utcnow(),
        "status":        "open",          # open | reviewed | resolved
    })

What Each Collection Contains

consumer_alerts.py — Complete File

from kafka import KafkaConsumer
from pymongo import MongoClient
from collections import defaultdict
from datetime import datetime
import json, statistics

BUFFER_SIZE = 5    # ticks per batch
ALERT_PCT   = 5.0  # deviation threshold %

prices_col = db["stock_prices"]   # every message
alerts_col = db["stock_alerts"]   # flagged batches

price_buffer = defaultdict(list)
prev_close   = {}

for msg in consumer:
    symbol, price = data["symbol"], float(data["price"])

    # Step 1: save every tick
    prices_col.insert_one({**data, "_saved_at": datetime.utcnow().isoformat()})

    # Step 2: buffer
    price_buffer[symbol].append(price)

    if len(price_buffer[symbol]) >= BUFFER_SIZE:
        stats   = compute_stats(price_buffer[symbol])
        reasons = check_alert(symbol, stats)

        if reasons:
            alerts_col.insert_one({
                "symbol": symbol, "stats": stats,
                "prev_close": prev_close.get(symbol),
                "alert_reasons": reasons,
                "alert_at": datetime.utcnow(), "status": "open"
            })
            print(f"🚨 ALERT [{symbol}] → {', '.join(reasons)}")

        prev_close[symbol] = stats["mean"]
        price_buffer[symbol] = []

Lab — Run & Observe

What Must Be Running

Expected Terminal Output

Check Atlas After 2–3 Minutes

• stock_prices — should show all tick documents (same as L8)
• stock_alerts — documents only appear when 5% threshold is breached
• Each alert document includes stats (full batch), prev_close, alert_reasons, status: "open"

Assignment 2 & What's Next

Assignment 2 — Due 1 September

Tag your GitHub submission: [A2][SDA-1], [A2][SDA-2], or [A2][SDA-G] based on your section.

Coming Up