Lecture 11 · Stream Processing
Windowing & Aggregations
Slicing the infinite stream into measurable, actionable chunks
Why You Can't Aggregate an Infinite Stream
In a traditional database: SELECT SUM(revenue) FROM orders WHERE date = '2025-08-23' — the data already exists in full.
In a streaming system:
- Data is infinite — it never stops arriving
- The query runs continuously — results change every second
- You can't do
SUM(all revenue) because "all" grows forever
Infinite stream: →→→→→→→→→→→→→→→→→→→→→→→→→→→
Solution — slice into finite windows:
[ window 1 ][ window 2 ][ window 3 ] ...
SUM, COUNT SUM, COUNT SUM, COUNT
| Business Question | Requires |
|---|
| "What were our total sales last quarter?" | Batch aggregation |
| "What were sales in the last 5 minutes?" | Tumbling window |
| "What's the rolling 1-hour average order value?" | Sliding window |
| "How long was this user's shopping session?" | Session window |
Tumbling Windows
Fixed-size, non-overlapping. Every event belongs to exactly ONE window.
Stream: →→→→→→→→→→→→→→→→→→→→→→→→→→→→→
|----5 min----|----5 min----|----5 min----|
[ Window 1 ][ Window 2 ][ Window 3 ]
SUM, COUNT SUM, COUNT SUM, COUNT
Best for
- Periodic revenue summaries
- Hourly / daily billing cycles
- Regulatory reporting (every 15 min)
- Flash sale performance per hour
Key Properties
- No overlap between windows
- Fixed equal duration
- Simple — unambiguous which window
- Closes and resets at interval end
Real Examples
- Stock exchange: 5-min OHLCV bars
- Uber: surge recalculated every 3 min
- Zepto: dark store throughput per 5 min
- NPCI: UPI volume per hour
💡 Mental model: Tumbling windows are like pages in a diary — each page covers exactly one day, and entries never appear on two pages at once.
Sliding Windows
Fixed-size, overlapping. The window advances by a step smaller than its duration. One event can appear in multiple windows.
Stream: →→→→→→→→→→→→→→→→→→→→→→→→→→→
Window 1: [====10 min====]
Window 2: [====10 min====]
Window 3: [====10 min====]
↑ Step: 5 min (50% overlap)
The Boundary Problem — Why Sliding Beats Tumbling for Fraud
Tumbling (5-min): T1(2:01) T2(2:04) T3(2:07)
Window 1 (2:00–2:05): sees T1, T2 — 2 txns ← looks normal
Window 2 (2:05–2:10): sees T3 — 1 txn ← looks normal
PROBLEM: cluster split across boundary → fraud missed
Sliding (5-min, 1-min step):
Window at 2:06 (2:01–2:06): sees T1, T2, T3 — 3 txns ← DETECTED
💡 Use sliding windows when events that belong together logically might straddle a fixed boundary — fraud clusters, rate limiting, trend detection.
Session Windows
Variable-length, behaviour-driven. A window opens when a user becomes active; it closes after a configurable period of inactivity.
User A: →Event→Event→Event→ (gap > timeout) →Event→Event→
[----Session 1-------] [--Session 2--]
↑ inactivity timeout = window closes
Defined By
- An inactivity timeout
- e.g. 30-minute gap = session ends
- Duration varies per user
- No artificial clock boundary
Best for
- User journey / funnel analysis
- App session duration
- Customer support grouping
- IoT device activity periods
Real Examples
- Netflix: watch session tracking
- Google Analytics: 30-min timeout
- Banking: ATM visit grouping
- Call centre: 2-hr episode window
💡 Key insight: Session window output reveals user behaviour patterns — short session with high spend, or long session with no conversion — that fixed windows would average away.
Event Time vs Processing Time
| Timestamp | Meaning | Example |
|---|
| Event time | When it actually happened (logged at source) | Payment at 2:03:45 PM |
| Processing time | When the consumer read the event | Consumer got it at 2:04:12 PM |
The Late Data Problem
Mobile app logs a click offline → arrives 45 minutes late. Your 2:00–2:05 window has already closed. What do you do?
- Drop it — simple, may lose data
- Re-open the window — complex, delays results
- Route to side output — production best practice
Watermarks — The Late Data Policy
Watermark = max_event_time_seen − allowed_lateness
Example: Latest event time = 2:05:00, allowed_lateness = 30s
→ Watermark = 2:04:30
→ Windows ending before 2:04:30 can safely close
→ Events with event_time < 2:04:30 are "late"
Trade-off: Larger allowed_lateness = more accurate (fewer dropped events) but longer wait before window closes. Tune based on source reliability.
Choosing the Right Window
Is the window size fixed (time-based)?
NO → Does window depend on user inactivity?
YES → Session Window
NO → Count-based (rare, not covered today)
YES → Do windows overlap?
NO → Tumbling Window ✅ (clean periodic buckets)
YES → Sliding Window ✅ (rolling metrics, anomaly detection)
| Scenario | Window Type | Why |
|---|
| 5-minute revenue summary | Tumbling 5 min | Clean buckets, non-overlapping |
| Fraud pattern detection | Sliding 15 min / 1-min step | Catches cross-boundary clusters |
| User session duration | Session 30-min timeout | Aligns with user behaviour |
| Hourly cloud billing | Tumbling 60 min | Non-overlapping billing periods |
| Trending topics | Sliding 10 min / 2-min step | Smooth continuous signal |
| App user journey | Session 30-min timeout | Funnel analysis per session |
Lab: consumer_window.py
Reads from transactions-topic (L10 dataset) and runs both window types simultaneously.
| Terminal | Command | What It Does |
|---|
| 1 | docker compose up -d | Start Kafka (keep running) |
| 2 | python consumer_window.py | Start windowed consumer |
| 3 | python transaction_producer.py | Stream the transactions CSV |
Expected Output
=== Windowed Consumer Starting ===
Tumbling: 60s buckets | Sliding: 180s window / 30s step
[SLIDING] Last 180s: 31 events | Rolling Avg: ₹1,823.45
[SLIDING] Last 180s: 48 events | Rolling Avg: ₹1,856.80
[TUMBLING] 29 events | Total: ₹52,840.00 | Avg: ₹1,822.07
[SLIDING] Last 180s: 71 events | Rolling Avg: ₹2,140.80 ← fraud cluster entering
[SLIDING] Last 180s: 74 events | Rolling Avg: ₹2,380.50 ← peak: fraud in window
[TUMBLING] 34 events | Total: ₹61,200.00 | Avg: ₹1,800.00 ← tumbling stays flat!
[SLIDING] Last 180s: 49 events | Rolling Avg: ₹1,847.33 ← fraud aged out
Observe: Sliding average spikes during the fraud cluster (transactions across window boundaries). Tumbling average stays near ₹1,800 — the fraud is hidden by the bucket boundary. This is exactly the boundary problem in action.
Manual Windowing vs Production
Our consumer_window.py is educational. In production, it has real gaps:
| Limitation | Consequence |
|---|
| State in Python variables | Restart wipes all window history |
| Single-threaded | Can't scale horizontally |
| No late data handling | Late events silently ignored |
| No fault tolerance | Crash mid-window = lost aggregation |
Production Alternatives
| Framework | Latency | Throughput | Best For |
|---|
| Manual Python (ours) | < 100ms | ~1K/sec | Learning, prototyping |
| Apache Spark | ~200ms–2s | Millions/sec | Large scale, SQL teams (L12) |
| Apache Flink | < 10ms | Millions/sec | Financial, exactly-once (L13) |
| ksqlDB | ~50ms | 100K+/sec | SQL-only teams, Kafka-native |
MBA takeaway: Manual windowing is a teaching tool. Any system handling > 50,000 events/second needs a framework. L12 and L13 cover how to choose.
Key Takeaways
- ✓Windows slice an infinite stream into finite, aggregatable chunks — without them, streaming analytics is impossible
- ✓Tumbling = equal non-overlapping buckets → use for periodic reporting and billing
- ✓Sliding = overlapping, rolling → use for moving averages, anomaly detection, and fraud patterns that cross boundaries
- ✓Session = user-behaviour-driven, variable duration → use for journey and funnel analytics
- ✓Event time > processing time for accuracy — watermarks control how long you wait for late data
- ✓Manual windowing works for prototypes — Spark, Flink, or ksqlDB for production scale
Downloads
Dataset: use generate_transactions.py + transaction_producer.py from Lecture 10.