Apache Flink & True Streaming

Conceptual · Decision frameworks · No hands-on lab

Learning Objectives

• ✓ Explain in plain language the difference between micro-batch and true streaming
• ✓ Define exactly-once processing and explain why it matters for financial systems
• ✓ Articulate what "low latency" means in concrete business terms (ms → business impact)
• ✓ Apply the Flink vs Spark vs ksqlDB decision framework to real scenarios
• ✓ Complete the Module 3 framework: understand where each tool belongs in a data stack

Session Timeline

Time	Segment	Activity
0–10 min	Recap	L12 Spark — what is the micro-batch model in one sentence?
10–25 min	The Core Difference	What true streaming actually means
25–40 min	Exactly-Once Semantics	Why it exists, why banks care, why most don't need it
40–55 min	Latency in Business Context	When does 200ms vs 10ms change the outcome?
55–70 min	Full Decision Framework	Spark vs Flink vs ksqlDB vs Python — the complete map
70–82 min	Case Study: UPI Fraud	Walk through the real-time fraud architecture decision
82–88 min	Group Debate	"Spark is good enough for everything" — argue for or against
88–90 min	Module 3 Wrap-Up	The streaming landscape + preview of Module 4 (Grafana)

Section 1: Micro-Batch vs True Streaming

The Mental Model

Spark (micro-batch): Imagine a factory where a bell rings every 200 milliseconds. Workers grab everything in front of them, sort it, and send results forward — then wait for the next bell. The minimum time from arrival to processing: anywhere from 1ms (just after the bell) to 199ms (just before the next one).

Flink (true streaming): The same factory, but each worker processes an item the instant it arrives — no bell, no waiting. Processing time: 1–10 milliseconds from arrival.

Spark: →→→ [--200ms--] [--200ms--] [--200ms--]
                ↓            ↓            ↓
            run batch    run batch    run batch

Flink: →→→ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓
           (each event processed individually, within ~10ms)

Why This Architecture Difference Exists

Spark was designed first as a batch processing system — streaming was added later as "fast batching." Flink was designed from day one as a streaming system — batch processing was added as a special case. Neither is "better" — they made different primary trade-offs.

Latency in Business Terms

Latency	Roughly Equivalent To	Business Example
1ms	Blink of an eye ÷ 300	High-frequency trading signal
10ms	Blink of an eye ÷ 30	Payment fraud check (Flink typical)
100ms	Just perceptible as a pause	User feels slight lag
200ms	Noticeable delay	Spark micro-batch typical
500ms	Clearly slow	Google reports this hurts search usage
2,000ms	Very slow	User considers abandoning the page

Section 2: Exactly-Once Processing

The Three Processing Guarantees

Guarantee	Risk	Acceptable For	Not Acceptable For
At-most-once	Events may be lost on crash	Social media likes, page views	Financial transactions, medical records
At-least-once	Events may be double-counted on crash/replay	Analytics approximations, event logs	Billing, banking, inventory allocation
Exactly-once	No loss, no duplication — guaranteed	All financial and compliance cases	—

Why Exactly-Once Is Hard

When a system crashes mid-processing, it must answer on recovery: "Did I already write this result to the database before I crashed?" Achieving exactly-once requires coordinating the database write AND the Kafka offset confirmation as a single atomic operation.

Who Actually Needs Exactly-Once

Sector	Use Case	Need Exactly-Once?	Why
Banking	Transaction processing	Yes	Duplicate = double debit
E-commerce	Order revenue totals	Yes	Billing error = legal liability
Telecom	Call data records	Yes	Over/under billing customers
Marketing	Ad impression counting	No	~1% error is acceptable
Social media	Like/view counts	No	Approximate is fine
Logistics	Delivery ETA	No	Small error has no consequences

Spark vs Flink on Exactly-Once

• Spark: Exactly-once at the batch level — if a batch fails and is re-run, the entire batch is re-processed atomically
• Flink: Exactly-once at the per-event level — each event's processing and output are coordinated, even mid-batch

Section 3: When Latency Actually Matters

Scenario A — Online Payment Fraud Detection

With Spark (~200ms fraud check): user taps "Pay" and experiences essentially zero perceptible delay. With Flink (~10ms): same user experience. Business verdict: Spark is fine.

Scenario B — Payment Gateway at Very High Throughput

At 50,000 TPS, if the fraud pipeline becomes a bottleneck (batch takes longer than its trigger interval), transactions start queuing. Flink's consistent per-event processing avoids this queuing effect under load. Business verdict: At very high volume, Flink's consistency matters more than raw latency.

Scenario C — Financial Market Trading Alert

A 200ms delay means a competitor has already acted on the same signal. Flink at 10ms is competitive. For high-frequency trading, even Flink is often "too slow" — those systems use custom hardware. Business verdict: Spark is not appropriate here.

Section 4: The Complete Decision Framework

Dimension	Python Consumer	Apache Spark	ksqlDB	Apache Flink
Latency	< 100ms	~200ms–2s	~50ms	< 10ms
Scale limit	~1K events/sec	Millions/sec	Hundreds K/sec	Millions/sec
Exactly-once	No	Yes (batch)	Yes	Yes (per-event)
Infrastructure cost	None	Medium–High	Low	High
Talent cost	Low	Medium	Low	Very High
Best for	Learning, prototypes	Batch + streaming teams	SQL-native teams	Payments, trading

Organisational Cost Comparison

Factor	Apache Spark	Apache Flink
Monthly infra cost	₹30K–₹3L (Databricks/EMR)	₹50K–₹5L (self-managed or Confluent)
Time to first production job	2–4 weeks	6–12 weeks
Hiring difficulty	Moderate	Very Hard
Salary premium	40–60% over general engineer	80–120% over general engineer

Section 5: Case Study — UPI Fraud Detection

The Problem

NPCI processes ~5 billion UPI transactions/month (~1,900 TPS average, much higher at peak). Fraud must be detected and flagged before clearing: <300ms SLA.

Each transaction must be checked for:

1. New device/account combination
2. Unusual amount for this user's history
3. Mule account pattern (many small transactions to same payee today)
4. Location consistency with recent transactions
5. Active fraud alert on beneficiary VPA

Architecture Options

Option	Assessment
Manual Python Consumer	Single-threaded, ~1K events/sec. ❌ Cannot handle 1,900 TPS peak.
Apache Spark Streaming	Handles 1,900 TPS across distributed cluster. Micro-batch 200ms → within 300ms SLA. ✅ Scale sufficient, latency acceptable.
Apache Flink	~10ms per-transaction latency. Stateful per-user history in-memory (no database roundtrip for checks 1–4). ✅ Better latency margin, better for stateful checks. Higher cost.

Discussion Questions

1. The 300ms SLA: Spark processes in ~200ms, leaving 100ms buffer. Is that enough margin on peak days when transaction volume is 3× normal?
2. The stateful problem: Check 3 (mule account pattern) requires knowing all transactions to a payee in the last 24 hours. How do you query 24 hours of UPI history in <10ms?
3. Build vs buy: Custom streaming fraud pipeline vs licensed SaaS (SAS, NICE Actimize, Featurespace). What are the trade-offs?

Group Debate: "Spark Is Good Enough for Everything"

Time: 5 minutes to prepare arguments, 5 minutes of structured debate, 2 minutes for faculty summary.

Faculty debrief: Both positions have merit — context determines correctness. The question is always "what does the business need?" not "what's the best technology?"

Module 3 Summary

Module 2: We made Kafka work — producers, consumers, MongoDB, alert routing
Module 3: We learned to aggregate intelligently (windows), and understand the
          frameworks that handle this at scale (Spark, Flink)

Module 4: We stop building backends and focus on the output — dashboards.
          All of this data you've been streaming? Now we make it visible.

Key Takeaways