Apache Spark: The Engine Behind Big Data Streaming

Technology literacy · Decision frameworks · MBA case studies

Learning Objectives

• ✓ Explain in plain language what Apache Spark is and why it replaced Hadoop
• ✓ Describe the micro-batch model and what it means for latency
• ✓ Identify scenarios where Spark is the right stream processing tool
• ✓ Read a Spark job description and understand what it does at a high level
• ✓ Evaluate the organisational cost of adopting Spark vs simpler alternatives

Session Timeline

Time	Segment	Activity
0–10 min	Recap	L11 windowing — which window type would you use for fraud?
10–30 min	The Story of Spark	Why it was built, what problem it solved, who uses it
30–45 min	Micro-Batch Model	How Spark Streaming works — trigger intervals explained
45–60 min	Faculty Demo	Live spark_consumer.py — observe output, understand structure
60–70 min	Decision Framework	When Spark is right, when it is overkill
70–82 min	Case Study Discussion	Flipkart, Swiggy — why did they choose Spark?
82–88 min	Group Exercise	"Which tool?" — match 5 scenarios to the right framework
88–90 min	Wrap-Up	Key takeaways + preview of L13 (Flink)

Section 1: The Story of Apache Spark

The Problem Spark Solved

Before Spark, the dominant big data technology was Hadoop MapReduce (circa 2006–2012). MapReduce was revolutionary — it let you process petabytes of data across thousands of machines. But it had one painful limitation: it read from disk and wrote back to disk between every step of a multi-step job.

MapReduce workflow:
Step 1: Read from disk → process → write to disk
Step 2: Read those results from disk → process → write to disk again
...
A 10-step ML job = 10 disk reads + 10 disk writes. ~4 hours.

Spark workflow:
Step 1: Read from disk → keep in memory
Step 2: Process (in memory) → keep in memory
...
Final: Write result to disk

The same 10-step job: 10–100× faster.

Spark's Rise

Year	Milestone
2009	Created at UC Berkeley AMPLab
2013	Donated to Apache Software Foundation
2016	Spark 2.0: Structured Streaming API (DataFrame-native)
2021	Databricks (Spark's commercial backer) valued at $43 billion
2024	~80% of Fortune 500 companies run Spark workloads

Section 2: How Spark Streaming Works — The Micro-Batch Model

The Fundamental Design

Spark Structured Streaming collects events for a short interval (the trigger interval), then processes that chunk as a mini-batch, emits results, and repeats. The trigger interval controls the latency vs cost balance:

Trigger Interval	Latency	Cost	When to Use
50ms	~50ms	Very High	Near-real-time financial systems
200ms (default)	~200ms	High	Most production streaming
1 second	~1s	Medium	Dashboards, moderate freshness
10 seconds	~10s	Low	Aggregations, batch-ish workloads
1 minute	~1min	Very Low	Periodic summaries

The DataFrame Mental Model

The most important concept: a stream looks like an infinite table. You write almost the same code for batch analysis and live streaming. Your team doesn't need to learn a completely different paradigm.

Traditional table (static): fixed number of rows, same result every query.

Spark Streaming DataFrame (live):
Row 1: arrived 10:01:00
Row 2: arrived 10:01:01
Row 3: arriving right now...
Row 4: arriving in 2 seconds...
  ↳ infinite rows; query result changes every trigger interval

What "Distributed" Actually Means

Spark splits your stream across multiple workers (executors), each handling a portion of the load in parallel. On a laptop: one machine, multiple threads. In production: 10–100 machines, each handling a partition of the Kafka topic.

Section 3: Faculty Demo — Spark in Action

This is an observation exercise. Do not install Spark. The goal is to read and understand the code structure.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, from_json, window, sum as spark_sum, count, avg
from pyspark.sql.types import StructType, StructField, StringType, DoubleType

# 1. Start the Spark engine
spark = SparkSession.builder.appName("SDA-Demo").getOrCreate()

# 2. Read a live Kafka topic — creates an "infinite DataFrame"
raw_stream = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "transactions-topic") \
    .load()

# 3. Parse JSON payload into typed columns
parsed = raw_stream.select(from_json(col("value").cast("string"), schema).alias("d")).select("d.*")

# 4. Group by 1-minute tumbling windows — same syntax as batch SQL
windowed = parsed.groupBy(window("timestamp", "1 minute")) \
    .agg(count("*").alias("transactions"), spark_sum("amount").alias("total_spend"))

# 5. Write results every 10 seconds
query = windowed.writeStream.outputMode("update").format("console") \
    .trigger(processingTime="10 seconds").start()
query.awaitTermination()

Discussion questions during demo:

1. The stream is an "infinite table." Why is that useful to a data engineer?
2. Point at window("timestamp", "1 minute") — What type of window is this? What happens to events that span two windows?
3. Point at outputMode("update") — What's the alternative? When would you use "complete" mode?

Section 4: The Spark Decision Framework

Three Questions Before Recommending Spark

Question 1 — How much data?

• < 1 million events/day → Manual Python consumer (L7–L10) is fine
• 1M–100M events/day → Spark starts to make sense
• > 100M events/day → Spark (or Flink) is appropriate

Question 2 — How fresh must the data be?

• "Within 1 hour" → Batch job, no streaming needed
• "Within 1 minute" → Spark Streaming (default trigger)
• "Within 1 second" → Spark with fast trigger, or Flink
• "Within 100ms" → Flink only

Question 3 — What does your team know?

• Team knows Python/SQL → Spark (PySpark + SparkSQL)
• Team only knows SQL → ksqlDB (SQL on Kafka, no cluster)
• Team is 2 engineers → Manual Python consumer, grow into Spark

The Organisational Cost — Often Underestimated

Cost	What It Means
Infrastructure	Cluster or Databricks/EMR — ₹15,000–₹5,00,000/month
Expertise	Spark engineers are 2–3× more expensive than general Python engineers
Complexity	Debugging a distributed job is harder than debugging a Python script
Setup time	Initial cluster setup + CI/CD integration: 2–6 weeks engineering time

Section 5: Case Study Discussion

Case A: Flipkart — Big Billion Days

~500 million events/hour during peak. Latency required: <2 minutes for seller dashboards and inventory alerts. Team already Spark-native on Databricks/Azure. Python consumer would be overwhelmed immediately at this scale.

Discussion: At what point in Flipkart's growth would you have made the switch to Spark?

Case B: Swiggy — Delivery ETA Prediction

Multiple simultaneous streams: restaurant POS, driver GPS (every 10s), traffic API data. Spark's MLlib lets the prediction model run in the same pipeline as data processing. Latency required: <30 seconds for ETA updates.

Discussion: What window type would you use for the GPS ping stream — tumbling (every 30s) or sliding (rolling 2 min)?

Case C: Small FinTech — The Counter-Example

200,000 transactions/day = ~2 events/second. Existing Python consumer handles this at 5% CPU utilisation. Spark would add ₹40,000/month in Databricks costs + 1 engineer to maintain it. Latency requirement: within 5 minutes.

Group Exercise: "Which Tool?"

Work in pairs. For each scenario, select the right tool and justify in 2 sentences.

Options: Manual Python Consumer | Apache Spark | Apache Flink | ksqlDB

#	Scenario	Your Choice	Why (2 sentences)
1	EdTech startup, 5,000 students, real-time quiz score aggregation
2	HDFC Bank, 10M transactions/hour, fraud must be flagged < 200ms
3	Zepto, order fulfilment metrics per dark store per 5-min window
4	Small logistics company, "live" delivery dashboard updated every 10 min
5	Stock exchange, trade volume per symbol per second for regulatory reporting

Key Takeaways