Lecture 07 · Module 2

Reading from Kafka

Console Consumer · Python Consumer · Two Producers, One Stream

📅 Module 2 — Making Kafka Work

⏱ 90 min

🎓 MBA Streaming Data Analytics

02 / 08 Agenda

Today's 90 Minutes

Time	Segment	Format
0 – 10	Recap — TCS + INFY producers from Lecture 6	Q&A
10 – 20	What is a consumer? How it connects to a topic	Lecture
20 – 40	Console consumer — verify stream is live	Demo + Lab
40 – 70	Python consumer — read from stock-topic	Lab
70 – 85	One consumer reading both TCS & INFY data	Lab
85 – 90	Takeaways + Lecture 8 preview	Wrap-up

03 / 08 Recap

Where We Left Off — Lecture 6

In Lecture 6 you built two producers that both write to stock-topic. Today we read from it.

🐍

tcs_producer.py

TCS.NS · yFinance
historical prices

→

📨

stock-topic

Kafka topic
all messages here

→

👁️

consumer.py

Today's goal
read everything

🐍

infy_producer.py

INFY.NS · yFinance
writes same topic

↗

Key insight: One consumer subscribing to stock-topic automatically sees messages from all producers writing to it — TCS and INFY arrive in the same stream.

04 / 08 Demo

Step 1 — Kafka Console Consumer

Before writing Python — verify your stream is live with zero code. Run a producer in one terminal, this in another.

Read all messages (from the beginning)

bash

docker exec -it sda-kafka-1 kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic stock-topic \
  --from-beginning

Read live messages only (from now)

bash

docker exec -it sda-kafka-1 kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic stock-topic

⚙️ Command Builder — Enter your details

Docker Container Name

Kafka Topic

Offset

Generated command:

05 / 08 Lab

Step 2 — Python Consumer

Subscribe to stock-topic and print every message. Open a terminal alongside your running producer.

python · consumer_stock.py

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'stock-topic',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',   # read ALL past messages first
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)

print("Listening on stock-topic...")
for msg in consumer:
    data = msg.value
    print(data)

Try it: Start the consumer first, then run python tcs_producer.py in another terminal. Watch messages arrive live.

Install kafka-python first

bash

pip install kafka-python

06 / 08 Lab

One Consumer, Two Producers

Run tcs_producer.py and infy_producer.py simultaneously. One consumer reads both — differentiate by the symbol field.

python · consumer_stock.py (updated)

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'stock-topic',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)

tcs_count = infy_count = 0

print("📡 Listening to stock-topic (TCS + INFY)...")
for msg in consumer:
    data  = msg.value
    sym   = data.get('symbol', '?')
    price = data.get('price', data.get('Close', '?'))

    if sym == 'TCS.NS':
        tcs_count += 1
        print(f"🔵 TCS   ₹{price}  (msg #{tcs_count})")
    elif sym == 'INFY.NS':
        infy_count += 1
        print(f"🟢 INFY  ₹{price}  (msg #{infy_count})")
    else:
        print(f"⚪ {sym} → {data}")

Expected output

terminal

📡 Listening to stock-topic (TCS + INFY)...
🔵 TCS   ₹3842.5  (msg #1)
🔵 TCS   ₹3851.0  (msg #2)
🟢 INFY  ₹1724.3  (msg #1)
🔵 TCS   ₹3860.2  (msg #3)
🟢 INFY  ₹1731.8  (msg #2)

07 / 08 Hands-On Lab

Lab — Run It All Together

Open 4 terminal windows. Follow the sequence below.

Terminal	Command	What it does
T1	`docker-compose up -d`	Start Kafka + Zookeeper
T2	`python tcs_producer.py`	Stream TCS prices → stock-topic
T3	`python infy_producer.py`	Stream INFY prices → stock-topic
T4	`python consumer_stock.py`	Read all messages, tag by symbol

Challenge: Modify consumer_stock.py to also print the count of messages received for each symbol every 5 messages. Hint: use a counter variable and if (tcs_count + infy_count) % 5 == 0.

Next → Lecture 8: Save this streaming data to MongoDB Atlas so it persists beyond the terminal session.

08 / 08 Wrap-Up

Key Takeaways

Console consumer = your debugger — verify data is flowing before writing any Python
earliest reads all past messages; latest reads only new ones
One topic, many producers — a single consumer sees all of them automatically
Filter by message fields — use symbol, source or any field to separate streams in code
Kafka is durable — messages stay in the topic; your consumer can re-read any time

Download Lab Files

🐍

consumer_stock.py

Read TCS + INFY from stock-topic

↓ .py

Next → Lecture 8: Save streaming data to MongoDB Atlas — free cluster setup + persistent storage