Your First Producer

Today's 90 Minutes

What Is a Producer?

A producer is any application that publishes events to a Kafka topic. It doesn't know or care who reads the data — it just sends.

What a producer does in 3 steps

1. Connect — establishes a TCP connection to the Kafka broker (your Docker container)
2. Serialize — converts your Python dict → bytes (usually JSON) so Kafka can store it
3. Send — publishes the message to a named topic; Kafka acknowledges receipt

Anatomy of producer.py

Let's break down every line and why it exists.

Python Environment Setup

A virtual environment (venv) keeps your project's libraries separate from your system Python — no version conflicts between projects.

Step 1 — Create the SDA project folder

# Open Terminal and run:
mkdir ~/SDA
cd ~/SDA

:: Open Command Prompt (cmd) and run:
:: SDA folder is usually on C: or D: — choose whichever drive you use
mkdir C:\SDA
cd C:\SDA
:: If your files are on D: drive instead:
:: mkdir D:\SDA  and  cd D:\SDA

Step 2 — Create the virtual environment

python3 -m venv venv

python -m venv venv

Activate & Install Packages

You must activate the venv every time you open a new terminal before running any Python files.

Activate the venv

cd ~/SDA
source venv/bin/activate
# Your prompt changes to: (venv) $

cd C:\SDA
venv\Scripts\activate
:: Your prompt changes to: (venv) C:\SDA>
:: If your SDA folder is on D: use:  cd D:\SDA

Install required libraries

pip install kafka-python yfinance

Verify installation

python -c "from kafka import KafkaProducer; import yfinance; print('✅ All good')"

Stream stock_data.csv to Kafka

Our first producer reads a CSV file row by row and streams each row as a JSON message, simulating a live tick feed.

import csv, time, json, sys
from kafka import KafkaProducer, errors

KAFKA_BROKER = 'localhost:9092'
TOPIC        = 'stock-topic'

try:
    producer = KafkaProducer(
        bootstrap_servers=KAFKA_BROKER,
        value_serializer=lambda v: json.dumps(v).encode('utf-8'),
        retries=5
    )
    producer.partitions_for(TOPIC)
    print("✅ Connected to Kafka at", KAFKA_BROKER)
except errors.NoBrokersAvailable:
    print("❌ Kafka not running — docker-compose up -d")
    sys.exit(1)

with open('stock_data.csv', 'r') as f:
    reader = csv.DictReader(f)
    for row in reader:
        producer.send(TOPIC, value=row)
        print(f"Produced: {row}")
        time.sleep(1)

producer.flush()

Run it

# Make sure docker-compose is up and venv is activated
python producer.py

Fetching Real Data with yFinance

Instead of a static CSV, we can pull real historical and live market data directly from Yahoo Finance using the yfinance Python library.

What it gives you

Quick test

import yfinance as yf
tcs = yf.Ticker("TCS.NS")
df  = tcs.history(period="1d", interval="1m")
print(df.tail(3))

TCS Live Data — tcs_producer.py

Fetches 5 days of 1-minute OHLCV bars for TCS.NS from Yahoo Finance and streams them to Kafka.

import json, time, sys
import yfinance as yf
from kafka import KafkaProducer, errors

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

tcs = yf.Ticker("TCS.NS")
df  = tcs.history(period="5d", interval="1m").reset_index()

for _, row in df.iterrows():
    record = {
        "symbol":    "TCS.NS",
        "timestamp": str(row["Datetime"]),
        "close":     round(float(row["Close"]), 2),
        "volume":    int(row["Volume"])
    }
    producer.send("stock-topic", value=record)
    print(f"TCS | {record['timestamp']} | ₹{record['close']}")
    time.sleep(0.5)

producer.flush()

BTC-USD — Real-Time 24×7

Bitcoin trades every second of every day. This producer polls the live price every 10 seconds and never stops — until you press Ctrl+C.

import json, time, sys
from datetime import datetime
import yfinance as yf
from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

print("₿ BTC-USD live stream — Ctrl+C to stop\n")
count = 0
while True:
    btc    = yf.Ticker("BTC-USD")
    record = {
        "symbol":    "BTC-USD",
        "timestamp": datetime.utcnow().isoformat(),
        "price":     round(float(btc.fast_info.last_price), 2),
        "market":    "crypto"
    }
    producer.send("stock-topic", value=record)
    count += 1
    print(f"[{count}] BTC: ${record['price']:,.2f}")
    time.sleep(10)   # poll every 10 seconds

Multiple Producers, One Topic

Kafka topics can receive events from many producers simultaneously. Each producer writes independently — Kafka merges all messages into a single ordered stream.

A consumer reading stock-topic receives both TCS and INFY messages interleaved, identified by the symbol field.

Why this is powerful

• Single pipeline, multiple sources — one consumer can analyse a whole portfolio
• Independent producers — TCS and INFY teams can deploy separately, no coordination needed
• Easy to add more — want RELIANCE? Add reliance_producer.py, zero changes to the consumer

TCS + INFY — Same Topic

Open two terminals. Run one producer in each. Both stream to stock-topic simultaneously.

# Only change vs tcs_producer.py: symbol and ticker
infy = yf.Ticker("INFY.NS")         # ← Infosys
df   = infy.history(period="5d", interval="1m").reset_index()

for _, row in df.iterrows():
    record = {
        "symbol":    "INFY.NS",     # ← Infosys symbol
        "timestamp": str(row["Datetime"]),
        "close":     round(float(row["Close"]), 2),
        "volume":    int(row["Volume"])
    }
    producer.send("stock-topic", value=record)

Run both at the same time

python tcs_producer.py

python infy_producer.py

Now open the Kafka console consumer in a third terminal to see both streams interleaved:

docker exec -it kafka kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic stock-topic --from-beginning

All Lab Files

Download all files for today's lab. Place them in your ~/SDA folder and run with your activated venv.

pip install kafka-python yfinance

Key Takeaways

• A producer publishes events — it doesn't know who reads them. Decoupling is the point.
• Always call producer.flush() before exiting — without it, the last messages may be silently dropped.
• Virtual environments prevent dependency conflicts — activate before every session.
• yFinance gives you real stock and crypto data — history for replays, fast_info for live polling.
• Multiple producers → one topic is normal. Each source adds its symbol field so consumers can filter.
• BTC-USD is the simplest real-time source — 24×7, no market hours, always moving.