Saving Streams to MongoDB

Today's 90 Minutes

Why Save Streaming Data?

A Kafka consumer that only prints to the terminal loses everything the moment you close it. Persistence unlocks the business value.

MongoDB Atlas — Free Cluster Setup

Takes about 5 minutes. You get 512 MB free forever — more than enough for this course.

1. Go to https://www.mongodb.com/cloud/atlas → click Try Free → sign up with Google or email.
2. Choose M0 Free tier → pick AWS Mumbai (ap-south-1) → click Create Deployment.
3. Create a database user — set a username and password. Write these down — you'll need them in your Python code.
4. Network Access → Add IP Address → Allow Access from Anywhere (0.0.0.0/0). This lets your laptop connect.
5. Click Connect → Drivers → Python → copy the connection string. It looks like:
   mongodb+srv://username:password@cluster0.xxxxx.mongodb.net/

Install pymongo & Test Connection

Install

pip install pymongo kafka-python

Test your Atlas connection

from pymongo import MongoClient

MONGO_URI = "mongodb+srv://username:password@cluster0.xxxxx.mongodb.net/"

client = MongoClient(MONGO_URI)

# List databases to confirm connection works
dbs = client.list_database_names()
print("✅ Connected! Databases:", dbs)

# Write a test document
db  = client["sda_course"]
db["test"].insert_one({"hello": "world"})
print("✅ Test document inserted!")

consumer_mongodb.py — Save Every Message

from kafka import KafkaConsumer
from pymongo import MongoClient
import json
from datetime import datetime

MONGO_URI = "mongodb+srv://username:password@cluster0.xxxxx.mongodb.net/"

client     = MongoClient(MONGO_URI)
db         = client["sda_course"]
collection = db["stock_prices"]
print(f"✅ Connected → {db.name}.{collection.name}")

consumer = KafkaConsumer(
    'stock-topic',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)

saved = 0
print("📡 Listening on stock-topic...")
for msg in consumer:
    data = msg.value
    data['_saved_at'] = datetime.utcnow().isoformat()

    result = collection.insert_one(data)
    saved += 1

    sym   = data.get('symbol', '?')
    price = data.get('price', data.get('Close', '?'))
    print(f"💾 [{sym}] ₹{price} → saved #{saved}")

Verify in Atlas Dashboard

After running consumer_mongodb.py, check your data in the Atlas web UI.

1. In Atlas → click Browse Collections
2. Navigate to sda_course → stock_prices
3. You'll see all saved JSON documents with symbol, price, _saved_at

Query from Python

# Count documents per symbol
from pymongo import MongoClient
client = MongoClient(MONGO_URI)
col    = client["sda_course"]["stock_prices"]

print("Total saved:", col.count_documents({}))
print("TCS records:", col.count_documents({"symbol": "TCS.NS"}))
print("INFY records:", col.count_documents({"symbol": "INFY.NS"}))

# Get latest 5 TCS prices
for doc in col.find({"symbol":"TCS.NS"}).sort("_id", -1).limit(5):
    print(doc["symbol"], doc.get("price", doc.get("Close")))

Key Takeaways

• Kafka → MongoDB is the simplest path from live stream to persistent storage
• MongoDB Atlas M0 is free — 512 MB, cloud-hosted, no server to manage
• Every message becomes a document — JSON in Kafka = JSON in MongoDB, no transformation needed
• Add metadata before saving — _saved_at, _kafka_offset make debugging much easier
• Query with Python — count_documents, find, sort, limit are your core tools

Download Lab Files

Next → Lecture 9: Build a live analytics dashboard from MongoDB data using Streamlit