Multi-Producer · Kafka · MongoDB · Enrichment · 3 Fraud Rules · Live Alerts
| Time | What We Do | Format |
|---|---|---|
| 0 – 10 | Module 2 recap — full path from L6 → L9 | Discussion |
| 10 – 20 | Why fraud detection? Architecture overview | Lecture |
| 20 – 30 | The dataset — 2000 synthetic transactions, 3 fraud patterns planted | Demo |
| 30 – 45 | Step 1 — Generate data + Run producer | Live build |
| 45 – 60 | Step 2 — consumer_store.py (save + user profiles) | Live build |
| 60 – 80 | Step 3 — consumer_fraud.py (3 rules + MongoDB enrichment) | Live build |
| 80 – 90 | See fraud alerts fire, discuss extensions, module wrap-up | Discussion |
Every card swipe triggers a real-time decision. Banks have milliseconds — not hours — to approve or hold.
| Batch System (Old) | Streaming System (Now) |
|---|---|
| Check transactions overnight | Check every transaction as it arrives |
| Fraud flagged next morning | Fraud flagged before transaction clears |
| Customer already abroad, card maxed | Transaction held, customer called instantly |
| No geographic context | Knows where card was used 3 minutes ago |
The pattern we use today: Kafka brings the current signal (this transaction). MongoDB holds the historical context (last known location, average spend). The consumer combines both to decide.
| Network | Peak Transactions/sec |
|---|---|
| Visa | 65,000 |
| Mastercard | 50,000+ |
| NPCI / UPI | 10,000+ |
None of these run on batch. All use streaming architectures similar to what you're building today.
Key insight: consumer_store.py runs first — it builds the historical record that consumer_fraud.py relies on. Storage and enrichment are inseparable.
Run generate_transactions.py to create transactions.csv — 2000 rows, 50 fake users, 8 Indian cities.
| Field | Example | Notes |
|---|---|---|
transaction_id | TXN-00847 | Unique per row |
user_id | USR-042 | 50 synthetic users |
user_name | Bhavna Thakur | Realistic Indian names |
amount | 4500.00 | INR, varies by merchant type |
merchant | Swiggy | 30 merchant types |
city / lat / lon | Mumbai / 19.076 / 72.877 | Real GPS coordinates |
timestamp | 2025-08-01 10:00:00 | Spread over 30 days |
python generate_transactions.py
# ✅ Generated 2000 transactions → transactions.csv
# Legitimate: 1903 | Fraud planted: 97
python transaction_producer.pyYou'll see the live feed:
Notice: The producer marks fraud rows with 🚨 in its own output — but the consumer doesn't know this yet. It's discovering fraud independently from Kafka + MongoDB. Leave this running and open Terminal 2.
Two MongoDB writes on every message:
# 1. Append raw transaction to audit log
txns.insert_one({**txn, "ingested_at": datetime.utcnow()})
# 2. Upsert user_profiles — freshest location + running totals
profiles.update_one(
{"user_id": txn["user_id"]},
{
"$set": {"last_city": txn["city"], "last_lat": lat, "last_lon": lon, ...},
"$inc": {"total_spend": amount, "txn_count": 1}
},
upsert=True
)upsert=True means| Situation | Behaviour |
|---|---|
| User seen for the first time | Creates a new document in user_profiles |
| User seen before | Updates only the specified fields — no duplicate |
Check Atlas: After 30 seconds, open MongoDB Atlas → Browse Collections → sda_course. You'll see user_profiles filling up with one document per user, each showing their latest city and cumulative spend.
For every Kafka message, query MongoDB for the user's history — then apply rules:
last_lat/lon (MongoDB) to current > 500 km AND time diff < 60 mintransactions collection by timestampAll three rules need MongoDB. None of them can fire on Kafka data alone — they need context that only the storage consumer has been building.
IMPOSSIBLE_TRAVEL_KM = 100. What new alerts fire? Are any of them false positives?