Banking Fraud Detection
End-to-End

Today's 90 Minutes

Time	What We Do	Format
0 – 10	Module 2 recap — full path from L6 → L9	Discussion
10 – 20	Why fraud detection? Architecture overview	Lecture
20 – 30	The dataset — 2,003 synthetic transactions, 3 fraud patterns planted	Demo
30 – 45	Step 1 — Generate data + Run producer	Live build
45 – 60	Step 2 — choose MongoDB or MySQL storage consumer	Live build
60 – 80	Step 3 — consumer_fraud.py (3 rules + MongoDB enrichment)	Live build
80 – 90	See fraud alerts fire, discuss extensions, module wrap-up	Discussion

Why Fraud Detection?

Every card swipe triggers a real-time decision. Banks have milliseconds — not hours — to approve or hold.

Batch System (Old)	Streaming System (Now)
Check transactions overnight	Check every transaction as it arrives
Fraud flagged next morning	Fraud flagged before transaction clears
Customer already abroad, card maxed	Transaction held, customer called instantly
No geographic context	Knows where card was used 3 minutes ago

Scale Reference

Network	Peak Transactions/sec
Visa	65,000
Mastercard	50,000+
NPCI / UPI	10,000+

None of these run on batch. All use streaming architectures similar to what you're building today.

What We Build Today

The Synthetic Dataset

Run generate_transactions.py to create transactions.csv — 2,003 rows, 50 fake users, 8 Indian cities.

Field	Example	Notes
transaction_id	TXN-00847	Unique per row
user_id	USR-042	50 synthetic users
user_name	Bhavna Thakur	Realistic Indian names
amount	4500.00	INR, varies by merchant type
merchant	Swiggy	30 merchant types
city / lat / lon	Mumbai / 19.076 / 72.877	Real GPS coordinates
timestamp	2025-08-01 10:00:00	Spread over 30 days

Planted Fraud Patterns (~5% of rows)

Generate Data & Run Producer

python generate_transactions.py
# ✅ Generated 2003 transactions → transactions.csv
#    Legitimate: 1927  |  Fraud planted: 76
#    Next run continues after the displayed ID range

python transaction_producer.py

You'll see the live feed:

Storage Consumer — Choose a Destination

The consumer can write the same Kafka stream to either destination:

Option	Use It When	Consumer
MongoDB	You want to run the three fraud rules using user profiles and recent history	consumer_store.py
MySQL	You want SQL analysis and a live data source for the Lecture 14–16 Grafana dashboards	consumer_store_mysql.py

Option A — MongoDB: Persist and Profile

Two MongoDB writes occur on every message:

# 1. Append raw transaction to audit log
txns.insert_one({**txn, "ingested_at": datetime.utcnow()})

# 2. Upsert user_profiles — freshest location + running totals
profiles.update_one(
    {"user_id": txn["user_id"]},
    {
        "$set": {"last_city": txn["city"], "last_lat": lat, "last_lon": lon, ...},
        "$inc": {"total_spend": amount, "txn_count": 1}
    },
    upsert=True
)

What upsert=True means

Situation	Behaviour
User seen for the first time	Creates a new document in user_profiles
User seen before	Updates only the specified fields — no duplicate

Option B — MySQL: Persist for SQL and Grafana

Start the free MySQL container used in Lecture 14, install the connector, and run the alternative consumer:

docker run -d \
  --name sda-mysql \
  -e MYSQL_ROOT_PASSWORD=admin \
  -p 3306:3306 \
  mysql:8.4

pip install kafka-python mysql-connector-python
python consumer_store_mysql.py

The script creates sda_course.transactions, converts numeric and date fields, and writes one row after each Kafka message. Its upsert protects against duplicate rows if Kafka replays a transaction.

# Save first, then commit the Kafka offset
cursor.execute(UPSERT_SQL, transaction_values(txn))
db.commit()
consumer.commit()

Option C — Persist Alert Events in MySQL

For the live Grafana extension, open another terminal and run a second consumer group. It reads the same topic, keeps only flagged transactions, classifies the classroom fraud pattern, and writes each alert to sda_course.fraud_alerts.

python consumer_alerts_mysql.py

# Check the live alert table
docker exec -it sda-mysql mysql -uroot -padmin \
  -e "SELECT alert_type, severity, COUNT(*)
      FROM sda_course.fraud_alerts
      GROUP BY alert_type, severity;"

The mysql-store-group and mysql-alert-group are independent. Each receives every transaction: one builds the transaction table; the other builds the operational alert queue.

consumer_fraud.py — 3 Fraud Rules

For every Kafka message, query MongoDB for the user's history — then apply rules:

What Good Output Looks Like

Legitimate transactions

Impossible Travel alert

Discussion Questions

1. Change IMPOSSIBLE_TRAVEL_KM = 100. What new alerts fire? Are any of them false positives?
2. A customer genuinely flies Mumbai → Dubai (2,200 km, 3.5 hr flight). Your rule would have blocked their arrival transaction. How do you solve this?
3. Rule 2 requires 5+ past transactions. A customer's first big purchase is always suspicious. What would you do for new users?

Module 2 Complete

• ✓Enrich → Decide → Act is the core pattern — Kafka brings now, MongoDB brings history, the consumer decides
• ✓Two consumers, one topic — store and fraud run simultaneously, each getting a full independent copy of every message
• ✓upsert is your state store — streaming systems need memory; MongoDB + upsert gives you that without extra infrastructure
• ✓Fraud rules are business rules — 500 km, 3× average, 60 seconds are management decisions, not engineering ones
• ✓Module 3 preview: when these hand-written consumers are not enough (scale, windowing, joins) — Spark & Flink take over

Downloads