Merchant Collection — Kafka + Docker + TypeScript

Stack

Layer	Choice	Why
Broker	Kafka (KRaft, no ZooKeeper)	Simpler Docker setup
DB	PostgreSQL (not MySQL)	JSONB, better write throughput, LISTEN/NOTIFY
Cache	Redis	Merchant account lookup + dedup
Stream	Server-Sent Events	One-way, works through proxies, native browser support
Runtime	Node.js + TypeScript	kafkajs, ioredis, pg

Why PostgreSQL over MySQL?
JSONB columns let you store raw transaction payloads without rigid schema migration.
pg's LISTEN/NOTIFY can optionally trigger SSE without polling.
Better concurrent write performance for high-volume inserts.

---

Project Structure

merchant-collection/
├── docker-compose.yml
├── package.json
├── tsconfig.json
├── .env.example            # Example environment variables
├── Dockerfile              # Container definition
├── src/
│   ├── config.ts           # env + kafka/pg/redis clients
│   ├── producer.ts         # simulates external party
│   ├── consumer.ts         # filter, enrich, persist
│   └── sse-gateway.ts      # Express SSE endpoint
└── init.sql                # DB schema

---

Running It

# 0. Setup Environment
cp .env.example .env

# 1. Start infrastructure (consumer + gateway)
docker-compose up -d

# 2. Scale the consumer (optional)
# This runs 3 instances of consumer.ts, load-balancing the 3 partitions
docker-compose up -d --scale consumer=3

# 3. Install local deps (for producer/testing)
npm install
# or
pnpm install

# 4. Access SSE Gateway
# Open http://localhost:3000/merchants/merchant-001/stream in your browser

# 5. Simulate external party producing transactions
npx ts-node src/producer.ts