A Kafka-inspired distributed log and messaging system written from scratch in C++17.
Andromeda is a serious systems programming project that implements the core ideas behind Apache Kafka: append-only partitioned logs, a binary wire protocol, simplified Raft consensus, durable consumer-group offsets, and safe log-segment retention. It is designed to be read, run, and explained in a systems design or distributed systems interview.
┌──────────────────────────────────────────────────────────────────────────┐
│ Producer / Consumer client │
│ (andromeda-cli or libandromeda_client) │
└──────────────────┬────────────────────────────────┬─────────────────────┘
│ TCP — binary framed protocol │
┌──────────▼────────────────────────────────▼──────────┐
│ BrokerServer (Boost.Asio) │
│ async acceptor → Session per connection │
│ │
│ ┌─────────────────────────┐ ┌───────────────────┐ │
│ │ TopicManager │ │ConsumerGroupManager│ │
│ │ FNV-1a key routing │ │ per-group .offsets │ │
│ │ round-robin fallback │ │ file on disk │ │
│ └────────────┬────────────┘ └───────────────────┘ │
│ │ │
│ ┌────────────▼────────────┐ ┌───────────────────┐ │
│ │ PartitionLog │ │ MetadataStore │ │
│ │ ┌────────────────────┐ │ │ metadata.dat │ │
│ │ │ LogSegment │ │ └───────────────────┘ │
│ │ │ <offset>.log │ │ │
│ │ │ <offset>.idx │ │ │
│ │ └────────────────────┘ │ │
│ └─────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────┐ │
│ │ RaftNode │ │
│ │ Follower → Candidate → Leader state machine │ │
│ │ election timer │ heartbeat timer │ │
│ │ ┌──────────────────────────────────────┐ │ │
│ │ │ RaftLog (raft.log + raft.meta) │ │ │
│ │ └──────────────────────────────────────┘ │ │
│ │ AppendEntries RPC ◄──────► peer brokers │ │
│ └────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────┘
Client → ProduceRequest → BrokerServer::Session
Session: is this node the Raft leader?
NO → ProduceResponse(NotLeader)
YES → RaftNode::propose(serialised command)
→ append to local RaftLog
→ AppendEntries RPC to each peer (detached threads)
→ majority of peers ack → advance commit_index
→ apply_callback fires
→ TopicManager::append → PartitionLog::append (WAL write)
→ ProduceResponse(partition, offset) → Client
Client → FetchRequest(topic, partition, start_offset, max)
TopicManager::read
→ locate segment whose base_offset ≤ start_offset
→ binary search using sparse .idx file
→ scan .log records, verify CRC-32 per record
→ FetchResponse(messages) → Client
| Subsystem | Detail |
|---|---|
| Broker TCP server | Async accept loop (Boost.Asio), one Session object per connection, frame-level async read/write |
| Binary wire protocol | 10-byte magic-framed header (ANDR + type + length), big-endian payload encoding, separate message types for all client and Raft RPCs |
| Topics and partitions | Create topics with N partitions; deterministic FNV-1a key routing; round-robin for keyless messages |
| Write-Ahead Log | Append-only .log segments with paired .idx sparse offset index; CRC-32 per record; crash recovery by replaying segments on startup |
| Monotonic offsets | Each partition maintains a strictly increasing next_offset; offsets are assigned only after a durable WAL write |
| Raft consensus | Follower/Candidate/Leader state machine; randomised election timeout; RequestVote and AppendEntries RPCs; majority-based commit index; persisted hard state (term, votedFor) in raft.meta |
| Single-node fast path | A broker with no configured peers immediately elects itself leader; proposed entries are committed without waiting for replication |
| Consumer group offsets | Per-group offset files on disk (<group>.offsets); survives broker crashes; commit-offset / fetch-offset CLI commands |
| Log segment retention | Old segments are deleted only when every byte they contain is below both the Raft commit watermark and every consumer group's committed offset — broken offset correctness is impossible by construction |
| Metadata persistence | Topic configurations (name, partition count, replication factor) stored in metadata.dat; recovered on restart |
| CLI client | create-topic, produce, consume, commit-offset, fetch-offset, inspect |
| Test suite | 44 Google Test cases across 6 files covering routing, WAL, offsets, consumer offsets, retention safety, and Raft log + election |
| Component | Choice | Why |
|---|---|---|
| Language | C++17 | std::filesystem, std::optional, structured bindings, no raw pointers |
| Networking | Boost.Asio (header-only) | Industry-standard async I/O; proactor pattern for the broker accept loop |
| Build | CMake 3.16+ | Modern target-based build graph; FetchContent for Google Test |
| Testing | Google Test 1.14 | TEST_F fixtures, gtest_discover_tests for CTest integration |
| Serialization | Hand-written big-endian encoder/decoder | Zero dependencies; easy to trace in a debugger |
| Hashing | FNV-1a 32-bit | Deterministic, stateless, O(key length) — the same algorithm Kafka's default partitioner used before murmur2 |
| Checksums | IEEE CRC-32 (table-driven) | Detects single-bit storage errors; same role as Kafka's record-level CRC |
andromeda/
├── CMakeLists.txt
├── README.md
├── config/
│ ├── broker0.conf 3-node cluster, node 0 (port 9092)
│ ├── broker1.conf 3-node cluster, node 1 (port 9093)
│ ├── broker2.conf 3-node cluster, node 2 (port 9094)
│ └── broker_standalone.conf Single-node reference config
├── scripts/
│ ├── start_cluster.sh Launch all 3 nodes locally
│ ├── demo.sh End-to-end CLI walkthrough (cluster)
│ └── demo_single_node.sh End-to-end CLI walkthrough (single-node)
├── include/andromeda/
│ ├── common/
│ │ ├── types.hpp Offset, Term, Index, TopicPartition, …
│ │ ├── logger.hpp Thread-safe timestamped logger (macros)
│ │ ├── crc32.hpp IEEE CRC-32 declaration
│ │ └── serializer.hpp Encoder / Decoder (big-endian binary)
│ ├── protocol/
│ │ ├── message_types.hpp All request/response structs + Raft RPCs
│ │ └── codec.hpp Frame builder / parser
│ ├── storage/
│ │ ├── log_record.hpp On-disk record layout (fixed + variable)
│ │ ├── log_segment.hpp Single .log / .idx segment pair
│ │ └── partition_log.hpp Ordered segment collection for one partition
│ ├── raft/
│ │ ├── raft_types.hpp RaftLogEntry, PeerAddress, ApplyCallback
│ │ ├── raft_log.hpp Persistent log + hard state (raft.log / .meta)
│ │ └── raft_node.hpp State machine declaration
│ ├── metadata/
│ │ └── metadata_store.hpp Topic config persistence (metadata.dat)
│ ├── consumer/
│ │ └── consumer_group_manager.hpp Group offset files
│ ├── broker/
│ │ ├── topic_manager.hpp Owns PartitionLog instances, routing logic
│ │ └── broker_server.hpp TCP server, Session, RaftNode wiring
│ └── client/
│ ├── producer.hpp
│ └── consumer.hpp
├── src/ Implementations (one .cpp per header)
└── tests/
├── test_routing.cpp Key routing determinism + single-node produce
├── test_wal.cpp Segment append, read, recovery
├── test_offsets.cpp Monotonic offsets, segment roll continuity
├── test_consumer_offsets.cpp Group offset persistence and isolation
├── test_retention.cpp Retention safety — consumer lag blocks deletes
└── test_raft.cpp RaftLog persistence + single-node election
| Dependency | Version | Notes |
|---|---|---|
| C++17 compiler | Clang 13+ / GCC 11+ | std::filesystem must be available |
| Boost | ≥ 1.74 | Header-only; Boost::headers target only — no compiled Boost libraries required |
| CMake | ≥ 3.16 | |
| Google Test | 1.14 | Auto-downloaded via FetchContent at configure time |
Install Boost on macOS:
brew install boostOn Ubuntu / Debian:
sudo apt install libboost-dev# 1. Clone and enter the repo
git clone <url> andromeda && cd andromeda
# 2. Configure (downloads Google Test automatically)
cmake -B build -DCMAKE_BUILD_TYPE=Release
# 3. Build all targets
cmake --build build --parallel
# Produced binaries
build/bin/andromeda-broker
build/bin/andromeda-cli# Run all tests with CTest
ctest --test-dir build --output-on-failure
# Run a specific suite directly
./build/tests/test_routing
./build/tests/test_wal
./build/tests/test_offsets
./build/tests/test_consumer_offsets
./build/tests/test_retention
./build/tests/test_raftAll 44 tests pass. The Raft suite includes single-node election (verifies a zero-peer node elects itself within the election timeout) and single-node propose-and-apply (verifies a committed entry fires the apply callback).
# No arguments — starts on port 9092, elects itself Raft leader in ~150 ms
./build/bin/andromeda-brokerYou will see a log line confirming leadership:
[02:30:01.203] INFO [broker-0] RaftNode: broker-0 → Leader term=1
With the broker running, open a second terminal and run the bundled walkthrough script. It creates a topic, produces messages, consumes from every partition, and exercises consumer group offsets:
./scripts/demo_single_node.sh # defaults to 127.0.0.1:9092
./scripts/demo_single_node.sh 127.0.0.1 9092 # explicit host and portOpen a second terminal. All commands follow:
andromeda-cli <host> <port> <command> [args...]
CLI="./build/bin/andromeda-cli 127.0.0.1 9092"
# Create a topic with 3 partitions
$CLI create-topic orders 3 1
# Produce messages — the key is hashed to select a partition deterministically
$CLI produce orders user-123 '{"action":"login","user":123}'
$CLI produce orders order-456 '{"action":"checkout","total":42}'
$CLI produce orders user-123 '{"action":"logout","user":123}'
# Consume from each partition (topic, partition, start_offset, max_count)
$CLI consume orders 0 0 50
$CLI consume orders 1 0 50
$CLI consume orders 2 0 50
# Inspect a partition — shows all message offsets and sizes
$CLI inspect orders 0
# Commit a consumer group offset (group, topic, partition, next_offset_to_read)
$CLI commit-offset my-app orders 0 2
# Fetch the last committed offset for the group
$CLI fetch-offset my-app orders 0./build/bin/andromeda-broker config/broker0.conf # port 9092
./build/bin/andromeda-broker config/broker1.conf # port 9093
./build/bin/andromeda-broker config/broker2.conf # port 9094
# Or use the convenience script:
./scripts/start_cluster.sh buildWait ~300 ms for election. Send all commands to the elected leader's port.
A 10-byte header (ANDR magic, 2-byte type, 4-byte length) lets the session
read exactly two async_read calls per request, with no scanning for
delimiters. The payload is big-endian binary, which is compact and
unambiguous.
Storing one index entry per 4 KB of log data means seeking to an arbitrary
offset costs at most O(segment_size / 4096) index entries to scan, while
the index itself stays tiny. This mirrors how Kafka's IndexFile works.
Corruption detection on every record means recovery stops at the first bad checksum rather than silently replaying garbage, which is the same guarantee Kafka provides via its record-level CRC.
Deterministic, zero-dependency, O(key length). Every client and broker
computes the same partition without coordination. The same idea underlies
Kafka's older DefaultPartitioner.
Leader election, log replication, and majority-based commit are all implemented correctly. Membership changes and Raft-level snapshots are omitted. The WAL's segment retention covers the same practical need as snapshotting for this use case.
Each AppendEntries and RequestVote call opens a fresh connection to the
peer, sends the frame, receives the response, and closes. This is simple and
correct. A production implementation would use persistent channels and
pipelining.
All Raft state mutations are posted onto an io_context strand — a
serialised executor — so the state machine is single-threaded by design
while peer RPC work runs on detached threads that post results back onto the
strand. This eliminates data races without a coarse lock on the hot path.
A log segment is only eligible for deletion when its highest offset is below both the Raft commit index and the minimum committed offset across all consumer groups. This makes it structurally impossible to delete a segment that a consumer still needs.
-
No client-side leader redirect. A produce request sent to a follower receives
NotLeader. The client must retry against the leader manually. A production broker would return the leader's address in the error response. -
No Raft log compaction. The
raft.logfile grows without bound. In a long-running cluster the leader would eventually need to snapshot the applied state and truncate old Raft entries. -
No batching or pipelining in the client. Each produce call is a synchronous round-trip. Kafka clients batch many records into a single request.
-
No TLS. All broker-to-broker and client-to-broker traffic is plaintext.
-
Static cluster membership. Peers are configured at startup; there is no way to add or remove nodes without restarting the cluster.
-
No consumer group coordinator. Each consumer independently tracks and commits its own offset. Kafka's group coordinator protocol (partition assignment, rebalance, heartbeat) is not implemented.
-
Single active segment per partition. The segment roll threshold is configurable but there is no time-based roll or background compaction.
- Leader-address redirect in
NotLeaderresponses - Persistent connection pool for inter-broker Raft RPCs
- Raft log compaction / state machine snapshot
- Client-side batching and async produce
- Consumer group coordinator with partition assignment
- TLS for client and inter-broker channels
- Prometheus-compatible metrics endpoint
- Time-based log segment rolling