keep

A Redis-like in-memory key-value database written in Go from scratch. No frameworks, just the standard library plus the Prometheus client. Built as a deep-dive into database internals: concurrency, wire protocols, write-ahead logging, snapshots, and cache eviction.

$ nc localhost 6380
SET name felix
OK
GET name
felix
RPUSH tasks deploy
1
EXPIRE name 60
1
TTL name
58

Features

• String and list values: GET/SET/DEL, plus deque-style lists (LPUSH, RPUSH, LPOP, RPOP, LRANGE with negative indexes)
• Concurrent: goroutine per connection, RWMutex-guarded store, fully race-detector tested
• TTL expiration: lazy check on read plus an active background sweeper (the same two-tier design Redis uses)
• Crash-safe: write-ahead log records every mutation before it is applied; survives kill -9
• Snapshots: periodic atomic JSON snapshots truncate the WAL, keeping recovery time bounded
• Graceful shutdown: SIGINT/SIGTERM triggers a final snapshot and clean exit
• LRU eviction: capped keyspace (default 10,000 keys) with O(1) least-recently-used eviction
• Auth: optional shared-secret AUTH with constant-time comparison
• Observability: Prometheus metrics and pprof profiling on :9100
• Ships anywhere: multi-stage Dockerfile producing a ~20MB distroless image; CI runs go vet and go test -race on every push

Benchmark: keep vs Redis

Measured with the included load generator (cmd/bench) using the same client, same machine (Apple Silicon, macOS), and same workload: 200,000 requests, 50 concurrent connections, 50/50 SET/GET mix, strict request-reply (no pipelining).

	keep	Redis 8
Throughput	90,352 ops/sec	106,857 ops/sec
Latency p50	539µs	452µs
Latency p99	932µs	880µs
Latency max	2.08ms	2.94ms

Roughly 85% of Redis throughput, while writing every SET to the WAL (Redis's default config does no per-write disk logging). CPU profiling shows 84% of time in network syscalls; the store, locks, and WAL are not the bottleneck at this load.

Reproduce it:

go run ./cmd/server                                   # terminal 1
go run ./cmd/bench -clients 50 -requests 200000       # terminal 2

# against Redis (it accepts inline commands; -redis switches reply parsing to RESP)
go run ./cmd/bench -addr localhost:6379 -redis -clients 50 -requests 200000

Architecture

flowchart TB
    C1[client] <--> H1
    C2[client] <--> H2
    subgraph keep process
        A[accept loop] -.spawns.-> H1[conn goroutine]
        A -.spawns.-> H2[conn goroutine]
        H1 --> P[parser]
        H2 --> P
        P --> D[dispatcher]
        D --> S[(in-memory store<br/>strings · lists · TTL · LRU)]
        SW[sweeper goroutine] --> S
        SN[snapshotter goroutine] --> S
    end
    D -->|before every write| W[wal.log]
    SN -->|atomic write + truncate WAL| F[snapshot.json]

Loading

• One goroutine per connection; the accept loop never blocks on a client.
• Parsing, dispatch, and storage are separate packages with one-way dependencies. The parser and dispatcher are pure (no I/O), which is what makes WAL replay trivial: recovery feeds logged lines through the exact same parse and execute path a live client uses.
• Log-before-do: a mutation is appended to the WAL before it touches memory, so the log never knows less than the store. A single mutex serializes append and apply so replay order always matches execution order.
• Recovery loads the latest snapshot, then replays the WAL tail. Snapshots are written to a temp file and atomically renamed, so a crash mid-snapshot can never corrupt the previous good one.
• Eviction is the classic map plus doubly-linked-list LRU: O(1) lookup, O(1) recency update, O(1) eviction. Reads count as usage.

Commands

Command	Reply	Notes
SET key value	OK	overwrites any type, clears TTL
GET key	value or (nil)	ERR wrong type on a list
DEL key	OK
LPUSH key value / RPUSH key value	new length	creates the list if absent
LPOP key / RPOP key	value or (nil)	popping the last element deletes the key
LRANGE key start stop	elements	negative indexes count from the end; 0 -1 = all
EXPIRE key seconds	1 or 0
TTL key	seconds, -1 no expiry, -2 missing
AUTH password	OK	required first when KEEP_PASSWORD is set

Running

go run ./cmd/server                       # listens on :6380, metrics on :9100
KEEP_PASSWORD=secret go run ./cmd/server  # with auth enabled
nc localhost 6380                         # talk to it

Data lives in data/ (wal.log and snapshot.json), created automatically.

Docker

docker build -t keep .
docker run -e KEEP_PASSWORD=secret -p 6380:6380 -p 9100:9100 keep

Metrics and profiling

curl localhost:9100/metrics                     # keep_commands_total, keep_keys,
                                                # keep_connected_clients, latency histogram
go tool pprof "http://localhost:9100/debug/pprof/profile?seconds=10"

Tests

go vet ./...
go test -race ./...

Every package has table-driven tests; the store's concurrency, expiry, eviction, and snapshot behavior are all covered, and CI enforces the race detector on every push.

Project structure

cmd/server        entrypoint: recovery, background workers, signal handling
cmd/bench         load generator (also speaks to Redis via inline commands)
internal/store    map + RWMutex, lists, TTL, LRU, snapshots
internal/protocol line to Command parsing, arity table, pure functions
internal/server   TCP accept loop, auth gate, dispatcher, WAL coordination
internal/wal      append-only log: open, append, truncate, replay
internal/metrics  Prometheus registry + /metrics + pprof

Known limitations (deliberate)

• Values cannot contain spaces (whitespace-delimited protocol; RESP support would fix this)
• A replayed EXPIRE restarts its countdown rather than keeping the original deadline (Redis solves this by rewriting to absolute timestamps)
• WAL writes are not fsynced, so data survives process death but not power loss (a config dial in real databases)
• Single-node only: no replication, clustering, or pub/sub