Hierarchical Topics

[shortishly]

Hierarchical Topics in Tansu

Kafka topics are great at fan-out: produce once, consume many times from any offset. But one thing they do not give you for free is per-key subscriptions. If you are streaming telemetry from a fleet of vehicles — each identified by its license plate — and a consumer only cares about one vehicle, it has to read the whole topic and discard everything that does not match. Hierarchical topics let you skip that work entirely.

The problem

Consider a telemetry topic where every message is keyed by the vehicle's license plate:

{"key": "CC54 RYD",
 "value": {"latitude": 52.79954588398599,
           "longitude": -4.097797077141778,
           "altitude": 163.17052057678418}}

{"key": "NN03 RYB",
 "value": {"latitude": 50.84287000432428,
           "longitude": -4.25542715891738,
           "altitude": 41.08461912439403}}

A dashboard that tracks CC54 RYD alone still has to page through every NN03 RYB, AB12 XYZ, and every other plate in the fleet. For a large fleet with high message rates that is a lot of wasted I/O.

Hierarchical Topics

Tansu introduces hierarchical topics — a way to address the subset of a topic that belongs to a single key using a derived topic name of the form topic/key:

Fetch target	Records returned
telemetry	All records from all vehicles
telemetry/CC54 RYD	Only records where key = CC54 RYD
telemetry/NN03 RYB	Only records where key = NN03 RYB

The hierarchical topic is not a real topic. No data is duplicated, no additional partitions are created. The SQL-based storage backends (PostgreSQL and libSQL) add a single AND r.k = $7 predicate to the existing fetch query, so the database does the filtering at the source.

Enabling the feature

Hierarchical topics are opt-in per topic. Set the tansu.virtual configuration key to true when creating the topic:

tansu topic create telemetry --config tansu.virtual=true 

Once the flag is set, any consumer can open a fetch against telemetry/CC54 RYD using the standard Kafka protocol — no client-side changes required.

What happens without the flag

Topics that do not carry tansu.virtual=true are completely unaffected. A topic name that happens to contain a / is treated as a literal name; the slash carries no special meaning. This means existing topics are safe, and you can still create topics named a/b if your naming convention requires it — they will simply stream all their records when fetched.

How it works

On the PostgreSQL backend the fetch query is extended with an extra predicate when a virtual key is detected:

-- Normal fetch
WHERE c.name = $1
  AND t.name = $2         -- base topic name
  AND tp.partition = $3
  AND r.offset_id >= $4
  AND r.offset_id < $6

-- Virtual keyed fetch (tansu.virtual = true, topic name = "base/key")
WHERE c.name = $1
  AND t.name = $2         -- base topic name ("telemetry")
  AND tp.partition = $3
  AND r.offset_id >= $4
  AND r.offset_id < $6
  AND r.k = $7            -- key bytes ("CC54 RYD")

The libSQL backend uses an identical extension to its own fetch query.

At request time the broker:

1. Splits the topic name on the first /.
2. Calls describe_config on the base topic to check for tansu.virtual=true.
3. If enabled, issues the keyed query with the UTF-8 bytes of the key suffix
   as $7 and resolves offsets and watermarks against the base topic.
4. If disabled (or the config is absent), the original topic name is used
   unchanged — business as usual.

The O(1) config lookup is cached at the storage layer, so the overhead on each fetch is negligible.

Offset semantics

Offsets in a hierarchical topic are the same offsets as in the base topic. This is intentional: if you need to re-read from a known point you use the offset you last committed against the base topic. Compaction, retention, and watermarks are all governed by the base topic's configuration.

Example: vehicle telemetry dashboard

Produce telemetry as usual:

kafka-console-producer.sh \
  --topic telemetry \
  --property "parse.key=true" \
  --property "key.separator=:" \
  --bootstrap-server localhost:9092
> CC54 RYD:{"latitude":52.799,"longitude":-4.097,"altitude":163.1}
> NN03 RYB:{"latitude":50.842,"longitude":-4.255,"altitude":41.08}

A consumer dashboard for a single vehicle subscribes to the hierarchical topic:

kafka-console-consumer.sh \
  --topic telemetry/CC54\ RYD \
  --from-beginning \
  --bootstrap-server localhost:9092

It receives only the CC54 RYD records, with zero application-level filtering and no impact on consumers of the base telemetry topic.

Availability

Hierarchical topics are supported on Tansu's SQL-backed storage engines:

• PostgreSQL (postgres://)
• libSQL / SQLite (sqlite://)

The in-memory and S3/object-store backends are unaffected; they continue to behave as standard Kafka-compatible topics.

The feature shipped in tansu-io/tansu#651, currently available on main. Planned to be part of the 0.7.0 release.

[omersiar]

Definitely a good idea/addition, I started to think, in vanilla Kafka segments could be keyed, but this might explode metadata on ram