From e3e8dcb8b6e71f995aba55c90687dcb3c0732b44 Mon Sep 17 00:00:00 2001 From: Paul Kuruvilla Date: Wed, 9 Jul 2025 16:49:42 -0700 Subject: [PATCH 01/52] Update stage description for Fetch API and correct MaxVersion details. --- .../consuming-messages-01-gs0.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/stage_descriptions/consuming-messages-01-gs0.md b/stage_descriptions/consuming-messages-01-gs0.md index 20403af..7101d86 100644 --- a/stage_descriptions/consuming-messages-01-gs0.md +++ b/stage_descriptions/consuming-messages-01-gs0.md @@ -1,10 +1,15 @@ In this stage, you'll add an entry for the `Fetch` API to the APIVersions response. -🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. +### The Fetch API -In the meantime, please use -[this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+gs0%3A+Include+Fetch+in+APIVersions&body=%3Cyour+question+here%3E) -to ask questions on the forum. +The [Fetch API](https://kafka.apache.org/protocol#The_Messages_Fetch) (API key `1`) is used to fetch messages from a Kafka topic. + +We've created an interactive protocol inspector for the request & response structures for `Fetch`: + +- 🔎 [Fetch Request (v17)](https://binspec.org/kafka-fetch-request-v17) +- 🔎 [Fetch Response (v17)](https://binspec.org/kafka-fetch-response-v17) + +In this stage, you'll only need to add an entry for the `Fetch` API to the APIVersions response you implemented in earlier stages. We'll get to responding to `Fetch` requests in later stages. ### Tests @@ -22,10 +27,10 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (No Error). - The response body contains at least one entry for the API key `1` (FETCH). -- The `MaxVersion` for the Fetch API is atleast 16. +- The `MaxVersion` for the Fetch API is atleast 17. ### Notes -- You don't have to implement support for the `Fetch` request in this stage. We'll get to this in later stages. -- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. -- The `MaxVersion` for the `Fetch` and `APIVersions` are different. For `APIVersions`, it is 4. For `Fetch`, it is 16. +- You don't have to implement support for handling `Fetch` requests in this stage. We'll get to this in later stages. +- You'll still need to include the entry for `APIVersions` in your response to pass earlier stages. +- The `MaxVersion` for the `Fetch` and `APIVersions` are different. For `APIVersions`, it is 4. For `Fetch`, it is 17. From 60802cf7c72ab4b0f9089ce7e1dd61a3805da2d6 Mon Sep 17 00:00:00 2001 From: Paul Kuruvilla Date: Wed, 9 Jul 2025 16:54:39 -0700 Subject: [PATCH 02/52] Update Fetch API version references from v17 to v16 in stage descriptions. --- stage_descriptions/consuming-messages-01-gs0.md | 8 ++++---- stage_descriptions/consuming-messages-02-dh6.md | 17 +++++++++++------ 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/stage_descriptions/consuming-messages-01-gs0.md b/stage_descriptions/consuming-messages-01-gs0.md index 7101d86..a0f68b3 100644 --- a/stage_descriptions/consuming-messages-01-gs0.md +++ b/stage_descriptions/consuming-messages-01-gs0.md @@ -6,8 +6,8 @@ The [Fetch API](https://kafka.apache.org/protocol#The_Messages_Fetch) (API key ` We've created an interactive protocol inspector for the request & response structures for `Fetch`: -- 🔎 [Fetch Request (v17)](https://binspec.org/kafka-fetch-request-v17) -- 🔎 [Fetch Response (v17)](https://binspec.org/kafka-fetch-response-v17) +- 🔎 [Fetch Request (v16)](https://binspec.org/kafka-fetch-request-v16) +- 🔎 [Fetch Response (v16)](https://binspec.org/kafka-fetch-response-v16) In this stage, you'll only need to add an entry for the `Fetch` API to the APIVersions response you implemented in earlier stages. We'll get to responding to `Fetch` requests in later stages. @@ -27,10 +27,10 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (No Error). - The response body contains at least one entry for the API key `1` (FETCH). -- The `MaxVersion` for the Fetch API is atleast 17. +- The `MaxVersion` for the Fetch API is atleast 16. ### Notes - You don't have to implement support for handling `Fetch` requests in this stage. We'll get to this in later stages. - You'll still need to include the entry for `APIVersions` in your response to pass earlier stages. -- The `MaxVersion` for the `Fetch` and `APIVersions` are different. For `APIVersions`, it is 4. For `Fetch`, it is 17. +- The `MaxVersion` for the `Fetch` and `APIVersions` are different. For `APIVersions`, it is 4. For `Fetch`, it is 16. diff --git a/stage_descriptions/consuming-messages-02-dh6.md b/stage_descriptions/consuming-messages-02-dh6.md index b413300..d358007 100644 --- a/stage_descriptions/consuming-messages-02-dh6.md +++ b/stage_descriptions/consuming-messages-02-dh6.md @@ -1,10 +1,15 @@ -In this stage, you'll implement the Fetch response for a Fetch request with no topics. +In this stage, you'll implement the response for a `Fetch` request with no topics. -🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. +### Fetch API response for no topics -In the meantime, please use -[this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+dh6%3A+Fetch+with+no+topics&body=%3Cyour+question+here%3E) -to ask questions on the forum. +A `Fetch` request includes a list of topics to fetch messages from. If the request contains an empty list of topics, the `responses` field in the response will be an empty array. + +Here are interactive visualizations of what the `Fetch` request & response will look like when the request contains an empty list of topics: + +- 🔎 [Fetch Request (v17) with no topics](https://binspec.org/kafka-fetch-request-v17-no-topics) +- 🔎 [Fetch Response (v17) with no topics](https://binspec.org/kafka-fetch-response-v17-no-topics) + +In this stage, you'll need to implement the response for a `Fetch` request with an empty list of topics. We'll get to handling `Fetch` requests with topics in later stages. ### Tests @@ -36,4 +41,4 @@ The tester will validate that: - You don't need to parse the fields in the `Fetch` request in this stage, we'll get to this in later stages. - The official docs for the `Fetch` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Fetch). Make sure -to scroll down to the "Fetch Response (Version: 16)" section. \ No newline at end of file + to scroll down to the "Fetch Response (Version: 16)" section. From df48cbe094c62d1e0389c3df4c388ddbc899478c Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 18:05:48 +0530 Subject: [PATCH 03/52] feat: add stage descriptions for Produce API extension --- stage_descriptions/produce-01_um3.md | 28 ++++++++++++++++++ stage_descriptions/produce-02_ck2.md | 34 ++++++++++++++++++++++ stage_descriptions/produce-03_ho8.md | 33 +++++++++++++++++++++ stage_descriptions/produce-04_ps7.md | 36 +++++++++++++++++++++++ stage_descriptions/produce-05_sb8.md | 36 +++++++++++++++++++++++ stage_descriptions/produce-06_mf2.md | 40 ++++++++++++++++++++++++++ stage_descriptions/produce-07_ar4.md | 43 ++++++++++++++++++++++++++++ 7 files changed, 250 insertions(+) create mode 100644 stage_descriptions/produce-01_um3.md create mode 100644 stage_descriptions/produce-02_ck2.md create mode 100644 stage_descriptions/produce-03_ho8.md create mode 100644 stage_descriptions/produce-04_ps7.md create mode 100644 stage_descriptions/produce-05_sb8.md create mode 100644 stage_descriptions/produce-06_mf2.md create mode 100644 stage_descriptions/produce-07_ar4.md diff --git a/stage_descriptions/produce-01_um3.md b/stage_descriptions/produce-01_um3.md new file mode 100644 index 0000000..43942fc --- /dev/null +++ b/stage_descriptions/produce-01_um3.md @@ -0,0 +1,28 @@ +In this stage, you'll add an entry for the `Produce` API to the APIVersions response. + +## APIVersions + +Your Kafka implementation should include the Produce API (key=0) in the ApiVersions response before implementing produce functionality. This would let the client know that the broker supports the Produce API. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (No Error). +- The response body contains at least one entry for the API key `0` (Produce). +- The `MaxVersion` for the Produce API is at least 11. + +## Notes + +- You don't have to implement support for the `Produce` request in this stage. We'll get to this in later stages. +- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. diff --git a/stage_descriptions/produce-02_ck2.md b/stage_descriptions/produce-02_ck2.md new file mode 100644 index 0000000..40fb57b --- /dev/null +++ b/stage_descriptions/produce-02_ck2.md @@ -0,0 +1,34 @@ +In this stage, you'll add support for handling Produce requests to non-existent topics with proper error responses. + +## Handling Non-Existent Topics + +Your Kafka implementation should validate topic existence and return appropriate error codes when a client tries to produce to a non-existent topic. Kafka stores metadata about topics in the `__cluster_metadata` topic. To check if a topic exists or not, you'll need to read the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. If the topic exists, the topic will have a directory with all the data required for it to operate at `/-`. +TODO: Do we need to explain the TOPIC_RECORD record in the `__cluster_metadata` log. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send a `Produce` (v11) request to a non-existent topic. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response inside response should correspond to the topic name in the request. +- The `partition` field in the partition response inside topic response should correspond to the partition in the request. +- The `offset` field in the partition response inside topic response should be `-1`. +- The `timestamp` field in the partition response inside topic response should be `-1`. +- The `log start offset` field in the partition response inside topic response should be `-1`. + +## Notes + +- You'll need to parse the `Produce` request in this stage to get the topic name and partition to send in the response. +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. +- The official Kafka docs don't cover the structure of records inside the `__cluster_metadata` topic, but you can find the definitions in the Kafka source code [here](https://github.com/apache/kafka/tree/5b3027dfcbcb62d169d4b4421260226e620459af/metadata/src/main/resources/common/metadata). diff --git a/stage_descriptions/produce-03_ho8.md b/stage_descriptions/produce-03_ho8.md new file mode 100644 index 0000000..3a99ad4 --- /dev/null +++ b/stage_descriptions/produce-03_ho8.md @@ -0,0 +1,33 @@ +In this stage, you'll add support for handling Produce requests to invalid partitions for known topics with proper error responses. + +## Handling Invalid Partitions + +Your Kafka implementation should validate that partition indices exist for known topics and return appropriate errors for invalid partitions. Kafka stores metadata about partitions in the `__cluster_metadata` topic. To check if a partition exists or not, you'll need to read the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. If the partition exists, the partition will have a directory with all the data required for it to operate at `/-`. +TODO: Do we need to explain the PARTITION_RECORD record in the `__cluster_metadata` log. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send a `Produce` (v11) request to a known topic with an invalid partition. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response inside response should correspond to the topic name in the request. +- The `partition` field in the partition response inside topic response should correspond to the partition in the request. +- The `offset` field in the partition response inside topic response should be `-1`. +- The `timestamp` field in the partition response inside topic response should be `-1`. +- The `log start offset` field in the partition response inside topic response should be `-1`. + +## Notes + +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. +- The official Kafka docs don't cover the structure of records inside the `__cluster_metadata` topic, but you can find the definitions in the Kafka source code [here](https://github.com/apache/kafka/tree/5b3027dfcbcb62d169d4b4421260226e620459af/metadata/src/main/resources/common/metadata). diff --git a/stage_descriptions/produce-04_ps7.md b/stage_descriptions/produce-04_ps7.md new file mode 100644 index 0000000..1db0dec --- /dev/null +++ b/stage_descriptions/produce-04_ps7.md @@ -0,0 +1,36 @@ +In this stage, you'll add support for successfully producing a single record to a valid topic and partition. + +## Single Record Production + +Your Kafka implementation should accept valid Produce requests, store the record in the appropriate log file, and return successful responses with assigned offsets. The record must be persisted to the topic's log file at `/-/00000000000000000000.log` using Kafka's on-disk log format. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send multiple successive `Produce` (v11) requests to a valid topic and partition with single records as payload. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (NO_ERROR). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response matches the topic name in the request. +- The `partition` field in the partition response matches the partition in the request. +- The `offset` field in the partition response contains the assigned offset to the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) +- The `timestamp` field in the partition response contains -1 (signifying that the timestamp is latest). +- The `log_start_offset` field in the partition response is `0`. + +The tester will also verify that the record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. + +## Notes + +- You'll need to implement log file writing using Kafka's binary log format. +- Records must be stored in RecordBatch format with proper CRC validation. +- The offset assignment should start from 0 for new partitions. +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). diff --git a/stage_descriptions/produce-05_sb8.md b/stage_descriptions/produce-05_sb8.md new file mode 100644 index 0000000..c0ea97e --- /dev/null +++ b/stage_descriptions/produce-05_sb8.md @@ -0,0 +1,36 @@ +In this stage, you'll add support for producing multiple records from a single request with multiple record batches in the payload. + +## Batch Processing + +Your Kafka implementation should handle multiple records in a single batch, assign sequential offsets, and validate the LastOffsetDelta field correctly. The RecordBatch containing multiple records must be stored atomically to the log file. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send a `Produce` (v11) request containing a RecordBatch with multiple records. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (NO_ERROR). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response matches the topic name in the request. +- The `partition` field in the partition response matches the partition in the request. +- The `offset` field in the partition response contains the base offset for the batch. +- The `timestamp` field in the partition response contains a valid timestamp. +- The `log_start_offset` field in the partition response is correct. + +The tester will also verify that the record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log` with sequential offsets. + +## Notes + +- Records within a batch must be assigned sequential offsets (e.g., if base offset is 5, records get offsets 5, 6, 7). +- The entire batch is treated as a single atomic operation. +- The response should return the base offset of the batch, not individual record offsets. +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). diff --git a/stage_descriptions/produce-06_mf2.md b/stage_descriptions/produce-06_mf2.md new file mode 100644 index 0000000..5d4f778 --- /dev/null +++ b/stage_descriptions/produce-06_mf2.md @@ -0,0 +1,40 @@ +In this stage, you'll add support for producing to multiple partitions of the same topic in a single request. + +## Partition Routing + +Your Kafka implementation should handle writes to multiple partitions within the same topic, manage independent offset assignment per partition, and aggregate responses correctly. Each partition maintains its own offset sequence independently. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple partitions of the same topic. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (NO_ERROR). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response matches the topic name in the request. +- Each partition in the request has a corresponding partition response. +- Each partition response contains: + - The correct `partition` field matching the request. + - An error code of `0` (NO_ERROR). + - A valid `offset` field with the assigned offset for that partition. + - A valid `timestamp` field. + - A correct `log_start_offset` field. +- Records are persisted to the correct partition log files. +- Offset assignment is independent per partition (partition 0 and partition 1 can both have offset 0). + +## Notes + +- Each partition maintains its own offset sequence starting from 0. +- Multiple partitions can be written to simultaneously in a single request. +- The response must include entries for all requested partitions. +- Partition-level errors should be handled independently (one partition failure shouldn't affect others). +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). diff --git a/stage_descriptions/produce-07_ar4.md b/stage_descriptions/produce-07_ar4.md new file mode 100644 index 0000000..4ced272 --- /dev/null +++ b/stage_descriptions/produce-07_ar4.md @@ -0,0 +1,43 @@ +In this stage, you'll add support for producing to multiple topics in a single request. + +## Cross-Topic Production + +Your Kafka implementation should handle complex requests with multiple topics, manage independent offset tracking per topic and partition, and handle complex response structures. Each topic maintains completely independent offset sequences. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh /tmp/server.properties +``` + +It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple topics with their respective partitions. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (NO_ERROR). +- The `throttle_time_ms` field in the response is `0`. +- Each topic in the request has a corresponding topic response. +- Each topic response contains: + - The correct `name` field matching the topic name in the request. + - Each partition in the request has a corresponding partition response. + - Each partition response contains: + - The correct `partition` field matching the request. + - An error code of `0` (NO_ERROR). + - A valid `offset` field with the assigned offset for that topic-partition. + - A valid `timestamp` field. + - A correct `log_start_offset` field. +- Records are persisted to the correct topic-partition log files. +- Offset assignment is independent per topic-partition combination. + +## Notes + +- Each topic-partition combination maintains its own independent offset sequence. +- Multiple topics can be written to simultaneously in a single request. +- The response structure is nested: topics contain partition responses. +- Topic-level and partition-level errors should be handled independently. +- This is the most complex produce scenario, combining multi-topic and multi-partition handling. +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). From 22a3996e614f1d50a731fccaee0882c22e38fffb Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 22:43:39 +0530 Subject: [PATCH 04/52] feat: enhance stage description for the Produce API, adding interactive protocol inspector links and clarifying APIVersions requirements. --- stage_descriptions/produce-01_um3.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/stage_descriptions/produce-01_um3.md b/stage_descriptions/produce-01_um3.md index 43942fc..4fbcb0b 100644 --- a/stage_descriptions/produce-01_um3.md +++ b/stage_descriptions/produce-01_um3.md @@ -1,8 +1,15 @@ In this stage, you'll add an entry for the `Produce` API to the APIVersions response. -## APIVersions +## The Produce API -Your Kafka implementation should include the Produce API (key=0) in the ApiVersions response before implementing produce functionality. This would let the client know that the broker supports the Produce API. +The [Produce API](https://kafka.apache.org/protocol#The_Messages_Produce) (API key `0`) is used to produce messages to a Kafka topic. + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) + +In this stage, you'll only need to add an entry for the `Produce` API to the APIVersions response you implemented in earlier stages. This will let the client know that the broker supports the `Produce` API. We'll get to responding to `Produce` requests in later stages. ## Tests @@ -25,4 +32,5 @@ The tester will validate that: ## Notes - You don't have to implement support for the `Produce` request in this stage. We'll get to this in later stages. -- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. +- You'll still need to include the entry for `APIVersions` in your response to pass earlier stages. +- The `MaxVersion` for the `Produce` and `APIVersions` are different. For `APIVersions`, it is 4. For `Produce`, it is 11. From 498472c5e7af0d1feba09e9a46a1b76ad2cb3aa5 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 22:43:49 +0530 Subject: [PATCH 05/52] feat: update stage description for handling Produce requests to non-existent topics, including error response details and interactive protocol inspector links --- stage_descriptions/produce-02_ck2.md | 36 +++++++++++++++++++--------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/stage_descriptions/produce-02_ck2.md b/stage_descriptions/produce-02_ck2.md index 40fb57b..ffd5114 100644 --- a/stage_descriptions/produce-02_ck2.md +++ b/stage_descriptions/produce-02_ck2.md @@ -1,9 +1,23 @@ -In this stage, you'll add support for handling Produce requests to non-existent topics with proper error responses. +In this stage, you'll add support for handling Produce requests to non-existent topics. -## Handling Non-Existent Topics +## Produce API response for non-existent topics -Your Kafka implementation should validate topic existence and return appropriate error codes when a client tries to produce to a non-existent topic. Kafka stores metadata about topics in the `__cluster_metadata` topic. To check if a topic exists or not, you'll need to read the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. If the topic exists, the topic will have a directory with all the data required for it to operate at `/-`. -TODO: Do we need to explain the TOPIC_RECORD record in the `__cluster_metadata` log. +When a Kafka broker receives a Produce request, it first needs to validate that the topic exists. If a topic doesn't exist, it returns an appropriate error code and response. +To validate that a topic exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `TOPIC_RECORD`. If there exists a `TOPIC_RECORD` with the given topic name and the topic ID, the topic exists. Else, the topic doesn't exist. + +If the topic doesn't exist, the broker returns an error code of `3` (UNKNOWN_TOPIC_OR_PARTITION). + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) + +We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: +- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) + +This would help you understand the structure of the `TOPIC_RECORD` record inside the `__cluster_metadata` topic's log file. + +In this stage, you'll need to implement the response for a `Produce` request with a non-existent topic. ## Tests @@ -13,19 +27,19 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `Produce` (v11) request to a non-existent topic. +It'll then connect to your server on port 9092 and send a `Produce` (v11) request with a non-existent topic. The tester will validate that: - The first 4 bytes of your response (the "message length") are valid. - The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). +- The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). - The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic response inside response should correspond to the topic name in the request. -- The `partition` field in the partition response inside topic response should correspond to the partition in the request. -- The `offset` field in the partition response inside topic response should be `-1`. -- The `timestamp` field in the partition response inside topic response should be `-1`. -- The `log start offset` field in the partition response inside topic response should be `-1`. +- The `name` field in the nested `topic` response inside the top level response body should correspond to the topic name in the request. +- The `index` field in the nested `partition` response inside the nested `topic` response should correspond to the partition in the request. +- The `base_offset` field in the partition response inside topic response should be `-1`. +- The `log_append_time_ms` field in the partition response inside topic response should be `-1`. +- The `log_start_offset` field in the partition response inside topic response should be `-1`. ## Notes From 6aebd100b7021a46fa6e48ff22a7db9b3eaa8468 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 22:45:28 +0530 Subject: [PATCH 06/52] feat: update stage description to clarify handling of Produce requests for invalid topics, including error response details --- stage_descriptions/produce-02_ck2.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/stage_descriptions/produce-02_ck2.md b/stage_descriptions/produce-02_ck2.md index ffd5114..a9670d8 100644 --- a/stage_descriptions/produce-02_ck2.md +++ b/stage_descriptions/produce-02_ck2.md @@ -1,6 +1,6 @@ -In this stage, you'll add support for handling Produce requests to non-existent topics. +In this stage, you'll add support for handling Produce requests to invalid topics. -## Produce API response for non-existent topics +## Produce API response for invalid topics When a Kafka broker receives a Produce request, it first needs to validate that the topic exists. If a topic doesn't exist, it returns an appropriate error code and response. To validate that a topic exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `TOPIC_RECORD`. If there exists a `TOPIC_RECORD` with the given topic name and the topic ID, the topic exists. Else, the topic doesn't exist. @@ -17,7 +17,7 @@ We've also created an interactive protocol inspector for the `__cluster_metadata This would help you understand the structure of the `TOPIC_RECORD` record inside the `__cluster_metadata` topic's log file. -In this stage, you'll need to implement the response for a `Produce` request with a non-existent topic. +In this stage, you'll need to implement the response for a `Produce` request with an invalid topic. ## Tests @@ -27,7 +27,7 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `Produce` (v11) request with a non-existent topic. +It'll then connect to your server on port 9092 and send a `Produce` (v11) request with an invalid topic. The tester will validate that: From f501f90ad218af104d157162333a48ee2c793c5c Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 22:53:29 +0530 Subject: [PATCH 07/52] feat: enhance stage description for handling Produce requests to invalid partitions, detailing error responses and adding interactive protocol inspector links --- stage_descriptions/produce-03_ho8.md | 34 ++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/stage_descriptions/produce-03_ho8.md b/stage_descriptions/produce-03_ho8.md index 3a99ad4..4f398e2 100644 --- a/stage_descriptions/produce-03_ho8.md +++ b/stage_descriptions/produce-03_ho8.md @@ -1,9 +1,23 @@ -In this stage, you'll add support for handling Produce requests to invalid partitions for known topics with proper error responses. +In this stage, you'll add support for handling Produce requests to invalid partitions for known topics. -## Handling Invalid Partitions +## Produce API response for invalid partitions -Your Kafka implementation should validate that partition indices exist for known topics and return appropriate errors for invalid partitions. Kafka stores metadata about partitions in the `__cluster_metadata` topic. To check if a partition exists or not, you'll need to read the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. If the partition exists, the partition will have a directory with all the data required for it to operate at `/-`. -TODO: Do we need to explain the PARTITION_RECORD record in the `__cluster_metadata` log. +When a Kafka broker receives a Produce request, it first needs to validate that the partition exists. If a partition doesn't exist, it returns an appropriate error code and response. +To validate that a partition exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `PARTITION_RECORD`. If there exists a `PARTITION_RECORD` with the given partition index, the UUID of the topic it is associated with and the UUID of directory it is associated with, the partition exists. Else, the partition doesn't exist. + +If the partition doesn't exist, the broker returns an error code of `3` (UNKNOWN_TOPIC_OR_PARTITION). + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) + +We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: +- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) + +This would help you understand the structure of the `PARTITION_RECORD` record inside the `__cluster_metadata` topic's log file. + +In this stage, you'll need to implement the response for a `Produce` request with an invalid partition. ## Tests @@ -19,13 +33,13 @@ The tester will validate that: - The first 4 bytes of your response (the "message length") are valid. - The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). +- The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). - The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic response inside response should correspond to the topic name in the request. -- The `partition` field in the partition response inside topic response should correspond to the partition in the request. -- The `offset` field in the partition response inside topic response should be `-1`. -- The `timestamp` field in the partition response inside topic response should be `-1`. -- The `log start offset` field in the partition response inside topic response should be `-1`. +- The `name` field in the nested `topic` response inside the top level response body should correspond to the topic name in the request. +- The `index` field in the nested `partition` response inside the nested `topic` response should correspond to the partition in the request. +- The `base_offset` field in the partition response inside topic response should be `-1`. +- The `log_append_time_ms` field in the partition response inside topic response should be `-1`. +- The `log_start_offset` field in the partition response inside topic response should be `-1`. ## Notes From dcfb5833705034a095931960b64af71899880b6b Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 22:54:28 +0530 Subject: [PATCH 08/52] feat: add TODO note for enhancing cluster metadata binspec details in Produce stage description --- stage_descriptions/produce-03_ho8.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/stage_descriptions/produce-03_ho8.md b/stage_descriptions/produce-03_ho8.md index 4f398e2..12f1667 100644 --- a/stage_descriptions/produce-03_ho8.md +++ b/stage_descriptions/produce-03_ho8.md @@ -45,3 +45,7 @@ The tester will validate that: - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. - The official Kafka docs don't cover the structure of records inside the `__cluster_metadata` topic, but you can find the definitions in the Kafka source code [here](https://github.com/apache/kafka/tree/5b3027dfcbcb62d169d4b4421260226e620459af/metadata/src/main/resources/common/metadata). + +TODO: @rohitpaulk our cluster metadata binspec doesn't go into details for each cluster metadata payload type. +Something for me to add. +In the meantime the github link works. \ No newline at end of file From 61b7f2af11f702a1763c732699d34cfb654a1978 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:01:21 +0530 Subject: [PATCH 09/52] feat: enhance Produce stage description to clarify single record production process, including validation steps, log file format, and updated response fields --- stage_descriptions/produce-04_ps7.md | 30 +++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/stage_descriptions/produce-04_ps7.md b/stage_descriptions/produce-04_ps7.md index 1db0dec..1425bcd 100644 --- a/stage_descriptions/produce-04_ps7.md +++ b/stage_descriptions/produce-04_ps7.md @@ -1,8 +1,24 @@ -In this stage, you'll add support for successfully producing a single record to a valid topic and partition. +In this stage, you'll add support for successfully producing a single record. ## Single Record Production -Your Kafka implementation should accept valid Produce requests, store the record in the appropriate log file, and return successful responses with assigned offsets. The record must be persisted to the topic's log file at `/-/00000000000000000000.log` using Kafka's on-disk log format. +When a Kafka broker receives a Produce request for a valid topic and partition, it needs to: + +1. **Validate the topic and partition exist** (using the `__cluster_metadata` topic's log file) +2. **Store the record** in the appropriate log file using Kafka's on-disk format +3. **Return a successful response** with the assigned offset + +The record must be persisted to the topic's log file at `/-/00000000000000000000.log` using Kafka's RecordBatch format. + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) + +We've also created an interactive protocol inspector for Kafka's log file format: +- 🔎 [Kafka Log File Format](https://binspec.org/kafka-log-file) + +This will help you understand how to write records to the log file in the correct binary format. ## Tests @@ -21,9 +37,9 @@ The tester will validate that: - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. - The `name` field in the topic response matches the topic name in the request. -- The `partition` field in the partition response matches the partition in the request. -- The `offset` field in the partition response contains the assigned offset to the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) -- The `timestamp` field in the partition response contains -1 (signifying that the timestamp is latest). +- The `index` field in the partition response matches the partition in the request. +- The `base_offset` field in the partition response contains the assigned offset to the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) +- The `log_append_time_ms` field in the partition response contains `-1` (signifying that the timestamp is latest). - The `log_start_offset` field in the partition response is `0`. The tester will also verify that the record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. @@ -32,5 +48,5 @@ The tester will also verify that the record is persisted to the appropriate log - You'll need to implement log file writing using Kafka's binary log format. - Records must be stored in RecordBatch format with proper CRC validation. -- The offset assignment should start from 0 for new partitions. -- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). +- The offset assignment should start from 0 for new partitions and increment for each subsequent record. +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. From a75a9feb3f0e937542caa40d9d44ca95727f4421 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:02:39 +0530 Subject: [PATCH 10/52] refactor: streamline Produce stage description for single record production by consolidating validation, storage, and response steps into a single sentence --- stage_descriptions/produce-04_ps7.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/stage_descriptions/produce-04_ps7.md b/stage_descriptions/produce-04_ps7.md index 1425bcd..e9dcec5 100644 --- a/stage_descriptions/produce-04_ps7.md +++ b/stage_descriptions/produce-04_ps7.md @@ -2,11 +2,7 @@ In this stage, you'll add support for successfully producing a single record. ## Single Record Production -When a Kafka broker receives a Produce request for a valid topic and partition, it needs to: - -1. **Validate the topic and partition exist** (using the `__cluster_metadata` topic's log file) -2. **Store the record** in the appropriate log file using Kafka's on-disk format -3. **Return a successful response** with the assigned offset +When a Kafka broker receives a Produce request for a valid topic and partition, it needs to validate that the topic and partition exist (using the `__cluster_metadata` topic's log file), store the record in the appropriate log file using Kafka's on-disk format, and return a successful response with the assigned offset. The record must be persisted to the topic's log file at `/-/00000000000000000000.log` using Kafka's RecordBatch format. From 719740e535a051d2e335608872a79636fca90e9a Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:11:47 +0530 Subject: [PATCH 11/52] feat: update Produce stage description to clarify Kafka's on-disk log format and add interactive protocol inspector link for cluster metadata topic's log file --- stage_descriptions/produce-04_ps7.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/stage_descriptions/produce-04_ps7.md b/stage_descriptions/produce-04_ps7.md index e9dcec5..3934c35 100644 --- a/stage_descriptions/produce-04_ps7.md +++ b/stage_descriptions/produce-04_ps7.md @@ -11,10 +11,11 @@ We've created an interactive protocol inspector for the request & response struc - 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) - 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) -We've also created an interactive protocol inspector for Kafka's log file format: -- 🔎 [Kafka Log File Format](https://binspec.org/kafka-log-file) +Kafka's on-disk log format is just records inside a RecordBatch. The same RecordBatch format that is used in the Produce request, Fetch request is also used in the on-disk log file. + +You can refer to the following interactive protocol inspector for Kafka's log file format: +- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) -This will help you understand how to write records to the log file in the correct binary format. ## Tests @@ -42,7 +43,6 @@ The tester will also verify that the record is persisted to the appropriate log ## Notes -- You'll need to implement log file writing using Kafka's binary log format. -- Records must be stored in RecordBatch format with proper CRC validation. +- On-disk log files must be stored in RecordBatch format with proper CRC validation. - The offset assignment should start from 0 for new partitions and increment for each subsequent record. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. From 536b47827b939ddfedf3b72382ba64fb7efc8e78 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:44:54 +0530 Subject: [PATCH 12/52] feat: enhance Produce stage description to clarify handling of multiple records in a single RecordBatch, including validation, storage, and updated response fields --- stage_descriptions/produce-05_sb8.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/stage_descriptions/produce-05_sb8.md b/stage_descriptions/produce-05_sb8.md index c0ea97e..04f3f3a 100644 --- a/stage_descriptions/produce-05_sb8.md +++ b/stage_descriptions/produce-05_sb8.md @@ -1,8 +1,13 @@ -In this stage, you'll add support for producing multiple records from a single request with multiple record batches in the payload. +In this stage, you'll add support for producing multiple records from a single request, with multiple records inside a single RecordBatch. ## Batch Processing -Your Kafka implementation should handle multiple records in a single batch, assign sequential offsets, and validate the LastOffsetDelta field correctly. The RecordBatch containing multiple records must be stored atomically to the log file. +When a Kafka broker receives a Produce request containing a RecordBatch with multiple records, it needs to validate that the topic and partition exist, assign sequential offsets to each record within the batch, and store the entire batch atomically to the log file. The RecordBatch containing multiple records must be stored as a single unit to the topic's log file at `/-/00000000000000000000.log`. + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) ## Tests @@ -21,16 +26,15 @@ The tester will validate that: - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. - The `name` field in the topic response matches the topic name in the request. -- The `partition` field in the partition response matches the partition in the request. -- The `offset` field in the partition response contains the base offset for the batch. -- The `timestamp` field in the partition response contains a valid timestamp. -- The `log_start_offset` field in the partition response is correct. +- The `index` field in the partition response matches the partition in the request. +- The `base_offset` field in the partition response contains 0 (the base offset for the batch). +- The `log_append_time_ms` field in the partition response contains `-1` (signifying that the timestamp is latest). +- The `log_start_offset` field in the partition response is `0`. The tester will also verify that the record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log` with sequential offsets. ## Notes - Records within a batch must be assigned sequential offsets (e.g., if base offset is 5, records get offsets 5, 6, 7). -- The entire batch is treated as a single atomic operation. - The response should return the base offset of the batch, not individual record offsets. -- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. From ced137d027ef64700e5cd5bd6be14f97aab29241 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:51:17 +0530 Subject: [PATCH 13/52] feat: update Produce stage description to clarify handling of multiple partitions, including validation, response structure, and log file requirements --- stage_descriptions/produce-06_mf2.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/stage_descriptions/produce-06_mf2.md b/stage_descriptions/produce-06_mf2.md index 5d4f778..3d3bac3 100644 --- a/stage_descriptions/produce-06_mf2.md +++ b/stage_descriptions/produce-06_mf2.md @@ -2,7 +2,14 @@ In this stage, you'll add support for producing to multiple partitions of the sa ## Partition Routing -Your Kafka implementation should handle writes to multiple partitions within the same topic, manage independent offset assignment per partition, and aggregate responses correctly. Each partition maintains its own offset sequence independently. +When a Kafka broker receives a Produce request targeting multiple partitions of the same topic, it needs to validate that the topic and all partitions exist, write records to each partition's log file independently, and return a response containing results for all partitions. Each partition maintains its own offset sequence independently, so partition 0 and partition 1 can both have records starting at offset 0. + +The request will contain multiple RecordBatches, one for each partition. Each RecordBatch will contain a single record. + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) ## Tests @@ -23,18 +30,18 @@ The tester will validate that: - The `name` field in the topic response matches the topic name in the request. - Each partition in the request has a corresponding partition response. - Each partition response contains: - - The correct `partition` field matching the request. + - The correct `index` field matching the partition in the request. - An error code of `0` (NO_ERROR). - - A valid `offset` field with the assigned offset for that partition. - - A valid `timestamp` field. - - A correct `log_start_offset` field. + - A valid `base_offset` field with the assigned offset for that partition. + - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is latest). + - The `log_start_offset` field is `0`. - Records are persisted to the correct partition log files. - Offset assignment is independent per partition (partition 0 and partition 1 can both have offset 0). +The tester will also verify that the records are persisted to the correct partition log files. + ## Notes -- Each partition maintains its own offset sequence starting from 0. -- Multiple partitions can be written to simultaneously in a single request. - The response must include entries for all requested partitions. -- Partition-level errors should be handled independently (one partition failure shouldn't affect others). -- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). +- On-disk log files must be stored in RecordBatch format with proper CRC validation. +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. From 8f99a3c710b0a36006316e28a6b3cd978b9715d9 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:54:44 +0530 Subject: [PATCH 14/52] feat: refine Produce stage descriptions to clarify handling of multiple topics and partitions, including validation, response structure, and offset management --- stage_descriptions/produce-06_mf2.md | 3 +-- stage_descriptions/produce-07_ar4.md | 23 ++++++++++++----------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/stage_descriptions/produce-06_mf2.md b/stage_descriptions/produce-06_mf2.md index 3d3bac3..0d9b8de 100644 --- a/stage_descriptions/produce-06_mf2.md +++ b/stage_descriptions/produce-06_mf2.md @@ -4,8 +4,6 @@ In this stage, you'll add support for producing to multiple partitions of the sa When a Kafka broker receives a Produce request targeting multiple partitions of the same topic, it needs to validate that the topic and all partitions exist, write records to each partition's log file independently, and return a response containing results for all partitions. Each partition maintains its own offset sequence independently, so partition 0 and partition 1 can both have records starting at offset 0. -The request will contain multiple RecordBatches, one for each partition. Each RecordBatch will contain a single record. - We've created an interactive protocol inspector for the request & response structures for `Produce`: - 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) @@ -20,6 +18,7 @@ The tester will execute your program like this: ``` It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple partitions of the same topic. +The request will contain multiple RecordBatches, one for each partition. Each RecordBatch will contain a single record. The tester will validate that: diff --git a/stage_descriptions/produce-07_ar4.md b/stage_descriptions/produce-07_ar4.md index 4ced272..1fd1fae 100644 --- a/stage_descriptions/produce-07_ar4.md +++ b/stage_descriptions/produce-07_ar4.md @@ -2,7 +2,12 @@ In this stage, you'll add support for producing to multiple topics in a single r ## Cross-Topic Production -Your Kafka implementation should handle complex requests with multiple topics, manage independent offset tracking per topic and partition, and handle complex response structures. Each topic maintains completely independent offset sequences. +When a Kafka broker receives a Produce request targeting multiple topics with their respective partitions, it needs to validate that all topics and partitions exist, write records to each topic-partition's log file independently, and return a nested response structure containing results for all topics and their partitions. Each topic-partition combination maintains its own independent offset sequence, so topic "foo" partition 0 and topic "bar" partition 0 can both have records starting at offset 0. + +We've created an interactive protocol inspector for the request & response structures for `Produce`: + +- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) +- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) ## Tests @@ -13,6 +18,7 @@ The tester will execute your program like this: ``` It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple topics with their respective partitions. +The request will contain data for multiple topics, and a single partition for each topic. The tester will validate that: @@ -25,19 +31,14 @@ The tester will validate that: - The correct `name` field matching the topic name in the request. - Each partition in the request has a corresponding partition response. - Each partition response contains: - - The correct `partition` field matching the request. + - The correct `index` field matching the partition in the request. - An error code of `0` (NO_ERROR). - - A valid `offset` field with the assigned offset for that topic-partition. - - A valid `timestamp` field. - - A correct `log_start_offset` field. + - A valid `base_offset` field with the assigned offset for that topic-partition. + - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is latest). + - The `log_start_offset` field is `0`. - Records are persisted to the correct topic-partition log files. - Offset assignment is independent per topic-partition combination. ## Notes -- Each topic-partition combination maintains its own independent offset sequence. -- Multiple topics can be written to simultaneously in a single request. -- The response structure is nested: topics contain partition responses. -- Topic-level and partition-level errors should be handled independently. -- This is the most complex produce scenario, combining multi-topic and multi-partition handling. -- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). +- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. From 8b0412f79fd9a48df493f67898e4bb540cacf4ab Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 10 Jul 2025 23:58:16 +0530 Subject: [PATCH 15/52] feat: add TODO for testing multiple requests in Produce stage, focusing on baseOffset and log_start_offset handling in Kafka --- stage_descriptions/produce-05_sb8.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/stage_descriptions/produce-05_sb8.md b/stage_descriptions/produce-05_sb8.md index 04f3f3a..4d533e6 100644 --- a/stage_descriptions/produce-05_sb8.md +++ b/stage_descriptions/produce-05_sb8.md @@ -38,3 +38,5 @@ The tester will also verify that the record is persisted to the appropriate log - Records within a batch must be assigned sequential offsets (e.g., if base offset is 5, records get offsets 5, 6, 7). - The response should return the base offset of the batch, not individual record offsets. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. + +TODO: Send multiple request per stage from stage 5 onwards. (Need to test how baseOffset & log_start_offset are handled by kafka.) \ No newline at end of file From 04bcea8eec5a0d8ec7c3696563cc3ffc941e9d38 Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 11 Jul 2025 00:01:29 +0530 Subject: [PATCH 16/52] feat: add new challenge extension for producing messages in Kafka, including multiple stages for handling various production scenarios such as invalid topics, partitions, and batch production --- course-definition.yml | 61 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/course-definition.yml b/course-definition.yml index c4f7483..15ef0d1 100644 --- a/course-definition.yml +++ b/course-definition.yml @@ -71,6 +71,15 @@ extensions: [fetch-api]: https://kafka.apache.org/protocol.html#The_Messages_Fetch + - slug: "producing-messages" + name: "Producing Messages" + description_markdown: | + In this challenge extension you'll add support for producing messages by implementing the [Produce][produce-api] API. + + Along the way you'll learn about how Kafka's Produce API works, how Kafka stores messages on disk and more. + + [produce-api]: https://kafka.apache.org/protocol.html#The_Messages_Produce + stages: - slug: "vi6" name: "Bind to a port" @@ -199,3 +208,55 @@ stages: difficulty: hard marketing_md: |- In this stage, you'll implement the Fetch response for a topic with multiple messages, reading them from disk. + + # Producing Messages + + - slug: "um3" + primary_extension_slug: "producing-messages" + name: "Include Produce in APIVersions" + difficulty: easy + marketing_md: |- + In this stage, you'll add the Produce API to the APIVersions response. + + - slug: "ck2" + primary_extension_slug: "producing-messages" + name: "Produce to Invalid topic" + difficulty: medium + marketing_md: |- + In this stage, you'll implement the Produce response for an invalid topic. + + - slug: "ho8" + primary_extension_slug: "producing-messages" + name: "Produce to Invalid partition" + difficulty: medium + marketing_md: |- + In this stage, you'll implement the Produce response for an invalid partition. + + - slug: "ps7" + primary_extension_slug: "producing-messages" + name: "Produce a single record" + difficulty: hard + marketing_md: |- + In this stage, you'll implement successfully producing a single record to disk. + + - slug: "sb8" + primary_extension_slug: "producing-messages" + name: "Batch Produce multiple records" + difficulty: hard + marketing_md: |- + In this stage, you'll implement producing multiple records in a single record batch. + + - slug: "mf2" + primary_extension_slug: "producing-messages" + name: "Produce to multiple partitions" + difficulty: hard + marketing_md: |- + In this stage, you'll implement producing to multiple partitions of the same topic. + + - slug: "ar4" + primary_extension_slug: "producing-messages" + name: "Produce to multiple topics" + difficulty: hard + marketing_md: |- + In this stage, you'll implement producing to multiple topics in a single request. + Stages: From 91824727b8fe18342912534ca0f2e73d5fbe67a4 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 12:14:40 +0530 Subject: [PATCH 17/52] chore: rename stage description files to match expected format --- .../{produce-01_um3.md => producing-messages-01-um3.md} | 0 .../{produce-02_ck2.md => producing-messages-02-ck2.md} | 0 .../{produce-03_ho8.md => producing-messages-03-ho8.md} | 0 .../{produce-04_ps7.md => producing-messages-04-ps7.md} | 0 .../{produce-05_sb8.md => producing-messages-05-sb8.md} | 0 .../{produce-06_mf2.md => producing-messages-06-mf2.md} | 0 .../{produce-07_ar4.md => producing-messages-07-ar4.md} | 0 7 files changed, 0 insertions(+), 0 deletions(-) rename stage_descriptions/{produce-01_um3.md => producing-messages-01-um3.md} (100%) rename stage_descriptions/{produce-02_ck2.md => producing-messages-02-ck2.md} (100%) rename stage_descriptions/{produce-03_ho8.md => producing-messages-03-ho8.md} (100%) rename stage_descriptions/{produce-04_ps7.md => producing-messages-04-ps7.md} (100%) rename stage_descriptions/{produce-05_sb8.md => producing-messages-05-sb8.md} (100%) rename stage_descriptions/{produce-06_mf2.md => producing-messages-06-mf2.md} (100%) rename stage_descriptions/{produce-07_ar4.md => producing-messages-07-ar4.md} (100%) diff --git a/stage_descriptions/produce-01_um3.md b/stage_descriptions/producing-messages-01-um3.md similarity index 100% rename from stage_descriptions/produce-01_um3.md rename to stage_descriptions/producing-messages-01-um3.md diff --git a/stage_descriptions/produce-02_ck2.md b/stage_descriptions/producing-messages-02-ck2.md similarity index 100% rename from stage_descriptions/produce-02_ck2.md rename to stage_descriptions/producing-messages-02-ck2.md diff --git a/stage_descriptions/produce-03_ho8.md b/stage_descriptions/producing-messages-03-ho8.md similarity index 100% rename from stage_descriptions/produce-03_ho8.md rename to stage_descriptions/producing-messages-03-ho8.md diff --git a/stage_descriptions/produce-04_ps7.md b/stage_descriptions/producing-messages-04-ps7.md similarity index 100% rename from stage_descriptions/produce-04_ps7.md rename to stage_descriptions/producing-messages-04-ps7.md diff --git a/stage_descriptions/produce-05_sb8.md b/stage_descriptions/producing-messages-05-sb8.md similarity index 100% rename from stage_descriptions/produce-05_sb8.md rename to stage_descriptions/producing-messages-05-sb8.md diff --git a/stage_descriptions/produce-06_mf2.md b/stage_descriptions/producing-messages-06-mf2.md similarity index 100% rename from stage_descriptions/produce-06_mf2.md rename to stage_descriptions/producing-messages-06-mf2.md diff --git a/stage_descriptions/produce-07_ar4.md b/stage_descriptions/producing-messages-07-ar4.md similarity index 100% rename from stage_descriptions/produce-07_ar4.md rename to stage_descriptions/producing-messages-07-ar4.md From 1600b2924512dc7e65a29fb89f155048f6b06c87 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 23:16:00 +0530 Subject: [PATCH 18/52] feat: update Produce stage description to include validation for both invalid topics and partitions, enhancing clarity on error handling and response structure --- .../producing-messages-02-ck2.md | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/stage_descriptions/producing-messages-02-ck2.md b/stage_descriptions/producing-messages-02-ck2.md index a9670d8..264808e 100644 --- a/stage_descriptions/producing-messages-02-ck2.md +++ b/stage_descriptions/producing-messages-02-ck2.md @@ -1,11 +1,22 @@ -In this stage, you'll add support for handling Produce requests to invalid topics. +In this stage, you'll add support for handling Produce requests to invalid topics or partitions. -## Produce API response for invalid topics +## Produce API response for invalid topics or partitions -When a Kafka broker receives a Produce request, it first needs to validate that the topic exists. If a topic doesn't exist, it returns an appropriate error code and response. -To validate that a topic exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `TOPIC_RECORD`. If there exists a `TOPIC_RECORD` with the given topic name and the topic ID, the topic exists. Else, the topic doesn't exist. +When a Kafka broker receives a Produce request, it needs to validate that both the topic and partition exist. If either the topic or partition doesn't exist, it returns an appropriate error code and response. -If the topic doesn't exist, the broker returns an error code of `3` (UNKNOWN_TOPIC_OR_PARTITION). +The broker performs validation in this order: +1. **Topic validation**: Check if the topic exists by reading the `__cluster_metadata` topic's log file +2. **Partition validation**: If the topic exists, check if the partition exists within that topic + +### Topic Validation + +To validate that a topic exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `TOPIC_RECORD`. If there exists a `TOPIC_RECORD` with the given topic name and the topic ID, the topic exists. + +### Partition Validation + +To validate that a partition exists, the broker reads the same `__cluster_metadata` topic's log file and finds the partition's metadata, which is a `record` (inside a RecordBatch) with a payload of type `PARTITION_RECORD`. If there exists a `PARTITION_RECORD` with the given partition index, the UUID of the topic it is associated with and the UUID of directory it is associated with, the partition exists. + +If either the topic or partition doesn't exist, the broker returns an error code of `3` (UNKNOWN_TOPIC_OR_PARTITION). We've created an interactive protocol inspector for the request & response structures for `Produce`: @@ -15,9 +26,9 @@ We've created an interactive protocol inspector for the request & response struc We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) -This would help you understand the structure of the `TOPIC_RECORD` record inside the `__cluster_metadata` topic's log file. +This would help you understand the structure of the `TOPIC_RECORD` and `PARTITION_RECORD` records inside the `__cluster_metadata` topic's log file. -In this stage, you'll need to implement the response for a `Produce` request with an invalid topic. +In this stage, you'll need to implement the response for a `Produce` request with either an invalid topic or invalid partition. ## Tests @@ -27,7 +38,7 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `Produce` (v11) request with an invalid topic. +It'll then connect to your server on port 9092 and send a `Produce` (v11) request either with an invalid topic name or a valid topic but invalid partition. The tester will validate that: From aaae417a478bc1e4bd2a9d91a0e7bcb034c34740 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 23:28:44 +0530 Subject: [PATCH 19/52] feat: add support for producing messages to multiple topics and partitions in a single request, including validation, independent offset management, and nested response structure --- .../producing-messages-03-ho8.md | 51 ------------------- ...04-ps7.md => producing-messages-03-ps7.md} | 0 ...05-sb8.md => producing-messages-04-sb8.md} | 0 ...06-mf2.md => producing-messages-05-mf2.md} | 0 ...07-ar4.md => producing-messages-06-ar4.md} | 0 5 files changed, 51 deletions(-) delete mode 100644 stage_descriptions/producing-messages-03-ho8.md rename stage_descriptions/{producing-messages-04-ps7.md => producing-messages-03-ps7.md} (100%) rename stage_descriptions/{producing-messages-05-sb8.md => producing-messages-04-sb8.md} (100%) rename stage_descriptions/{producing-messages-06-mf2.md => producing-messages-05-mf2.md} (100%) rename stage_descriptions/{producing-messages-07-ar4.md => producing-messages-06-ar4.md} (100%) diff --git a/stage_descriptions/producing-messages-03-ho8.md b/stage_descriptions/producing-messages-03-ho8.md deleted file mode 100644 index 12f1667..0000000 --- a/stage_descriptions/producing-messages-03-ho8.md +++ /dev/null @@ -1,51 +0,0 @@ -In this stage, you'll add support for handling Produce requests to invalid partitions for known topics. - -## Produce API response for invalid partitions - -When a Kafka broker receives a Produce request, it first needs to validate that the partition exists. If a partition doesn't exist, it returns an appropriate error code and response. -To validate that a partition exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `PARTITION_RECORD`. If there exists a `PARTITION_RECORD` with the given partition index, the UUID of the topic it is associated with and the UUID of directory it is associated with, the partition exists. Else, the partition doesn't exist. - -If the partition doesn't exist, the broker returns an error code of `3` (UNKNOWN_TOPIC_OR_PARTITION). - -We've created an interactive protocol inspector for the request & response structures for `Produce`: - -- 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) -- 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) - -We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: -- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) - -This would help you understand the structure of the `PARTITION_RECORD` record inside the `__cluster_metadata` topic's log file. - -In this stage, you'll need to implement the response for a `Produce` request with an invalid partition. - -## Tests - -The tester will execute your program like this: - -```bash -./your_program.sh /tmp/server.properties -``` - -It'll then connect to your server on port 9092 and send a `Produce` (v11) request to a known topic with an invalid partition. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the nested `topic` response inside the top level response body should correspond to the topic name in the request. -- The `index` field in the nested `partition` response inside the nested `topic` response should correspond to the partition in the request. -- The `base_offset` field in the partition response inside topic response should be `-1`. -- The `log_append_time_ms` field in the partition response inside topic response should be `-1`. -- The `log_start_offset` field in the partition response inside topic response should be `-1`. - -## Notes - -- The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. -- The official Kafka docs don't cover the structure of records inside the `__cluster_metadata` topic, but you can find the definitions in the Kafka source code [here](https://github.com/apache/kafka/tree/5b3027dfcbcb62d169d4b4421260226e620459af/metadata/src/main/resources/common/metadata). - -TODO: @rohitpaulk our cluster metadata binspec doesn't go into details for each cluster metadata payload type. -Something for me to add. -In the meantime the github link works. \ No newline at end of file diff --git a/stage_descriptions/producing-messages-04-ps7.md b/stage_descriptions/producing-messages-03-ps7.md similarity index 100% rename from stage_descriptions/producing-messages-04-ps7.md rename to stage_descriptions/producing-messages-03-ps7.md diff --git a/stage_descriptions/producing-messages-05-sb8.md b/stage_descriptions/producing-messages-04-sb8.md similarity index 100% rename from stage_descriptions/producing-messages-05-sb8.md rename to stage_descriptions/producing-messages-04-sb8.md diff --git a/stage_descriptions/producing-messages-06-mf2.md b/stage_descriptions/producing-messages-05-mf2.md similarity index 100% rename from stage_descriptions/producing-messages-06-mf2.md rename to stage_descriptions/producing-messages-05-mf2.md diff --git a/stage_descriptions/producing-messages-07-ar4.md b/stage_descriptions/producing-messages-06-ar4.md similarity index 100% rename from stage_descriptions/producing-messages-07-ar4.md rename to stage_descriptions/producing-messages-06-ar4.md From 404af0f2b7d313e162c1d733fc9ece7c83a379ac Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 23:28:50 +0530 Subject: [PATCH 20/52] feat: update Produce stage name and description to clarify handling of invalid topics or partitions, enhancing consistency in error response messaging --- course-definition.yml | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/course-definition.yml b/course-definition.yml index 15ef0d1..d802514 100644 --- a/course-definition.yml +++ b/course-definition.yml @@ -220,17 +220,10 @@ stages: - slug: "ck2" primary_extension_slug: "producing-messages" - name: "Produce to Invalid topic" + name: "Produce to Invalid topic or partition" difficulty: medium marketing_md: |- - In this stage, you'll implement the Produce response for an invalid topic. - - - slug: "ho8" - primary_extension_slug: "producing-messages" - name: "Produce to Invalid partition" - difficulty: medium - marketing_md: |- - In this stage, you'll implement the Produce response for an invalid partition. + In this stage, you'll implement the Produce response for an invalid topic or partition. - slug: "ps7" primary_extension_slug: "producing-messages" From 88f5c4d7094ada2386a0f467a7df181e30f62d56 Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 18 Jul 2025 00:31:19 +0530 Subject: [PATCH 21/52] fix: clarify language in Produce stage descriptions regarding request handling and response structure, ensuring consistency and accuracy --- stage_descriptions/producing-messages-01-um3.md | 4 ++-- stage_descriptions/producing-messages-02-ck2.md | 6 ++---- stage_descriptions/producing-messages-05-mf2.md | 1 + 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/stage_descriptions/producing-messages-01-um3.md b/stage_descriptions/producing-messages-01-um3.md index 4fbcb0b..7995cfc 100644 --- a/stage_descriptions/producing-messages-01-um3.md +++ b/stage_descriptions/producing-messages-01-um3.md @@ -31,6 +31,6 @@ The tester will validate that: ## Notes -- You don't have to implement support for the `Produce` request in this stage. We'll get to this in later stages. +- You don't have to implement support for handling `Produce` requests in this stage. We'll get to this in later stages. - You'll still need to include the entry for `APIVersions` in your response to pass earlier stages. -- The `MaxVersion` for the `Produce` and `APIVersions` are different. For `APIVersions`, it is 4. For `Produce`, it is 11. +- The `MaxVersion` for `Produce` and `APIVersions` are different. For `APIVersions`, it is 4. For `Produce`, it is 11. diff --git a/stage_descriptions/producing-messages-02-ck2.md b/stage_descriptions/producing-messages-02-ck2.md index 264808e..be631a6 100644 --- a/stage_descriptions/producing-messages-02-ck2.md +++ b/stage_descriptions/producing-messages-02-ck2.md @@ -24,11 +24,9 @@ We've created an interactive protocol inspector for the request & response struc - 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: -- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) +- 🔎 [__cluster_metadata log file](https://binspec.org/kafka-cluster-metadata) -This would help you understand the structure of the `TOPIC_RECORD` and `PARTITION_RECORD` records inside the `__cluster_metadata` topic's log file. - -In this stage, you'll need to implement the response for a `Produce` request with either an invalid topic or invalid partition. +In this stage, you'll need to implement the response for a `Produce` request with either an invalid topic or invalid partition. In later stages, you'll handle successfully producing messages to valid topics and partitions, persist messages to disk using Kafka's RecordBatch format, manage offset assignment, handle batch processing, and support multiple partitions and topics. ## Tests diff --git a/stage_descriptions/producing-messages-05-mf2.md b/stage_descriptions/producing-messages-05-mf2.md index 0d9b8de..0de155d 100644 --- a/stage_descriptions/producing-messages-05-mf2.md +++ b/stage_descriptions/producing-messages-05-mf2.md @@ -26,6 +26,7 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. +- There is a single topic present in the response. - The `name` field in the topic response matches the topic name in the request. - Each partition in the request has a corresponding partition response. - Each partition response contains: From 9627e0a35d36050e76ebe7df6f180170c7fc1aa1 Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 18 Jul 2025 00:31:36 +0530 Subject: [PATCH 22/52] fix: update stage names in course-definition.yml for improved clarity and consistency in topic and partition handling --- course-definition.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/course-definition.yml b/course-definition.yml index d802514..73cd241 100644 --- a/course-definition.yml +++ b/course-definition.yml @@ -220,7 +220,7 @@ stages: - slug: "ck2" primary_extension_slug: "producing-messages" - name: "Produce to Invalid topic or partition" + name: "Produce to an invalid topic or partition" difficulty: medium marketing_md: |- In this stage, you'll implement the Produce response for an invalid topic or partition. @@ -234,7 +234,7 @@ stages: - slug: "sb8" primary_extension_slug: "producing-messages" - name: "Batch Produce multiple records" + name: "Produce multiple records" difficulty: hard marketing_md: |- In this stage, you'll implement producing multiple records in a single record batch. @@ -252,4 +252,3 @@ stages: difficulty: hard marketing_md: |- In this stage, you'll implement producing to multiple topics in a single request. - Stages: From b54c66f730fa9e685e83362795937ed39eb9811d Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 18 Jul 2025 00:43:09 +0530 Subject: [PATCH 23/52] fix: enhance clarity in Produce stage descriptions by refining language around response structure and validation for topics and partitions, ensuring consistency in error handling and log file persistence --- .../producing-messages-02-ck2.md | 14 +++++----- .../producing-messages-03-ps7.md | 19 +++++++------- .../producing-messages-04-sb8.md | 17 ++++++------ .../producing-messages-05-mf2.md | 26 +++++++++---------- .../producing-messages-06-ar4.md | 7 +++-- 5 files changed, 42 insertions(+), 41 deletions(-) diff --git a/stage_descriptions/producing-messages-02-ck2.md b/stage_descriptions/producing-messages-02-ck2.md index be631a6..8c96dd6 100644 --- a/stage_descriptions/producing-messages-02-ck2.md +++ b/stage_descriptions/producing-messages-02-ck2.md @@ -14,7 +14,7 @@ To validate that a topic exists, the broker reads the `__cluster_metadata` topic ### Partition Validation -To validate that a partition exists, the broker reads the same `__cluster_metadata` topic's log file and finds the partition's metadata, which is a `record` (inside a RecordBatch) with a payload of type `PARTITION_RECORD`. If there exists a `PARTITION_RECORD` with the given partition index, the UUID of the topic it is associated with and the UUID of directory it is associated with, the partition exists. +To validate that a partition exists, the broker reads the same `__cluster_metadata` topic's log file and finds the partition's metadata, which is a `record` (inside a RecordBatch) with a payload of type `PARTITION_RECORD`. If there exists a `PARTITION_RECORD` with the given partition index, the UUID of the topic it is associated with, and the UUID of the directory it is associated with, the partition exists. If either the topic or partition doesn't exist, the broker returns an error code of `3` (UNKNOWN_TOPIC_OR_PARTITION). @@ -44,11 +44,13 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). - The `throttle_time_ms` field in the response is `0`. -- The `name` field in the nested `topic` response inside the top level response body should correspond to the topic name in the request. -- The `index` field in the nested `partition` response inside the nested `topic` response should correspond to the partition in the request. -- The `base_offset` field in the partition response inside topic response should be `-1`. -- The `log_append_time_ms` field in the partition response inside topic response should be `-1`. -- The `log_start_offset` field in the partition response inside topic response should be `-1`. +- The topic response contains: + - The `name` field should correspond to the topic name in the request. + - The partition response contains: + - The `index` field should correspond to the partition in the request. + - The `base_offset` field should be `-1`. + - The `log_append_time_ms` field should be `-1`. + - The `log_start_offset` field should be `-1`. ## Notes diff --git a/stage_descriptions/producing-messages-03-ps7.md b/stage_descriptions/producing-messages-03-ps7.md index 3934c35..8e4d5a4 100644 --- a/stage_descriptions/producing-messages-03-ps7.md +++ b/stage_descriptions/producing-messages-03-ps7.md @@ -11,7 +11,7 @@ We've created an interactive protocol inspector for the request & response struc - 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) - 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) -Kafka's on-disk log format is just records inside a RecordBatch. The same RecordBatch format that is used in the Produce request, Fetch request is also used in the on-disk log file. +Kafka's on-disk log format is just records inside a RecordBatch. The same RecordBatch format that is used in the Produce request and Fetch request is also used in the on-disk log file. You can refer to the following interactive protocol inspector for Kafka's log file format: - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) @@ -25,7 +25,7 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send multiple successive `Produce` (v11) requests to a valid topic and partition with single records as payload. +It'll then connect to your server on port 9092 and send multiple successive `Produce` (v11) requests to a valid topic and partition with single records as the payload. The tester will validate that: @@ -33,13 +33,14 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic response matches the topic name in the request. -- The `index` field in the partition response matches the partition in the request. -- The `base_offset` field in the partition response contains the assigned offset to the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) -- The `log_append_time_ms` field in the partition response contains `-1` (signifying that the timestamp is latest). -- The `log_start_offset` field in the partition response is `0`. - -The tester will also verify that the record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. +- The topic response contains: + - The `name` field matches the topic name in the request. + - The partition response contains: + - The `index` field matches the partition in the request. + - The `base_offset` field contains the assigned offset to the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) + - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - The `log_start_offset` field is `0`. +- The record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. ## Notes diff --git a/stage_descriptions/producing-messages-04-sb8.md b/stage_descriptions/producing-messages-04-sb8.md index 4d533e6..086ace1 100644 --- a/stage_descriptions/producing-messages-04-sb8.md +++ b/stage_descriptions/producing-messages-04-sb8.md @@ -25,17 +25,18 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic response matches the topic name in the request. -- The `index` field in the partition response matches the partition in the request. -- The `base_offset` field in the partition response contains 0 (the base offset for the batch). -- The `log_append_time_ms` field in the partition response contains `-1` (signifying that the timestamp is latest). -- The `log_start_offset` field in the partition response is `0`. - -The tester will also verify that the record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log` with sequential offsets. +- The topic response contains: + - The `name` field matches the topic name in the request. + - The partition response contains: + - The `index` field matches the partition in the request. + - The `base_offset` field contains 0 (the base offset for the batch). + - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - The `log_start_offset` field is `0`. +- The records are persisted to the appropriate log file on disk at `/-/00000000000000000000.log` with sequential offsets. ## Notes -- Records within a batch must be assigned sequential offsets (e.g., if base offset is 5, records get offsets 5, 6, 7). +- Records within a batch must be assigned sequential offsets (e.g., if the base offset is 5, records get offsets 5, 6, 7). - The response should return the base offset of the batch, not individual record offsets. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. diff --git a/stage_descriptions/producing-messages-05-mf2.md b/stage_descriptions/producing-messages-05-mf2.md index 0de155d..7061428 100644 --- a/stage_descriptions/producing-messages-05-mf2.md +++ b/stage_descriptions/producing-messages-05-mf2.md @@ -17,8 +17,7 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple partitions of the same topic. -The request will contain multiple RecordBatches, one for each partition. Each RecordBatch will contain a single record. +It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple partitions of the same topic. The request will contain multiple RecordBatches, one for each partition. Each RecordBatch will contain a single record. The tester will validate that: @@ -27,18 +26,17 @@ The tester will validate that: - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. - There is a single topic present in the response. -- The `name` field in the topic response matches the topic name in the request. -- Each partition in the request has a corresponding partition response. -- Each partition response contains: - - The correct `index` field matching the partition in the request. - - An error code of `0` (NO_ERROR). - - A valid `base_offset` field with the assigned offset for that partition. - - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is latest). - - The `log_start_offset` field is `0`. -- Records are persisted to the correct partition log files. -- Offset assignment is independent per partition (partition 0 and partition 1 can both have offset 0). - -The tester will also verify that the records are persisted to the correct partition log files. +- The topic response contains: + - The `name` field matches the topic name in the request. + - Each partition in the request has a corresponding partition response. + - Each partition response contains: + - The correct `index` field matching the partition in the request. + - An error code of `0` (NO_ERROR). + - A valid `base_offset` field with the assigned offset for that partition. + - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - The `log_start_offset` field is `0`. +- Records are persisted to the correct partition log files on disk. +- Offset assignment is independent per partition (partition 0 and partition 1 can both have offset 0). ## Notes diff --git a/stage_descriptions/producing-messages-06-ar4.md b/stage_descriptions/producing-messages-06-ar4.md index 1fd1fae..7857da8 100644 --- a/stage_descriptions/producing-messages-06-ar4.md +++ b/stage_descriptions/producing-messages-06-ar4.md @@ -17,8 +17,7 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple topics with their respective partitions. -The request will contain data for multiple topics, and a single partition for each topic. +It'll then connect to your server on port 9092 and send a `Produce` (v11) request targeting multiple topics with their respective partitions. The request will contain data for multiple topics, and a single partition for each topic. The tester will validate that: @@ -34,9 +33,9 @@ The tester will validate that: - The correct `index` field matching the partition in the request. - An error code of `0` (NO_ERROR). - A valid `base_offset` field with the assigned offset for that topic-partition. - - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is latest). + - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. -- Records are persisted to the correct topic-partition log files. +- Records are persisted to the correct topic-partition log files on disk. - Offset assignment is independent per topic-partition combination. ## Notes From e4168e880cb25e3560abb7c06487493486da35c5 Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 18 Jul 2025 00:48:24 +0530 Subject: [PATCH 24/52] fix: improve clarity in Produce stage descriptions by refining language on response structure and validation for invalid topics and partitions, ensuring consistency in error handling --- stage_descriptions/producing-messages-02-ck2.md | 14 +++++++------- stage_descriptions/producing-messages-03-ps7.md | 2 +- stage_descriptions/producing-messages-04-sb8.md | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/stage_descriptions/producing-messages-02-ck2.md b/stage_descriptions/producing-messages-02-ck2.md index 8c96dd6..97649b9 100644 --- a/stage_descriptions/producing-messages-02-ck2.md +++ b/stage_descriptions/producing-messages-02-ck2.md @@ -1,6 +1,6 @@ In this stage, you'll add support for handling Produce requests to invalid topics or partitions. -## Produce API response for invalid topics or partitions +## Produce API Response for Invalid Topics or Partitions When a Kafka broker receives a Produce request, it needs to validate that both the topic and partition exist. If either the topic or partition doesn't exist, it returns an appropriate error code and response. @@ -36,7 +36,7 @@ The tester will execute your program like this: ./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `Produce` (v11) request either with an invalid topic name or a valid topic but invalid partition. +It'll then connect to your server on port 9092 and send a `Produce` (v11) request with either an invalid topic name or a valid topic but invalid partition. The tester will validate that: @@ -45,12 +45,12 @@ The tester will validate that: - The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). - The `throttle_time_ms` field in the response is `0`. - The topic response contains: - - The `name` field should correspond to the topic name in the request. + - The `name` field corresponds to the topic name in the request. - The partition response contains: - - The `index` field should correspond to the partition in the request. - - The `base_offset` field should be `-1`. - - The `log_append_time_ms` field should be `-1`. - - The `log_start_offset` field should be `-1`. + - The `index` field corresponds to the partition in the request. + - The `base_offset` field is `-1`. + - The `log_append_time_ms` field is `-1`. + - The `log_start_offset` field is `-1`. ## Notes diff --git a/stage_descriptions/producing-messages-03-ps7.md b/stage_descriptions/producing-messages-03-ps7.md index 8e4d5a4..687e389 100644 --- a/stage_descriptions/producing-messages-03-ps7.md +++ b/stage_descriptions/producing-messages-03-ps7.md @@ -37,7 +37,7 @@ The tester will validate that: - The `name` field matches the topic name in the request. - The partition response contains: - The `index` field matches the partition in the request. - - The `base_offset` field contains the assigned offset to the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) + - The `base_offset` field contains the assigned offset for the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. - The record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. diff --git a/stage_descriptions/producing-messages-04-sb8.md b/stage_descriptions/producing-messages-04-sb8.md index 086ace1..6334416 100644 --- a/stage_descriptions/producing-messages-04-sb8.md +++ b/stage_descriptions/producing-messages-04-sb8.md @@ -40,4 +40,4 @@ The tester will validate that: - The response should return the base offset of the batch, not individual record offsets. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. -TODO: Send multiple request per stage from stage 5 onwards. (Need to test how baseOffset & log_start_offset are handled by kafka.) \ No newline at end of file +TODO: Send multiple requests per stage from stage 5 onwards. (Need to test how baseOffset & log_start_offset are handled by Kafka.) \ No newline at end of file From eb3199a1bf2138bdb0b899daef2554e33a4d6ed7 Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 18 Jul 2025 00:51:12 +0530 Subject: [PATCH 25/52] fix: refine language in Produce stage descriptions for consistency in error code representation and response structure validation --- stage_descriptions/producing-messages-01-um3.md | 2 +- stage_descriptions/producing-messages-02-ck2.md | 4 ++-- stage_descriptions/producing-messages-03-ps7.md | 2 +- stage_descriptions/producing-messages-04-sb8.md | 4 ++-- stage_descriptions/producing-messages-05-mf2.md | 4 ++-- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/stage_descriptions/producing-messages-01-um3.md b/stage_descriptions/producing-messages-01-um3.md index 7995cfc..d1b3ee5 100644 --- a/stage_descriptions/producing-messages-01-um3.md +++ b/stage_descriptions/producing-messages-01-um3.md @@ -25,7 +25,7 @@ The tester will validate that: - The first 4 bytes of your response (the "message length") are valid. - The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the response body is `0` (No Error). +- The error code in the response body is `0` (NO_ERROR). - The response body contains at least one entry for the API key `0` (Produce). - The `MaxVersion` for the Produce API is at least 11. diff --git a/stage_descriptions/producing-messages-02-ck2.md b/stage_descriptions/producing-messages-02-ck2.md index 97649b9..fb8848b 100644 --- a/stage_descriptions/producing-messages-02-ck2.md +++ b/stage_descriptions/producing-messages-02-ck2.md @@ -45,9 +45,9 @@ The tester will validate that: - The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). - The `throttle_time_ms` field in the response is `0`. - The topic response contains: - - The `name` field corresponds to the topic name in the request. + - The `name` field matches the topic name in the request. - The partition response contains: - - The `index` field corresponds to the partition in the request. + - The `index` field matches the partition in the request. - The `base_offset` field is `-1`. - The `log_append_time_ms` field is `-1`. - The `log_start_offset` field is `-1`. diff --git a/stage_descriptions/producing-messages-03-ps7.md b/stage_descriptions/producing-messages-03-ps7.md index 687e389..d1a4276 100644 --- a/stage_descriptions/producing-messages-03-ps7.md +++ b/stage_descriptions/producing-messages-03-ps7.md @@ -38,7 +38,7 @@ The tester will validate that: - The partition response contains: - The `index` field matches the partition in the request. - The `base_offset` field contains the assigned offset for the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) - - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. - The record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. diff --git a/stage_descriptions/producing-messages-04-sb8.md b/stage_descriptions/producing-messages-04-sb8.md index 6334416..3ae9f4e 100644 --- a/stage_descriptions/producing-messages-04-sb8.md +++ b/stage_descriptions/producing-messages-04-sb8.md @@ -29,8 +29,8 @@ The tester will validate that: - The `name` field matches the topic name in the request. - The partition response contains: - The `index` field matches the partition in the request. - - The `base_offset` field contains 0 (the base offset for the batch). - - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - The `base_offset` field is 0 (the base offset for the batch). + - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. - The records are persisted to the appropriate log file on disk at `/-/00000000000000000000.log` with sequential offsets. diff --git a/stage_descriptions/producing-messages-05-mf2.md b/stage_descriptions/producing-messages-05-mf2.md index 7061428..5df0bf0 100644 --- a/stage_descriptions/producing-messages-05-mf2.md +++ b/stage_descriptions/producing-messages-05-mf2.md @@ -31,9 +31,9 @@ The tester will validate that: - Each partition in the request has a corresponding partition response. - Each partition response contains: - The correct `index` field matching the partition in the request. - - An error code of `0` (NO_ERROR). - A valid `base_offset` field with the assigned offset for that partition. - - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - The error code is `0` (NO_ERROR). + - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. - Records are persisted to the correct partition log files on disk. - Offset assignment is independent per partition (partition 0 and partition 1 can both have offset 0). From 1c71d02c0c70e474b7bb5e51271bebe2caaf52cc Mon Sep 17 00:00:00 2001 From: Ryan Date: Fri, 18 Jul 2025 00:59:44 +0530 Subject: [PATCH 26/52] fix: enhance clarity in Produce stage descriptions by refining language on response structure and validation for multiple records and cross-topic production, ensuring consistency in error handling and log file persistence --- stage_descriptions/producing-messages-02-ck2.md | 6 +++--- stage_descriptions/producing-messages-03-ps7.md | 15 +++++++-------- stage_descriptions/producing-messages-04-sb8.md | 10 ++++------ stage_descriptions/producing-messages-05-mf2.md | 10 +++++----- stage_descriptions/producing-messages-06-ar4.md | 16 ++++++++-------- 5 files changed, 27 insertions(+), 30 deletions(-) diff --git a/stage_descriptions/producing-messages-02-ck2.md b/stage_descriptions/producing-messages-02-ck2.md index fb8848b..5b60d16 100644 --- a/stage_descriptions/producing-messages-02-ck2.md +++ b/stage_descriptions/producing-messages-02-ck2.md @@ -26,7 +26,7 @@ We've created an interactive protocol inspector for the request & response struc We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: - 🔎 [__cluster_metadata log file](https://binspec.org/kafka-cluster-metadata) -In this stage, you'll need to implement the response for a `Produce` request with either an invalid topic or invalid partition. In later stages, you'll handle successfully producing messages to valid topics and partitions, persist messages to disk using Kafka's RecordBatch format, manage offset assignment, handle batch processing, and support multiple partitions and topics. +In this stage, you'll need to implement the response for a `Produce` request with either an invalid topic or invalid partition. In later stages, you'll handle successfully producing messages to valid topics and partitions and persist messages to disk using Kafka's RecordBatch format. ## Tests @@ -44,9 +44,9 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The `error_code` in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). - The `throttle_time_ms` field in the response is `0`. -- The topic response contains: +- Inside the topic response: - The `name` field matches the topic name in the request. - - The partition response contains: + - Inside the partition response: - The `index` field matches the partition in the request. - The `base_offset` field is `-1`. - The `log_append_time_ms` field is `-1`. diff --git a/stage_descriptions/producing-messages-03-ps7.md b/stage_descriptions/producing-messages-03-ps7.md index d1a4276..d0bb75b 100644 --- a/stage_descriptions/producing-messages-03-ps7.md +++ b/stage_descriptions/producing-messages-03-ps7.md @@ -2,7 +2,7 @@ In this stage, you'll add support for successfully producing a single record. ## Single Record Production -When a Kafka broker receives a Produce request for a valid topic and partition, it needs to validate that the topic and partition exist (using the `__cluster_metadata` topic's log file), store the record in the appropriate log file using Kafka's on-disk format, and return a successful response with the assigned offset. +When a Kafka broker receives a Produce request, it needs to validate that the topic and partition exist (using the `__cluster_metadata` topic's log file), store the record in the appropriate log file using Kafka's on-disk format, and return a successful response with the assigned offset. The record must be persisted to the topic's log file at `/-/00000000000000000000.log` using Kafka's RecordBatch format. @@ -11,12 +11,11 @@ We've created an interactive protocol inspector for the request & response struc - 🔎 [Produce Request (v11)](https://binspec.org/kafka-produce-request-v11) - 🔎 [Produce Response (v11)](https://binspec.org/kafka-produce-response-v11) -Kafka's on-disk log format is just records inside a RecordBatch. The same RecordBatch format that is used in the Produce request and Fetch request is also used in the on-disk log file. +Kafka's on-disk log format is just records inside a `RecordBatch`. The same RecordBatch format that is used in the `Produce` request and `Fetch` request is also used in the on-disk log file. You can refer to the following interactive protocol inspector for Kafka's log file format: - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) - ## Tests The tester will execute your program like this: @@ -33,17 +32,17 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. -- The topic response contains: +- Inside the topic response: - The `name` field matches the topic name in the request. - - The partition response contains: + - Inside the partition response: - The `index` field matches the partition in the request. - - The `base_offset` field contains the assigned offset for the record. (The offset is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) + - The `base_offset` field contains the assigned offset for the record. (The `base_offset` is the offset of the record in the partition, not the offset of the batch. So 0 for the first record, 1 for the second record, and so on.) - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. - The record is persisted to the appropriate log file on disk at `/-/00000000000000000000.log`. ## Notes -- On-disk log files must be stored in RecordBatch format with proper CRC validation. -- The offset assignment should start from 0 for new partitions and increment for each subsequent record. +- On-disk log files must be stored in `RecordBatch` format with proper CRC validation. +- The offset assignment should start from `0` for new partitions and increment for each subsequent record. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. diff --git a/stage_descriptions/producing-messages-04-sb8.md b/stage_descriptions/producing-messages-04-sb8.md index 3ae9f4e..ba2bc0d 100644 --- a/stage_descriptions/producing-messages-04-sb8.md +++ b/stage_descriptions/producing-messages-04-sb8.md @@ -1,8 +1,8 @@ -In this stage, you'll add support for producing multiple records from a single request, with multiple records inside a single RecordBatch. +In this stage, you'll add support for producing multiple records from a single request. ## Batch Processing -When a Kafka broker receives a Produce request containing a RecordBatch with multiple records, it needs to validate that the topic and partition exist, assign sequential offsets to each record within the batch, and store the entire batch atomically to the log file. The RecordBatch containing multiple records must be stored as a single unit to the topic's log file at `/-/00000000000000000000.log`. +When a Kafka broker receives a `Produce` request containing a `RecordBatch` with multiple records, it needs to validate that the topic and partition exist, assign sequential offsets to each record within the batch, and store the entire batch atomically to the log file. The `RecordBatch` containing multiple records must be stored as a single unit to the topic's log file at `/-/00000000000000000000.log`. We've created an interactive protocol inspector for the request & response structures for `Produce`: @@ -25,9 +25,9 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. -- The topic response contains: +- Inside the topic response: - The `name` field matches the topic name in the request. - - The partition response contains: + - Inside the partition response: - The `index` field matches the partition in the request. - The `base_offset` field is 0 (the base offset for the batch). - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). @@ -39,5 +39,3 @@ The tester will validate that: - Records within a batch must be assigned sequential offsets (e.g., if the base offset is 5, records get offsets 5, 6, 7). - The response should return the base offset of the batch, not individual record offsets. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. - -TODO: Send multiple requests per stage from stage 5 onwards. (Need to test how baseOffset & log_start_offset are handled by Kafka.) \ No newline at end of file diff --git a/stage_descriptions/producing-messages-05-mf2.md b/stage_descriptions/producing-messages-05-mf2.md index 5df0bf0..09601f2 100644 --- a/stage_descriptions/producing-messages-05-mf2.md +++ b/stage_descriptions/producing-messages-05-mf2.md @@ -26,12 +26,12 @@ The tester will validate that: - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. - There is a single topic present in the response. -- The topic response contains: +- Inside the topic response: - The `name` field matches the topic name in the request. - Each partition in the request has a corresponding partition response. - - Each partition response contains: - - The correct `index` field matching the partition in the request. - - A valid `base_offset` field with the assigned offset for that partition. + - Inside each partition response: + - The `index` field matches the partition in the request. + - The `base_offset` field contains the assigned offset for that partition. - The error code is `0` (NO_ERROR). - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. @@ -41,5 +41,5 @@ The tester will validate that: ## Notes - The response must include entries for all requested partitions. -- On-disk log files must be stored in RecordBatch format with proper CRC validation. +- On-disk log files must be stored in `RecordBatch` format with proper CRC validation. - The official docs for the `Produce` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_Produce). Make sure to scroll down to the "Produce Response (Version: 11)" section. diff --git a/stage_descriptions/producing-messages-06-ar4.md b/stage_descriptions/producing-messages-06-ar4.md index 7857da8..e309c1c 100644 --- a/stage_descriptions/producing-messages-06-ar4.md +++ b/stage_descriptions/producing-messages-06-ar4.md @@ -2,7 +2,7 @@ In this stage, you'll add support for producing to multiple topics in a single r ## Cross-Topic Production -When a Kafka broker receives a Produce request targeting multiple topics with their respective partitions, it needs to validate that all topics and partitions exist, write records to each topic-partition's log file independently, and return a nested response structure containing results for all topics and their partitions. Each topic-partition combination maintains its own independent offset sequence, so topic "foo" partition 0 and topic "bar" partition 0 can both have records starting at offset 0. +When a Kafka broker receives a `Produce` request targeting multiple topics with their respective partitions, it needs to validate that all topics and partitions exist, write records to each topic-partition's log file independently, and return a nested response structure containing results for all topics and their partitions. Each topic-partition combination maintains its own independent offset sequence, so topic "foo" partition 0 and topic "bar" partition 0 can both have records starting at offset 0. We've created an interactive protocol inspector for the request & response structures for `Produce`: @@ -26,14 +26,14 @@ The tester will validate that: - The error code in the response body is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. - Each topic in the request has a corresponding topic response. -- Each topic response contains: - - The correct `name` field matching the topic name in the request. +- Inside each topic response: + - The `name` field matches the topic name in the request. - Each partition in the request has a corresponding partition response. - - Each partition response contains: - - The correct `index` field matching the partition in the request. - - An error code of `0` (NO_ERROR). - - A valid `base_offset` field with the assigned offset for that topic-partition. - - The `log_append_time_ms` field contains `-1` (signifying that the timestamp is the latest). + - Inside each partition response: + - The `index` field matches the partition in the request. + - The error code is `0` (NO_ERROR). + - The `base_offset` field contains the assigned offset for that topic-partition. + - The `log_append_time_ms` field is `-1` (signifying that the timestamp is the latest). - The `log_start_offset` field is `0`. - Records are persisted to the correct topic-partition log files on disk. - Offset assignment is independent per topic-partition combination. From d51f956f31bd71acc32bea4bd391cd9386abb7f6 Mon Sep 17 00:00:00 2001 From: Ryan Date: Mon, 14 Jul 2025 19:03:37 +0530 Subject: [PATCH 27/52] wip: add detailed breakdown for CreateTopics API stages and error handling Introduce two new markdown files outlining the structure and error cases for the CreateTopics API in Kafka. The first file provides a logical progression of stages for implementing the API, while the second file details early error cases that can be handled with request parsing and response encoding. This aligns with existing documentation practices and enhances clarity for future development. --- stage_descriptions/create-topics.md | 73 ++++ stage_descriptions/create-topics2.md | 555 +++++++++++++++++++++++++++ 2 files changed, 628 insertions(+) create mode 100644 stage_descriptions/create-topics.md create mode 100644 stage_descriptions/create-topics2.md diff --git a/stage_descriptions/create-topics.md b/stage_descriptions/create-topics.md new file mode 100644 index 0000000..0eb247a --- /dev/null +++ b/stage_descriptions/create-topics.md @@ -0,0 +1,73 @@ +Based on my analysis of the codebase, I can see how the produce extension was broken down + into logical stages and provide recommendations for structuring the create-topics extension + similarly. + + How Produce Extension Stages Were Organized + + The produce extension follows this logical progression: + + P1-P2: Foundation & Discovery + - P1: Include Produce in APIVersions response + - P2: Handle unknown topics (error handling) + + P3-P4: Basic Validation & Core Functionality + - P3: Handle invalid partitions (error handling) + - P4: Single record produce (basic success case) + + P5-P7: Increasing Complexity + - P5: Multiple messages in single batch + - P6: Multiple partitions of same topic + - P7: Multiple topics in single request + + Recommended Create-Topics Extension Structure + + Following the same pattern, here's how create-topics could be broken down: + + Stage CT1: Include CreateTopics in APIVersions + + - Add API key 19 (CreateTopics) to APIVersions response + - Foundation stage enabling API discovery + + Stage CT2: CreateTopics with Invalid Topic Name + + - Handle invalid characters, reserved names, empty names + - Error code: INVALID_TOPIC_EXCEPTION (17) + + Stage CT3: CreateTopics with Reserved Topic Name + + - Handle attempts to create system topics (__cluster_metadata) + - Error code: INVALID_REQUEST (42) + + Stage CT4: CreateTopics with Existing Topic Name + + - Handle duplicate topic creation attempts + - Error code: TOPIC_ALREADY_EXISTS (36) + + Stage CT5: CreateTopics with Invalid Parameters + + - Handle negative/zero partitions, invalid replication factors + - Error codes: INVALID_PARTITIONS (37), INVALID_REPLICATION_FACTOR (38) + + Stage CT6: CreateTopics with Valid Parameters (Success Case) + + - Successfully create single topic with valid parameters + - Core success functionality + + Optional Advanced Stages: + + - CT7: Multiple topics in single request + - CT8: Topic-specific configurations + - CT9: Validation-only mode + + Key Differences from Produce Extension + + 1. Administrative vs Operational: CreateTopics affects cluster metadata permanently + 2. Complex Validation: Topic names, parameters, system restrictions + 3. Different Error Codes: Specific to topic creation scenarios + 4. Metadata Persistence: Updates cluster metadata, not message logs + + This structure provides a logical progression from API discovery through comprehensive error + handling to successful topic creation, following the established patterns while addressing + the unique requirements of the CreateTopics API. + + diff --git a/stage_descriptions/create-topics2.md b/stage_descriptions/create-topics2.md new file mode 100644 index 0000000..bcdc72e --- /dev/null +++ b/stage_descriptions/create-topics2.md @@ -0,0 +1,555 @@ +╭───────────────────────────────────────────────────╮ +│ ✻ Welcome to Claude Code! │ +│ │ +│ /help for help, /status for your current setup │ +│ │ +│ cwd: /Users/ryang/Developer/tmp/kafka/kafka │ +╰───────────────────────────────────────────────────╯ + +> How does the CreateTopic API work ? + +⏺ I'll help you understand how the CreateTopic API works in Kafka. Let me search through the codebase to find + the relevant implementation. + +⏺ Task(Search CreateTopic API implementation) + ⎿ Done (37 tool uses · 50.2k tokens · 5m 20.9s) + +⏺ The CreateTopic API in Kafka follows a controller-based architecture with these key components: + + Core Flow: + 1. Client sends CreateTopicsRequest to the controller broker + 2. Controller validates permissions, configurations, and partition assignments + 3. Controller writes metadata records to the Raft log + 4. Metadata propagates to all brokers + 5. Response returns success/failure for each topic + + Key Files: + - clients/src/main/java/org/apache/kafka/clients/admin/Admin.java - Client interface + - clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java - Request class + - core/src/main/scala/kafka/server/ControllerApis.scala - Request handler + - metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java - Core logic + + Validation Steps: + - Topic name validation and conflict checking + - Partition count and replication factor validation + - Authorization (CREATE permissions) + - Configuration policy enforcement + - Quota and rate limiting + + Partition Assignment: + - Manual: Use provided replica assignments + - Automatic: ReplicaPlacer distributes across available brokers + - Validates ISR availability + + The API supports both synchronous validation (validateOnly=true) and asynchronous creation with configurable + timeouts. + +> What are some error cases of the API, that I can implement early before touching the core logic ? + +⏺ Task(Find CreateTopic API error cases) + ⎿ Done (38 tool uses · 52.6k tokens · 4m 40.6s) + +⏺ Here are the early validation error cases you can implement before touching core logic: + + Input Validation Errors (Immediate Validation) + + 1. Topic Name Validation - INVALID_TOPIC_EXCEPTION (17) + - Empty topic name or "." or ".." + - Length > 249 characters + - Invalid characters (only ASCII alphanumerics, '.', '_', '-') + - Internal topic collision detection + + 2. Partition Count Validation - INVALID_PARTITIONS (37) + - Negative or zero partition count + - Message: "Number of partitions was set to an invalid non-positive value." + + 3. Replication Factor Validation - INVALID_REPLICATION_FACTOR (38) + - Negative or zero replication factor + - Message: "Replication factor must be larger than 0, or -1 to use the default value." + + 4. Request Format Validation - INVALID_REQUEST (42) + - Duplicate topic names in single request + - Version compatibility issues (validateOnly in v0, defaults in v<4) + + 5. Manual Assignment Validation - INVALID_REPLICA_ASSIGNMENT (39) + - Assignment with non-(-1) replication factor/partition count + - Non-consecutive partition indices + - Empty replica lists + - Duplicate broker IDs in assignment + + 6. Configuration Validation - INVALID_CONFIG (40) + - Null config values + - Message: "Null value not supported for topic configs" + + 7. Authorization Check - TOPIC_AUTHORIZATION_FAILED (29) + - No CREATE permission on CLUSTER or TOPIC resource + + 8. Existing Topic Check - TOPIC_ALREADY_EXISTS (36) + - Topic name already exists + + 9. Internal Topic Protection - INVALID_REQUEST (42) + - Attempting to create __cluster_metadata topic + + These validations happen in ControllerApis.scala:396-442 and ReplicationControlManager.java:635-765 before any + core topic creation logic, making them perfect for early error detection. + +> Yep these are just validations tho. As part of the core logic. I was talking more about when we send a PRODUCE + request with an non-existent topic it just returns an error. So we have to parse the request, encode the + response but no core logic. Something similar here ? + +⏺ Ah, I understand now! You're looking for request parsing and response encoding error cases that happen at the + protocol level, before any business logic. Let me search for those. + +⏺ Task(Find protocol-level CreateTopic errors) + ⎿ Done (33 tool uses · 69.6k tokens · 2m 57.4s) + +⏺ Perfect! Now I understand what you're looking for. Here are the protocol-level error cases for CreateTopic + that you can implement with just request parsing and response encoding: + + Protocol-Level Error Cases (No Core Logic) + + 1. Request Deserialization Errors - INVALID_REQUEST (42) + - Malformed request buffer (corrupted array lengths, invalid byte arrays) + - Buffer underflow/overflow during parsing + - Invalid field types or corrupted data + + 2. Version Mismatch Errors - UNSUPPORTED_VERSION (35) + - Using validateOnly with version 0 + - Using default partitions/replication factor with version < 4 + - Requesting unsupported API version + + 3. Message Size Limit - MESSAGE_TOO_LARGE (10) + - Request exceeds broker's message.max.bytes limit + - Happens during request parsing before any business logic + + 4. Required Field Validation - INVALID_REQUEST (42) + - Missing topic name in request + - Invalid request structure per protocol schema + + 5. Array Size Validation - INVALID_REQUEST (42) + - Malformed array lengths that would cause allocation errors + - Prevents oversized collection allocation + + The key files for implementing these: + - clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java:96-109 - getErrorResponse() + method + - clients/src/main/java/org/apache/kafka/common/requests/RequestContext.java:119-129 - request parsing + - clients/src/main/resources/common/message/CreateTopicsRequest.json - protocol schema + + These errors are caught during request deserialization and can return proper CreateTopicsResponse with + per-topic error codes, just like how PRODUCE returns errors for non-existent topics - no business logic + required! + +> Nope. See this doc. /Users/ryang/Developer/work/course-testers/kafka-tester/Kafka-breakdown.md\ + I need something similar for the CreateTopic API. + +⏺ Read(../../../work/course-testers/kafka-tester/Kafka-breakdown.md) + ⎿ Read 327 lines (ctrl+r to expand) + +⏺ Perfect! Now I understand what you're looking for. You want early error cases for the CreateTopic API that can + be implemented with just request parsing and response encoding, similar to how the Produce API handles + non-existent topics. + + Here's what I would suggest for CreateTopic API stages: + + CreateTopic API Early Error Cases + + Stage 1: Include CreateTopic in APIVersions + - Add CreateTopic API (key=19) to APIVersions response + - MaxVersion should be at least 7 + - No actual CreateTopic logic needed + + Stage 2: CreateTopic with Invalid Topic Name + - Parse CreateTopic request to extract topic name + - Check for invalid topic names (empty, ".", "..", invalid chars) + - Return INVALID_TOPIC_EXCEPTION (17) error response + - No core topic creation logic needed + + Stage 3: CreateTopic with Invalid Partition/Replication Values + - Parse partition count and replication factor from request + - Return INVALID_PARTITIONS (37) for partition count ≤ 0 + - Return INVALID_REPLICATION_FACTOR (38) for replication factor ≤ 0 + - Just request parsing + response encoding + + Stage 4: CreateTopic with Existing Topic + - Check if topic directory already exists in log directory + - Return TOPIC_ALREADY_EXISTS (36) if found + - Simple filesystem check, no metadata parsing needed + + Stage 5: CreateTopic with Authorization Check + - Return TOPIC_AUTHORIZATION_FAILED (29) based on simple config + - No real authorization logic, just return the error code + - Test the response structure + + These stages would let you implement and test: + - Request parsing for CreateTopic + - Response encoding with proper error codes + - Per-topic error handling in responses + - Basic filesystem checks + + Just like the Produce API stages, these focus on protocol handling and basic validation before any complex + topic creation logic. + +> Can you improve this ? + +⏺ Update Todos + ⎿ ☐ Analyze CreateTopic API protocol structure and error cases + ☐ Design progressive stages for CreateTopic API implementation + ☐ Write detailed stage descriptions with test cases + +⏺ Task(Analyze CreateTopic protocol details) + ⎿ Done (25 tool uses · 47.2k tokens · 2m 28.4s) + +⏺ Update Todos + ⎿ ☒ Analyze CreateTopic API protocol structure and error cases + ☐ Design progressive stages for CreateTopic API implementation + ☐ Write detailed stage descriptions with test cases + +⏺ Update Todos + ⎿ ☒ Analyze CreateTopic API protocol structure and error cases + ☒ Design progressive stages for CreateTopic API implementation + ☐ Write detailed stage descriptions with test cases + +⏺ Kafka / CreateTopics API + + Stage 1: Include CreateTopics in APIVersions + + In this stage, you'll add an entry for the CreateTopics API to the APIVersions response. + + APIVersions + + Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before + implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics + API. + + Tests + + The tester will execute your program like this: + + ./your_program.sh + + It'll then connect to your server on port 9092 and send a valid APIVersions (v4) request. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the response body is 0 (No Error). + - The response body contains at least one entry for the API key 19 (CreateTopics). + - The MaxVersion for the CreateTopics API is at least 7. + + Notes + + - You don't have to implement support for the CreateTopics request in this stage. We'll get to this in later + stages. + - You'll still need to include the entry for APIVersions in your response to pass the previous stage. + + Stage 2: CreateTopics with Invalid Topic Names + + In this stage, you'll add support for handling CreateTopics requests with invalid topic names and return + proper error responses. + + Handling Invalid Topic Names + + Your Kafka implementation should validate topic names and return appropriate error codes when a client tries + to create topics with invalid names. Topic names must follow these rules: + - Cannot be empty string + - Cannot be "." or ".." + - Maximum length of 249 characters + - Only ASCII alphanumerics, '.', '_', and '-' are allowed + - Cannot have period/underscore collision patterns + + Tests + + The tester will execute your program like this: + + ./your_program.sh + + It'll then connect to your server on port 9092 and send a CreateTopics (v7) request with an invalid topic + name. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the topic result is 17 (INVALID_TOPIC_EXCEPTION). + - The throttle_time_ms field in the response is 0. + - The name field in the topic result corresponds to the topic name in the request. + - The error_message field contains a descriptive error message. + + Notes + + - You'll need to parse the CreateTopics request in this stage to get the topic names. + - The official docs for the CreateTopics request can be found + https://kafka.apache.org/protocol.html#The_Messages_CreateTopics. + - Topic name validation logic is in the Kafka source code at + clients/src/main/java/org/apache/kafka/common/internals/Topic.java. + + Stage 3: CreateTopics with Invalid Partition/Replication Parameters + + In this stage, you'll add support for handling CreateTopics requests with invalid partition counts and + replication factors. + + Handling Invalid Parameters + + Your Kafka implementation should validate numeric parameters and return appropriate errors: + - Partition count must be greater than 0 (or -1 for defaults) + - Replication factor must be greater than 0 (or -1 for defaults) + - Both parameters cannot be 0 or negative (except -1) + + Tests + + The tester will execute your program like this: + + ./your_program.sh + + It'll then connect to your server on port 9092 and send CreateTopics (v7) requests with invalid parameters. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the topic result is 37 (INVALID_PARTITIONS) for invalid partition count. + - The error code in the topic result is 38 (INVALID_REPLICATION_FACTOR) for invalid replication factor. + - The throttle_time_ms field in the response is 0. + - The name field in the topic result corresponds to the topic name in the request. + - The error_message field contains a descriptive error message. + + Notes + + - Test cases will include: partition count = 0, partition count = -2, replication factor = 0, replication + factor = -2. + - Valid values are positive integers or -1 (for broker defaults). + + Stage 4: CreateTopics with Existing Topics + + In this stage, you'll add support for detecting when a topic already exists and return the appropriate error + response. + + Handling Existing Topics + + Your Kafka implementation should check if a topic already exists before attempting to create it. You can check + for existing topics by: + - Looking for topic directories in the log directory (e.g., /tmp/kraft-combined-logs/topic-name-0/) + - Reading the __cluster_metadata topic for TopicRecord entries + + Tests + + The tester will execute your program like this: + + ./your_program.sh + # Create a topic first + # Then try to create the same topic again + + It'll then connect to your server on port 9092 and send a CreateTopics (v7) request for an existing topic. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the topic result is 36 (TOPIC_ALREADY_EXISTS). + - The throttle_time_ms field in the response is 0. + - The name field in the topic result corresponds to the topic name in the request. + - The error_message field contains a descriptive error message. + + Notes + + - This stage requires basic topic existence checking without full topic creation logic. + - You can use filesystem checks or metadata parsing to detect existing topics. + + Stage 5: CreateTopics with Invalid Manual Assignment + + In this stage, you'll add support for validating manual partition assignments and return appropriate error + responses. + + Handling Invalid Manual Assignment + + Your Kafka implementation should validate manual partition assignments when provided: + - Manual assignment requires partition count and replication factor to be -1 + - Partition indices must be consecutive starting from 0 + - Replica lists cannot be empty + - Broker IDs must be valid (though you can mock this validation) + - No duplicate broker IDs within a partition assignment + + Tests + + The tester will execute your program like this: + + ./your_program.sh + + It'll then connect to your server on port 9092 and send CreateTopics (v7) requests with invalid manual + assignments. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the topic result is 39 (INVALID_REPLICA_ASSIGNMENT). + - The throttle_time_ms field in the response is 0. + - The name field in the topic result corresponds to the topic name in the request. + - The error_message field contains a descriptive error message. + + Notes + + - Test cases include: non-(-1) partition count with assignment, non-consecutive partitions, empty replica + lists, duplicate broker IDs. + - Manual assignment validation is complex but can be implemented without broker state. + + Stage 6: CreateTopics with Authorization Failure + + In this stage, you'll add support for authorization checks and return appropriate error responses. + + Handling Authorization + + Your Kafka implementation should check authorization before creating topics: + - Require CREATE permission on CLUSTER resource or specific TOPIC resource + - You can implement a simple authorization check based on configuration or mock it + - Return authorization failure for unauthorized requests + + Tests + + The tester will execute your program like this: + + ./your_program.sh + + It'll then connect to your server on port 9092 and send CreateTopics (v7) requests without proper + authorization. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the topic result is 29 (TOPIC_AUTHORIZATION_FAILED). + - The throttle_time_ms field in the response is 0. + - The name field in the topic result corresponds to the topic name in the request. + - The error_message field contains "Authorization failed." + + Notes + + - This stage focuses on authorization error handling rather than implementing full ACL logic. + - You can use a simple configuration-based approach to determine authorization. + + --- + Setup Commands + + # Clean environment + rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs + + # Generate cluster ID and format storage + KAFKA_CLUSTER_ID="$(/usr/local/kafka-latest/bin/kafka-storage.sh random-uuid)" + /usr/local/kafka-latest/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c + /usr/local/kafka-latest/config/kraft/server.properties + + # Start Kafka server + /usr/local/kafka-latest/bin/kafka-server-start.sh /usr/local/kafka-latest/config/kraft/server.properties + + # Test with Kafka CLI + /usr/local/kafka-latest/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 + --partitions 3 + + Testing Commands + + # Build tester + make build + + # Run individual test + go run main.go + + # Verify with Kafka CLI + /usr/local/kafka-latest/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 + + Error Code Reference + + | Code | Name | Stages | + |------|----------------------------|---------------| + | 0 | NO_ERROR | Success cases | + | 17 | INVALID_TOPIC_EXCEPTION | 2 | + | 29 | TOPIC_AUTHORIZATION_FAILED | 6 | + | 36 | TOPIC_ALREADY_EXISTS | 4 | + | 37 | INVALID_PARTITIONS | 3 | + | 38 | INVALID_REPLICATION_FACTOR | 3 | + | 39 | INVALID_REPLICA_ASSIGNMENT | 5 | + +⏺ Update Todos + ⎿ ☒ Analyze CreateTopic API protocol structure and error cases + ☒ Design progressive stages for CreateTopic API implementation + ☒ Write detailed stage descriptions with test cases + +⏺ Here's an improved version of the CreateTopics API breakdown that follows the same progressive structure as + your Produce API document, with early error cases that can be implemented with just request parsing and + response encoding before touching any core logic. + +> Please write to a file ? + +⏺ Write(CreateTopics-breakdown.md) + ⎿ Wrote 258 lines to CreateTopics-breakdown.md + # Kafka / CreateTopics API + ## Stage 1: Include CreateTopics in APIVersions + In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response. + ### APIVersions + Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before + implementing topic creation functionality. This lets the client know that the broker supports the + CreateTopics API. + … +248 lines (ctrl+r to expand) + +⏺ Created the CreateTopics API breakdown document at + /Users/ryang/Developer/tmp/kafka/kafka/CreateTopics-breakdown.md. The document follows the same progressive + structure as your Produce API breakdown, with 6 stages that focus on early error cases that can be implemented + with just request parsing and response encoding before touching any core topic creation logic. + +> What is the manual assignment thing ? + +⏺ Task(Explain manual assignment in CreateTopics) + ⎿ Done (25 tool uses · 57.3k tokens · 2m 56.6s) + +⏺ Manual assignment in CreateTopics lets you explicitly specify which brokers should host each partition's + replicas, instead of letting Kafka automatically place them. + + Manual vs Automatic Assignment + + Automatic (Default): + - You specify numPartitions: 3 and replicationFactor: 2 + - Kafka automatically distributes partitions across brokers + + Manual: + - You specify exactly which brokers host each partition + - Set numPartitions: -1 and replicationFactor: -1 + - Provide an assignments array + + Manual Assignment Structure + + { + "assignments": [ + {"partitionIndex": 0, "brokerIds": [1, 2]}, // Partition 0 on brokers 1,2 + {"partitionIndex": 1, "brokerIds": [2, 3]}, // Partition 1 on brokers 2,3 + {"partitionIndex": 2, "brokerIds": [3, 1]} // Partition 2 on brokers 3,1 + ] + } + + When It's Used + + - Rack awareness: Ensure replicas are on different racks + - Performance optimization: Place partitions on specific hardware + - Compliance: Meet data locality requirements + - Migration: Control placement when moving topics + + Validation Rules + + - Partition indices must be consecutive (0, 1, 2...) + - All partitions must have same replication factor + - No duplicate brokers within a partition + - Brokers must exist in the cluster + - Can't mix manual assignment with numPartitions/replicationFactor + + The CLI equivalent is: + kafka-topics.sh --create --topic my-topic --replica-assignment "1:2,2:3,3:1" + + This gives you precise control over partition placement instead of relying on Kafka's automatic distribution + algorithm. + +╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ > Try "fix lint errors" │ +╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ + ? for shortcuts + + From 08fb9b806f1c93821df9b6a6436022ceffb3f93f Mon Sep 17 00:00:00 2001 From: Ryan Date: Wed, 16 Jul 2025 22:12:10 +0530 Subject: [PATCH 28/52] chore: remove outdated markdown files for CreateTopics API stages and error handling. --- stage_descriptions/create-topics.md | 73 ---- stage_descriptions/create-topics2.md | 555 --------------------------- 2 files changed, 628 deletions(-) delete mode 100644 stage_descriptions/create-topics.md delete mode 100644 stage_descriptions/create-topics2.md diff --git a/stage_descriptions/create-topics.md b/stage_descriptions/create-topics.md deleted file mode 100644 index 0eb247a..0000000 --- a/stage_descriptions/create-topics.md +++ /dev/null @@ -1,73 +0,0 @@ -Based on my analysis of the codebase, I can see how the produce extension was broken down - into logical stages and provide recommendations for structuring the create-topics extension - similarly. - - How Produce Extension Stages Were Organized - - The produce extension follows this logical progression: - - P1-P2: Foundation & Discovery - - P1: Include Produce in APIVersions response - - P2: Handle unknown topics (error handling) - - P3-P4: Basic Validation & Core Functionality - - P3: Handle invalid partitions (error handling) - - P4: Single record produce (basic success case) - - P5-P7: Increasing Complexity - - P5: Multiple messages in single batch - - P6: Multiple partitions of same topic - - P7: Multiple topics in single request - - Recommended Create-Topics Extension Structure - - Following the same pattern, here's how create-topics could be broken down: - - Stage CT1: Include CreateTopics in APIVersions - - - Add API key 19 (CreateTopics) to APIVersions response - - Foundation stage enabling API discovery - - Stage CT2: CreateTopics with Invalid Topic Name - - - Handle invalid characters, reserved names, empty names - - Error code: INVALID_TOPIC_EXCEPTION (17) - - Stage CT3: CreateTopics with Reserved Topic Name - - - Handle attempts to create system topics (__cluster_metadata) - - Error code: INVALID_REQUEST (42) - - Stage CT4: CreateTopics with Existing Topic Name - - - Handle duplicate topic creation attempts - - Error code: TOPIC_ALREADY_EXISTS (36) - - Stage CT5: CreateTopics with Invalid Parameters - - - Handle negative/zero partitions, invalid replication factors - - Error codes: INVALID_PARTITIONS (37), INVALID_REPLICATION_FACTOR (38) - - Stage CT6: CreateTopics with Valid Parameters (Success Case) - - - Successfully create single topic with valid parameters - - Core success functionality - - Optional Advanced Stages: - - - CT7: Multiple topics in single request - - CT8: Topic-specific configurations - - CT9: Validation-only mode - - Key Differences from Produce Extension - - 1. Administrative vs Operational: CreateTopics affects cluster metadata permanently - 2. Complex Validation: Topic names, parameters, system restrictions - 3. Different Error Codes: Specific to topic creation scenarios - 4. Metadata Persistence: Updates cluster metadata, not message logs - - This structure provides a logical progression from API discovery through comprehensive error - handling to successful topic creation, following the established patterns while addressing - the unique requirements of the CreateTopics API. - - diff --git a/stage_descriptions/create-topics2.md b/stage_descriptions/create-topics2.md deleted file mode 100644 index bcdc72e..0000000 --- a/stage_descriptions/create-topics2.md +++ /dev/null @@ -1,555 +0,0 @@ -╭───────────────────────────────────────────────────╮ -│ ✻ Welcome to Claude Code! │ -│ │ -│ /help for help, /status for your current setup │ -│ │ -│ cwd: /Users/ryang/Developer/tmp/kafka/kafka │ -╰───────────────────────────────────────────────────╯ - -> How does the CreateTopic API work ? - -⏺ I'll help you understand how the CreateTopic API works in Kafka. Let me search through the codebase to find - the relevant implementation. - -⏺ Task(Search CreateTopic API implementation) - ⎿ Done (37 tool uses · 50.2k tokens · 5m 20.9s) - -⏺ The CreateTopic API in Kafka follows a controller-based architecture with these key components: - - Core Flow: - 1. Client sends CreateTopicsRequest to the controller broker - 2. Controller validates permissions, configurations, and partition assignments - 3. Controller writes metadata records to the Raft log - 4. Metadata propagates to all brokers - 5. Response returns success/failure for each topic - - Key Files: - - clients/src/main/java/org/apache/kafka/clients/admin/Admin.java - Client interface - - clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java - Request class - - core/src/main/scala/kafka/server/ControllerApis.scala - Request handler - - metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java - Core logic - - Validation Steps: - - Topic name validation and conflict checking - - Partition count and replication factor validation - - Authorization (CREATE permissions) - - Configuration policy enforcement - - Quota and rate limiting - - Partition Assignment: - - Manual: Use provided replica assignments - - Automatic: ReplicaPlacer distributes across available brokers - - Validates ISR availability - - The API supports both synchronous validation (validateOnly=true) and asynchronous creation with configurable - timeouts. - -> What are some error cases of the API, that I can implement early before touching the core logic ? - -⏺ Task(Find CreateTopic API error cases) - ⎿ Done (38 tool uses · 52.6k tokens · 4m 40.6s) - -⏺ Here are the early validation error cases you can implement before touching core logic: - - Input Validation Errors (Immediate Validation) - - 1. Topic Name Validation - INVALID_TOPIC_EXCEPTION (17) - - Empty topic name or "." or ".." - - Length > 249 characters - - Invalid characters (only ASCII alphanumerics, '.', '_', '-') - - Internal topic collision detection - - 2. Partition Count Validation - INVALID_PARTITIONS (37) - - Negative or zero partition count - - Message: "Number of partitions was set to an invalid non-positive value." - - 3. Replication Factor Validation - INVALID_REPLICATION_FACTOR (38) - - Negative or zero replication factor - - Message: "Replication factor must be larger than 0, or -1 to use the default value." - - 4. Request Format Validation - INVALID_REQUEST (42) - - Duplicate topic names in single request - - Version compatibility issues (validateOnly in v0, defaults in v<4) - - 5. Manual Assignment Validation - INVALID_REPLICA_ASSIGNMENT (39) - - Assignment with non-(-1) replication factor/partition count - - Non-consecutive partition indices - - Empty replica lists - - Duplicate broker IDs in assignment - - 6. Configuration Validation - INVALID_CONFIG (40) - - Null config values - - Message: "Null value not supported for topic configs" - - 7. Authorization Check - TOPIC_AUTHORIZATION_FAILED (29) - - No CREATE permission on CLUSTER or TOPIC resource - - 8. Existing Topic Check - TOPIC_ALREADY_EXISTS (36) - - Topic name already exists - - 9. Internal Topic Protection - INVALID_REQUEST (42) - - Attempting to create __cluster_metadata topic - - These validations happen in ControllerApis.scala:396-442 and ReplicationControlManager.java:635-765 before any - core topic creation logic, making them perfect for early error detection. - -> Yep these are just validations tho. As part of the core logic. I was talking more about when we send a PRODUCE - request with an non-existent topic it just returns an error. So we have to parse the request, encode the - response but no core logic. Something similar here ? - -⏺ Ah, I understand now! You're looking for request parsing and response encoding error cases that happen at the - protocol level, before any business logic. Let me search for those. - -⏺ Task(Find protocol-level CreateTopic errors) - ⎿ Done (33 tool uses · 69.6k tokens · 2m 57.4s) - -⏺ Perfect! Now I understand what you're looking for. Here are the protocol-level error cases for CreateTopic - that you can implement with just request parsing and response encoding: - - Protocol-Level Error Cases (No Core Logic) - - 1. Request Deserialization Errors - INVALID_REQUEST (42) - - Malformed request buffer (corrupted array lengths, invalid byte arrays) - - Buffer underflow/overflow during parsing - - Invalid field types or corrupted data - - 2. Version Mismatch Errors - UNSUPPORTED_VERSION (35) - - Using validateOnly with version 0 - - Using default partitions/replication factor with version < 4 - - Requesting unsupported API version - - 3. Message Size Limit - MESSAGE_TOO_LARGE (10) - - Request exceeds broker's message.max.bytes limit - - Happens during request parsing before any business logic - - 4. Required Field Validation - INVALID_REQUEST (42) - - Missing topic name in request - - Invalid request structure per protocol schema - - 5. Array Size Validation - INVALID_REQUEST (42) - - Malformed array lengths that would cause allocation errors - - Prevents oversized collection allocation - - The key files for implementing these: - - clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java:96-109 - getErrorResponse() - method - - clients/src/main/java/org/apache/kafka/common/requests/RequestContext.java:119-129 - request parsing - - clients/src/main/resources/common/message/CreateTopicsRequest.json - protocol schema - - These errors are caught during request deserialization and can return proper CreateTopicsResponse with - per-topic error codes, just like how PRODUCE returns errors for non-existent topics - no business logic - required! - -> Nope. See this doc. /Users/ryang/Developer/work/course-testers/kafka-tester/Kafka-breakdown.md\ - I need something similar for the CreateTopic API. - -⏺ Read(../../../work/course-testers/kafka-tester/Kafka-breakdown.md) - ⎿ Read 327 lines (ctrl+r to expand) - -⏺ Perfect! Now I understand what you're looking for. You want early error cases for the CreateTopic API that can - be implemented with just request parsing and response encoding, similar to how the Produce API handles - non-existent topics. - - Here's what I would suggest for CreateTopic API stages: - - CreateTopic API Early Error Cases - - Stage 1: Include CreateTopic in APIVersions - - Add CreateTopic API (key=19) to APIVersions response - - MaxVersion should be at least 7 - - No actual CreateTopic logic needed - - Stage 2: CreateTopic with Invalid Topic Name - - Parse CreateTopic request to extract topic name - - Check for invalid topic names (empty, ".", "..", invalid chars) - - Return INVALID_TOPIC_EXCEPTION (17) error response - - No core topic creation logic needed - - Stage 3: CreateTopic with Invalid Partition/Replication Values - - Parse partition count and replication factor from request - - Return INVALID_PARTITIONS (37) for partition count ≤ 0 - - Return INVALID_REPLICATION_FACTOR (38) for replication factor ≤ 0 - - Just request parsing + response encoding - - Stage 4: CreateTopic with Existing Topic - - Check if topic directory already exists in log directory - - Return TOPIC_ALREADY_EXISTS (36) if found - - Simple filesystem check, no metadata parsing needed - - Stage 5: CreateTopic with Authorization Check - - Return TOPIC_AUTHORIZATION_FAILED (29) based on simple config - - No real authorization logic, just return the error code - - Test the response structure - - These stages would let you implement and test: - - Request parsing for CreateTopic - - Response encoding with proper error codes - - Per-topic error handling in responses - - Basic filesystem checks - - Just like the Produce API stages, these focus on protocol handling and basic validation before any complex - topic creation logic. - -> Can you improve this ? - -⏺ Update Todos - ⎿ ☐ Analyze CreateTopic API protocol structure and error cases - ☐ Design progressive stages for CreateTopic API implementation - ☐ Write detailed stage descriptions with test cases - -⏺ Task(Analyze CreateTopic protocol details) - ⎿ Done (25 tool uses · 47.2k tokens · 2m 28.4s) - -⏺ Update Todos - ⎿ ☒ Analyze CreateTopic API protocol structure and error cases - ☐ Design progressive stages for CreateTopic API implementation - ☐ Write detailed stage descriptions with test cases - -⏺ Update Todos - ⎿ ☒ Analyze CreateTopic API protocol structure and error cases - ☒ Design progressive stages for CreateTopic API implementation - ☐ Write detailed stage descriptions with test cases - -⏺ Kafka / CreateTopics API - - Stage 1: Include CreateTopics in APIVersions - - In this stage, you'll add an entry for the CreateTopics API to the APIVersions response. - - APIVersions - - Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before - implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics - API. - - Tests - - The tester will execute your program like this: - - ./your_program.sh - - It'll then connect to your server on port 9092 and send a valid APIVersions (v4) request. - - The tester will validate that: - - - The first 4 bytes of your response (the "message length") are valid. - - The correlation ID in the response header matches the correlation ID in the request header. - - The error code in the response body is 0 (No Error). - - The response body contains at least one entry for the API key 19 (CreateTopics). - - The MaxVersion for the CreateTopics API is at least 7. - - Notes - - - You don't have to implement support for the CreateTopics request in this stage. We'll get to this in later - stages. - - You'll still need to include the entry for APIVersions in your response to pass the previous stage. - - Stage 2: CreateTopics with Invalid Topic Names - - In this stage, you'll add support for handling CreateTopics requests with invalid topic names and return - proper error responses. - - Handling Invalid Topic Names - - Your Kafka implementation should validate topic names and return appropriate error codes when a client tries - to create topics with invalid names. Topic names must follow these rules: - - Cannot be empty string - - Cannot be "." or ".." - - Maximum length of 249 characters - - Only ASCII alphanumerics, '.', '_', and '-' are allowed - - Cannot have period/underscore collision patterns - - Tests - - The tester will execute your program like this: - - ./your_program.sh - - It'll then connect to your server on port 9092 and send a CreateTopics (v7) request with an invalid topic - name. - - The tester will validate that: - - - The first 4 bytes of your response (the "message length") are valid. - - The correlation ID in the response header matches the correlation ID in the request header. - - The error code in the topic result is 17 (INVALID_TOPIC_EXCEPTION). - - The throttle_time_ms field in the response is 0. - - The name field in the topic result corresponds to the topic name in the request. - - The error_message field contains a descriptive error message. - - Notes - - - You'll need to parse the CreateTopics request in this stage to get the topic names. - - The official docs for the CreateTopics request can be found - https://kafka.apache.org/protocol.html#The_Messages_CreateTopics. - - Topic name validation logic is in the Kafka source code at - clients/src/main/java/org/apache/kafka/common/internals/Topic.java. - - Stage 3: CreateTopics with Invalid Partition/Replication Parameters - - In this stage, you'll add support for handling CreateTopics requests with invalid partition counts and - replication factors. - - Handling Invalid Parameters - - Your Kafka implementation should validate numeric parameters and return appropriate errors: - - Partition count must be greater than 0 (or -1 for defaults) - - Replication factor must be greater than 0 (or -1 for defaults) - - Both parameters cannot be 0 or negative (except -1) - - Tests - - The tester will execute your program like this: - - ./your_program.sh - - It'll then connect to your server on port 9092 and send CreateTopics (v7) requests with invalid parameters. - - The tester will validate that: - - - The first 4 bytes of your response (the "message length") are valid. - - The correlation ID in the response header matches the correlation ID in the request header. - - The error code in the topic result is 37 (INVALID_PARTITIONS) for invalid partition count. - - The error code in the topic result is 38 (INVALID_REPLICATION_FACTOR) for invalid replication factor. - - The throttle_time_ms field in the response is 0. - - The name field in the topic result corresponds to the topic name in the request. - - The error_message field contains a descriptive error message. - - Notes - - - Test cases will include: partition count = 0, partition count = -2, replication factor = 0, replication - factor = -2. - - Valid values are positive integers or -1 (for broker defaults). - - Stage 4: CreateTopics with Existing Topics - - In this stage, you'll add support for detecting when a topic already exists and return the appropriate error - response. - - Handling Existing Topics - - Your Kafka implementation should check if a topic already exists before attempting to create it. You can check - for existing topics by: - - Looking for topic directories in the log directory (e.g., /tmp/kraft-combined-logs/topic-name-0/) - - Reading the __cluster_metadata topic for TopicRecord entries - - Tests - - The tester will execute your program like this: - - ./your_program.sh - # Create a topic first - # Then try to create the same topic again - - It'll then connect to your server on port 9092 and send a CreateTopics (v7) request for an existing topic. - - The tester will validate that: - - - The first 4 bytes of your response (the "message length") are valid. - - The correlation ID in the response header matches the correlation ID in the request header. - - The error code in the topic result is 36 (TOPIC_ALREADY_EXISTS). - - The throttle_time_ms field in the response is 0. - - The name field in the topic result corresponds to the topic name in the request. - - The error_message field contains a descriptive error message. - - Notes - - - This stage requires basic topic existence checking without full topic creation logic. - - You can use filesystem checks or metadata parsing to detect existing topics. - - Stage 5: CreateTopics with Invalid Manual Assignment - - In this stage, you'll add support for validating manual partition assignments and return appropriate error - responses. - - Handling Invalid Manual Assignment - - Your Kafka implementation should validate manual partition assignments when provided: - - Manual assignment requires partition count and replication factor to be -1 - - Partition indices must be consecutive starting from 0 - - Replica lists cannot be empty - - Broker IDs must be valid (though you can mock this validation) - - No duplicate broker IDs within a partition assignment - - Tests - - The tester will execute your program like this: - - ./your_program.sh - - It'll then connect to your server on port 9092 and send CreateTopics (v7) requests with invalid manual - assignments. - - The tester will validate that: - - - The first 4 bytes of your response (the "message length") are valid. - - The correlation ID in the response header matches the correlation ID in the request header. - - The error code in the topic result is 39 (INVALID_REPLICA_ASSIGNMENT). - - The throttle_time_ms field in the response is 0. - - The name field in the topic result corresponds to the topic name in the request. - - The error_message field contains a descriptive error message. - - Notes - - - Test cases include: non-(-1) partition count with assignment, non-consecutive partitions, empty replica - lists, duplicate broker IDs. - - Manual assignment validation is complex but can be implemented without broker state. - - Stage 6: CreateTopics with Authorization Failure - - In this stage, you'll add support for authorization checks and return appropriate error responses. - - Handling Authorization - - Your Kafka implementation should check authorization before creating topics: - - Require CREATE permission on CLUSTER resource or specific TOPIC resource - - You can implement a simple authorization check based on configuration or mock it - - Return authorization failure for unauthorized requests - - Tests - - The tester will execute your program like this: - - ./your_program.sh - - It'll then connect to your server on port 9092 and send CreateTopics (v7) requests without proper - authorization. - - The tester will validate that: - - - The first 4 bytes of your response (the "message length") are valid. - - The correlation ID in the response header matches the correlation ID in the request header. - - The error code in the topic result is 29 (TOPIC_AUTHORIZATION_FAILED). - - The throttle_time_ms field in the response is 0. - - The name field in the topic result corresponds to the topic name in the request. - - The error_message field contains "Authorization failed." - - Notes - - - This stage focuses on authorization error handling rather than implementing full ACL logic. - - You can use a simple configuration-based approach to determine authorization. - - --- - Setup Commands - - # Clean environment - rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs - - # Generate cluster ID and format storage - KAFKA_CLUSTER_ID="$(/usr/local/kafka-latest/bin/kafka-storage.sh random-uuid)" - /usr/local/kafka-latest/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c - /usr/local/kafka-latest/config/kraft/server.properties - - # Start Kafka server - /usr/local/kafka-latest/bin/kafka-server-start.sh /usr/local/kafka-latest/config/kraft/server.properties - - # Test with Kafka CLI - /usr/local/kafka-latest/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 - --partitions 3 - - Testing Commands - - # Build tester - make build - - # Run individual test - go run main.go - - # Verify with Kafka CLI - /usr/local/kafka-latest/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 - - Error Code Reference - - | Code | Name | Stages | - |------|----------------------------|---------------| - | 0 | NO_ERROR | Success cases | - | 17 | INVALID_TOPIC_EXCEPTION | 2 | - | 29 | TOPIC_AUTHORIZATION_FAILED | 6 | - | 36 | TOPIC_ALREADY_EXISTS | 4 | - | 37 | INVALID_PARTITIONS | 3 | - | 38 | INVALID_REPLICATION_FACTOR | 3 | - | 39 | INVALID_REPLICA_ASSIGNMENT | 5 | - -⏺ Update Todos - ⎿ ☒ Analyze CreateTopic API protocol structure and error cases - ☒ Design progressive stages for CreateTopic API implementation - ☒ Write detailed stage descriptions with test cases - -⏺ Here's an improved version of the CreateTopics API breakdown that follows the same progressive structure as - your Produce API document, with early error cases that can be implemented with just request parsing and - response encoding before touching any core logic. - -> Please write to a file ? - -⏺ Write(CreateTopics-breakdown.md) - ⎿ Wrote 258 lines to CreateTopics-breakdown.md - # Kafka / CreateTopics API - ## Stage 1: Include CreateTopics in APIVersions - In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response. - ### APIVersions - Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before - implementing topic creation functionality. This lets the client know that the broker supports the - CreateTopics API. - … +248 lines (ctrl+r to expand) - -⏺ Created the CreateTopics API breakdown document at - /Users/ryang/Developer/tmp/kafka/kafka/CreateTopics-breakdown.md. The document follows the same progressive - structure as your Produce API breakdown, with 6 stages that focus on early error cases that can be implemented - with just request parsing and response encoding before touching any core topic creation logic. - -> What is the manual assignment thing ? - -⏺ Task(Explain manual assignment in CreateTopics) - ⎿ Done (25 tool uses · 57.3k tokens · 2m 56.6s) - -⏺ Manual assignment in CreateTopics lets you explicitly specify which brokers should host each partition's - replicas, instead of letting Kafka automatically place them. - - Manual vs Automatic Assignment - - Automatic (Default): - - You specify numPartitions: 3 and replicationFactor: 2 - - Kafka automatically distributes partitions across brokers - - Manual: - - You specify exactly which brokers host each partition - - Set numPartitions: -1 and replicationFactor: -1 - - Provide an assignments array - - Manual Assignment Structure - - { - "assignments": [ - {"partitionIndex": 0, "brokerIds": [1, 2]}, // Partition 0 on brokers 1,2 - {"partitionIndex": 1, "brokerIds": [2, 3]}, // Partition 1 on brokers 2,3 - {"partitionIndex": 2, "brokerIds": [3, 1]} // Partition 2 on brokers 3,1 - ] - } - - When It's Used - - - Rack awareness: Ensure replicas are on different racks - - Performance optimization: Place partitions on specific hardware - - Compliance: Meet data locality requirements - - Migration: Control placement when moving topics - - Validation Rules - - - Partition indices must be consecutive (0, 1, 2...) - - All partitions must have same replication factor - - No duplicate brokers within a partition - - Brokers must exist in the cluster - - Can't mix manual assignment with numPartitions/replicationFactor - - The CLI equivalent is: - kafka-topics.sh --create --topic my-topic --replica-assignment "1:2,2:3,3:1" - - This gives you precise control over partition placement instead of relying on Kafka's automatic distribution - algorithm. - -╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│ > Try "fix lint errors" │ -╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ - ? for shortcuts - - From 3d09cf1f29b5f7eb17e87786e7017d10a283ad09 Mon Sep 17 00:00:00 2001 From: Ryan Date: Wed, 16 Jul 2025 22:28:21 +0530 Subject: [PATCH 29/52] feat: add initial stage description for CreateTopics API implementation Introduce the first stage description for the CreateTopics API, detailing the inclusion of the API in the APIVersions response. This stage outlines the necessary tests and notes for implementation, setting the groundwork for subsequent stages. --- stage_descriptions/create-topics-01_yb1.md | 28 ++++++++++++++++++++++ stage_descriptions/create-topics-02_ve7.md | 0 stage_descriptions/create-topics-03_cb8.md | 0 stage_descriptions/create-topics-04_us2.md | 0 stage_descriptions/create-topics-05_rk2.md | 0 stage_descriptions/create-topics-06_fl3.md | 0 stage_descriptions/create-topics-07_hh9.md | 0 7 files changed, 28 insertions(+) create mode 100644 stage_descriptions/create-topics-01_yb1.md create mode 100644 stage_descriptions/create-topics-02_ve7.md create mode 100644 stage_descriptions/create-topics-03_cb8.md create mode 100644 stage_descriptions/create-topics-04_us2.md create mode 100644 stage_descriptions/create-topics-05_rk2.md create mode 100644 stage_descriptions/create-topics-06_fl3.md create mode 100644 stage_descriptions/create-topics-07_hh9.md diff --git a/stage_descriptions/create-topics-01_yb1.md b/stage_descriptions/create-topics-01_yb1.md new file mode 100644 index 0000000..e9d7563 --- /dev/null +++ b/stage_descriptions/create-topics-01_yb1.md @@ -0,0 +1,28 @@ +In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response. + +### APIVersions + +Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics API. + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (No Error). +- The response body contains at least one entry for the API key `19` (CreateTopics). +- The `MaxVersion` for the CreateTopics API is at least 7. + +### Notes + +- You don't have to implement support for the `CreateTopics` request in this stage. We'll get to this in later stages. +- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. diff --git a/stage_descriptions/create-topics-02_ve7.md b/stage_descriptions/create-topics-02_ve7.md new file mode 100644 index 0000000..e69de29 diff --git a/stage_descriptions/create-topics-03_cb8.md b/stage_descriptions/create-topics-03_cb8.md new file mode 100644 index 0000000..e69de29 diff --git a/stage_descriptions/create-topics-04_us2.md b/stage_descriptions/create-topics-04_us2.md new file mode 100644 index 0000000..e69de29 diff --git a/stage_descriptions/create-topics-05_rk2.md b/stage_descriptions/create-topics-05_rk2.md new file mode 100644 index 0000000..e69de29 diff --git a/stage_descriptions/create-topics-06_fl3.md b/stage_descriptions/create-topics-06_fl3.md new file mode 100644 index 0000000..e69de29 diff --git a/stage_descriptions/create-topics-07_hh9.md b/stage_descriptions/create-topics-07_hh9.md new file mode 100644 index 0000000..e69de29 From a3bb12c1b7fb39d9a49aeea2535a365ca2518ca5 Mon Sep 17 00:00:00 2001 From: Ryan Date: Wed, 16 Jul 2025 22:28:31 +0530 Subject: [PATCH 30/52] feat: add comprehensive stage descriptions for CreateTopics API implementation Introduce detailed stage descriptions for the CreateTopics API, covering various scenarios such as invalid topic names, existing topics, and authorization checks. This addition enhances the documentation for implementing the API and provides clear guidelines for error handling and validation processes. --- stage_descriptions/createtopics.md | 258 +++++++++++++++++++++++++++++ stage_descriptions/final-stages.md | 25 +++ stage_descriptions/stages.yaml | 28 ++++ 3 files changed, 311 insertions(+) create mode 100644 stage_descriptions/createtopics.md create mode 100644 stage_descriptions/final-stages.md create mode 100644 stage_descriptions/stages.yaml diff --git a/stage_descriptions/createtopics.md b/stage_descriptions/createtopics.md new file mode 100644 index 0000000..9043e29 --- /dev/null +++ b/stage_descriptions/createtopics.md @@ -0,0 +1,258 @@ +# Kafka / CreateTopics API + +## Stage 1: Include CreateTopics in APIVersions + +In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response. + +### APIVersions + +Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics API. + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the response body is `0` (No Error). +- The response body contains at least one entry for the API key `19` (CreateTopics). +- The `MaxVersion` for the CreateTopics API is at least 7. + +### Notes + +- You don't have to implement support for the `CreateTopics` request in this stage. We'll get to this in later stages. +- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. + +## Stage 2: CreateTopics with Invalid Topic Names + +In this stage, you'll add support for handling CreateTopics requests with invalid topic names and return proper error responses. + +### Handling Invalid Topic Names + +Your Kafka implementation should validate topic names and return appropriate error codes when a client tries to create topics with invalid names. Topic names must follow these rules: +- Cannot be empty string +- Cannot be "." or ".." +- Maximum length of 249 characters +- Only ASCII alphanumerics, '.', '_', and '-' are allowed +- Cannot have period/underscore collision patterns + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with an invalid topic name. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic result is `17` (INVALID_TOPIC_EXCEPTION). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic result corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. + +### Notes + +- You'll need to parse the `CreateTopics` request in this stage to get the topic names. +- The official docs for the `CreateTopics` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_CreateTopics). +- Topic name validation logic is in the Kafka source code at `clients/src/main/java/org/apache/kafka/common/internals/Topic.java`. + +## Stage 3: CreateTopics with Invalid Partition/Replication Parameters + +In this stage, you'll add support for handling CreateTopics requests with invalid partition counts and replication factors. + +### Handling Invalid Parameters + +Your Kafka implementation should validate numeric parameters and return appropriate errors: +- Partition count must be greater than 0 (or -1 for defaults) +- Replication factor must be greater than 0 (or -1 for defaults) +- Both parameters cannot be 0 or negative (except -1) + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests with invalid parameters. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic result is `37` (INVALID_PARTITIONS) for invalid partition count. +- The error code in the topic result is `38` (INVALID_REPLICATION_FACTOR) for invalid replication factor. +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic result corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. + +### Notes + +- Test cases will include: partition count = 0, partition count = -2, replication factor = 0, replication factor = -2. +- Valid values are positive integers or -1 (for broker defaults). + +## Stage 4: CreateTopics with Existing Topics + +In this stage, you'll add support for detecting when a topic already exists and return the appropriate error response. + +### Handling Existing Topics + +Your Kafka implementation should check if a topic already exists before attempting to create it. You can check for existing topics by: +- Looking for topic directories in the log directory (e.g., `/tmp/kraft-combined-logs/topic-name-0/`) +- Reading the `__cluster_metadata` topic for TopicRecord entries + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +# Create a topic first +# Then try to create the same topic again +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request for an existing topic. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic result is `36` (TOPIC_ALREADY_EXISTS). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic result corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. + +### Notes + +- This stage requires basic topic existence checking without full topic creation logic. +- You can use filesystem checks or metadata parsing to detect existing topics. + +## Stage 5: CreateTopics with Invalid Manual Assignment + +In this stage, you'll add support for validating manual partition assignments and return appropriate error responses. + +### Handling Invalid Manual Assignment + +Your Kafka implementation should validate manual partition assignments when provided: +- Manual assignment requires partition count and replication factor to be -1 +- Partition indices must be consecutive starting from 0 +- Replica lists cannot be empty +- Broker IDs must be valid (though you can mock this validation) +- No duplicate broker IDs within a partition assignment + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests with invalid manual assignments. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic result is `39` (INVALID_REPLICA_ASSIGNMENT). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic result corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. + +### Notes + +- Test cases include: non-(-1) partition count with assignment, non-consecutive partitions, empty replica lists, duplicate broker IDs. +- Manual assignment validation is complex but can be implemented without broker state. + +## Stage 6: CreateTopics with Authorization Failure + +In this stage, you'll add support for authorization checks and return appropriate error responses. + +### Handling Authorization + +Your Kafka implementation should check authorization before creating topics: +- Require CREATE permission on CLUSTER resource or specific TOPIC resource +- You can implement a simple authorization check based on configuration or mock it +- Return authorization failure for unauthorized requests + +### Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests without proper authorization. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic result is `29` (TOPIC_AUTHORIZATION_FAILED). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic result corresponds to the topic name in the request. +- The `error_message` field contains "Authorization failed." + +### Notes + +- This stage focuses on authorization error handling rather than implementing full ACL logic. +- You can use a simple configuration-based approach to determine authorization. + +--- + +## Setup Commands + +```bash +# Clean environment +rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs + +# Generate cluster ID and format storage +KAFKA_CLUSTER_ID="$(/usr/local/kafka-latest/bin/kafka-storage.sh random-uuid)" +/usr/local/kafka-latest/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /usr/local/kafka-latest/config/kraft/server.properties + +# Start Kafka server +/usr/local/kafka-latest/bin/kafka-server-start.sh /usr/local/kafka-latest/config/kraft/server.properties + +# Test with Kafka CLI +/usr/local/kafka-latest/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 3 +``` + +## Testing Commands + +```bash +# Build tester +make build + +# Run individual test +go run main.go + +# Verify with Kafka CLI +/usr/local/kafka-latest/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 +``` + +## Error Code Reference + +| Code | Name | Stages | +|------|------|---------| +| 0 | NO_ERROR | Success cases | +| 17 | INVALID_TOPIC_EXCEPTION | 2 | +| 29 | TOPIC_AUTHORIZATION_FAILED | 6 | +| 36 | TOPIC_ALREADY_EXISTS | 4 | +| 37 | INVALID_PARTITIONS | 3 | +| 38 | INVALID_REPLICATION_FACTOR | 3 | +| 39 | INVALID_REPLICA_ASSIGNMENT | 5 | \ No newline at end of file diff --git a/stage_descriptions/final-stages.md b/stage_descriptions/final-stages.md new file mode 100644 index 0000000..3379eff --- /dev/null +++ b/stage_descriptions/final-stages.md @@ -0,0 +1,25 @@ +Stage CT1: Include CreateTopics in APIVersions +- Add API key 19 (CreateTopics) to APIVersions response +- Foundation stage enabling API discovery + +Stage CT2: CreateTopics with Invalid Topic Name (Hard code error) +- Handle invalid characters, reserved names, empty names +- Error code: INVALID_TOPIC_EXCEPTION (17) + +Stage CT3: CreateTopics with Reserved Topic Name (Check __cluster_metadata) +- Handle attempts to create system topics (__cluster_metadata) +- Error code: INVALID_REQUEST (42) + +Stage CT4: CreateTopics with Existing Topic Name (Read __cluster_metadata) +- Handle duplicate topic creation attempts +- Error code: TOPIC_ALREADY_EXISTS (36) + +Stage CT5: CreateTopics with Valid Parameters (Success Case) +- Successfully create single topic with valid parameters +- Core success functionality + +Stage CT6: Multiple topics in single request +- Handle multiple topics in a single request + +Stage CT7: CreateTopics with Validation Only +- Handle validation only mode diff --git a/stage_descriptions/stages.yaml b/stage_descriptions/stages.yaml new file mode 100644 index 0000000..67b1e76 --- /dev/null +++ b/stage_descriptions/stages.yaml @@ -0,0 +1,28 @@ +- Include CreateTopics in APIVersions +- CreateTopics with Invalid Topic Name (empty, ., .., length > 249, invalid characters) +- CreateTopics with Invalid Partition Values (negative, zero) +- CreateTopics with Invalid Replication Values (negative, zero, > broker count) +- CreateTopics with Reserved Topic Name (__cluster_metadata) +- CreateTopics with Existing Topic (read __cluster_metadata) +- CreateTopics with Authorization Check (no CREATE permission on CLUSTER or TOPIC resource) +- CreateTopics with Single Topic (single topic in single request) +- CreateTopics with Multiple Topics (multiple topics in single request) +- CreateTopics with Manual Assignments (manual assignment of partitions and replicas) +- CreateTopics with Validation Only +- CreateTopics with Invalid Request (Duplicate topic names in single request) + + + 1. Topic Name Validation - INVALID_TOPIC_EXCEPTION (17) + - Empty topic name or "." or ".." + - Length > 249 characters + - Invalid characters (only ASCII alphanumerics, '.', '_', '-') + - Internal topic collision detection + + + +Stage CT8: CreateTopics with Invalid Parameters +- Handle negative/zero partitions, invalid replication factors +- Duplicate topic names in single request +- Error codes: INVALID_PARTITIONS (37), INVALID_REPLICATION_FACTOR (38) + +- CT8: Topic-specific configurations From e5c54361ff59702b5f409cde9d93f61609866bf1 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:11:54 +0530 Subject: [PATCH 31/52] feat: update CreateTopics API stage description to clarify API inclusion and testing Enhance the stage description for the CreateTopics API by providing a clearer structure, including a dedicated section for the API and its tests. Update the MaxVersion requirement for the CreateTopics API and add links to interactive protocol inspectors for better guidance on request and response structures. --- stage_descriptions/create-topics-01_yb1.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/stage_descriptions/create-topics-01_yb1.md b/stage_descriptions/create-topics-01_yb1.md index e9d7563..f1a58d0 100644 --- a/stage_descriptions/create-topics-01_yb1.md +++ b/stage_descriptions/create-topics-01_yb1.md @@ -1,10 +1,17 @@ In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response. -### APIVersions +## The CreateTopics API -Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics API. +The [CreateTopics API](https://kafka.apache.org/protocol#The_Messages_CreateTopics) (API key `19`) is used to create topics in a Kafka cluster. We'll use a single broker in our cluster for this extension. -### Tests +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +In this stage, you'll only need to add an entry for the `CreateTopics` API to the APIVersions response you implemented in earlier stages. This will let the client know that the broker supports the `CreateTopics` API. We'll get to responding to `CreateTopics` requests in later stages. + +## Tests The tester will execute your program like this: @@ -20,9 +27,10 @@ The tester will validate that: - The correlation ID in the response header matches the correlation ID in the request header. - The error code in the response body is `0` (No Error). - The response body contains at least one entry for the API key `19` (CreateTopics). -- The `MaxVersion` for the CreateTopics API is at least 7. +- The `MaxVersion` for the CreateTopics API is at least 6. -### Notes +## Notes - You don't have to implement support for the `CreateTopics` request in this stage. We'll get to this in later stages. -- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. +- You'll still need to include the entry for `APIVersions` in your response to pass previous stages. +- The `MaxVersion` for the `CreateTopics` API is 7. From f3e49d22c50a29b95d711598be145949f7693a87 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:19:37 +0530 Subject: [PATCH 32/52] feat: add stage description for handling invalid topic names in CreateTopics API Introduce a new stage description that outlines the handling of `CreateTopics` requests with invalid topic names. This includes validation rules, expected error responses, and testing procedures to ensure compliance with Kafka's naming conventions. --- stage_descriptions/create-topics-02_ve7.md | 46 ++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/stage_descriptions/create-topics-02_ve7.md b/stage_descriptions/create-topics-02_ve7.md index e69de29..46975f1 100644 --- a/stage_descriptions/create-topics-02_ve7.md +++ b/stage_descriptions/create-topics-02_ve7.md @@ -0,0 +1,46 @@ +In this stage, you'll add support for handling `CreateTopics` requests with invalid topic names. + +## CreateTopics API response for invalid topics + +When a Kafka broker receives a CreateTopics request, it first needs to validate that the topic name follows Kafka's naming rules. If a topic name is invalid, it returns an appropriate error code and response without attempting to create the topic. + +Topic names must follow these rules: +- Cannot be empty string +- Cannot be "." or ".." +- Maximum length of 249 characters +- Only ASCII alphanumerics, '.', '_', and '-' are allowed + +If the topic name is invalid, the broker returns an error code of `17` (INVALID_TOPIC_EXCEPTION). + +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +In this stage, you'll need to implement basic topic name validation without needing to check against existing topics or system topics. You can hard code the error response for invalid topic names in this stage. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with an invalid topic name. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic response is `17` (INVALID_TOPIC_EXCEPTION). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. +- The `num_partitions` and `replication_factor` fields are `-1`. + +## Notes + +- You'll need to parse the `CreateTopics` request in this stage to get the topic names. +- The official docs for the `CreateTopics` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_CreateTopics). +- Topic name validation logic is in the Kafka source code at `clients/src/main/java/org/apache/kafka/common/internals/Topic.java`. \ No newline at end of file From 33d3dc9b4d8e877dcf77d5bbaeb5617b17b6ec53 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:20:55 +0530 Subject: [PATCH 33/52] feat: add stage description for handling reserved topic names in CreateTopics API Introduce a new stage description that details the handling of `CreateTopics` requests with reserved topic names. This includes validation rules, expected error responses, and testing procedures to ensure compliance with Kafka's naming conventions, particularly focusing on the `__cluster_metadata` topic and other system-reserved names. --- stage_descriptions/create-topics-03_cb8.md | 48 ++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/stage_descriptions/create-topics-03_cb8.md b/stage_descriptions/create-topics-03_cb8.md index e69de29..e7b5afa 100644 --- a/stage_descriptions/create-topics-03_cb8.md +++ b/stage_descriptions/create-topics-03_cb8.md @@ -0,0 +1,48 @@ +In this stage, you'll add support for handling CreateTopics requests with reserved topic names. + +## CreateTopics API response for reserved topics + +When a Kafka broker receives a CreateTopics request, it needs to validate that the topic name is not reserved for system use. If a client tries to create a topic with a reserved name, the broker returns an appropriate error code and response without attempting to create the topic. + +Reserved topic names include: +- `__cluster_metadata` - KRaft metadata topic +- Topics starting with `__` (double underscore) - System topics + +The broker validates this by checking if the topic name starts with `__` or matches specific reserved patterns. + +If the topic name is reserved, the broker returns an error code of `42` (INVALID_REQUEST). + +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: +- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) + +In this stage, you'll need to check if the topic name is reserved before allowing creation. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with a reserved topic name. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic response is `42` (INVALID_REQUEST). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. +- The `num_partitions` and `replication_factor` fields are `-1`. + +## Notes + +- This stage focuses on detecting reserved topic names, particularly `__cluster_metadata`. +- The `__cluster_metadata` topic is automatically created by the KRaft system and should not be manually created. \ No newline at end of file From 8a1220a9b842302bf27be7ed54b5d3f2876f64ec Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:22:41 +0530 Subject: [PATCH 34/52] feat: add stage description for handling existing topics in CreateTopics API Introduce a new stage description that details the process for detecting existing topics in the CreateTopics API. This includes validation steps, expected error responses, and testing procedures to ensure proper handling of the `TOPIC_ALREADY_EXISTS` error code. Additionally, links to interactive protocol inspectors for request and response structures are provided to aid in understanding the implementation. --- stage_descriptions/create-topics-04_us2.md | 47 ++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/stage_descriptions/create-topics-04_us2.md b/stage_descriptions/create-topics-04_us2.md index e69de29..943c8fc 100644 --- a/stage_descriptions/create-topics-04_us2.md +++ b/stage_descriptions/create-topics-04_us2.md @@ -0,0 +1,47 @@ +In this stage, you'll add support for detecting when a topic already exists and return the appropriate error response. + +## CreateTopics API response for existing topics + +When a Kafka broker receives a CreateTopics request, it needs to validate that the topic doesn't already exist before attempting to create it. If a topic already exists, it returns an appropriate error code and response without attempting to create the topic again. + +To validate that a topic exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `TOPIC_RECORD`. If there exists a `TOPIC_RECORD` with the given topic name and the topic ID, the topic exists. + +If the topic already exists, the broker returns an error code of `36` (TOPIC_ALREADY_EXISTS). + +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: +- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) + +This would help you understand the structure of the `TOPIC_RECORD` record inside the `__cluster_metadata` topic's log file. + +In this stage, you'll need to implement topic existence checking by reading the cluster metadata. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +# Create a topic first +# Then try to create the same topic again +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request for an existing topic. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic response is `36` (TOPIC_ALREADY_EXISTS). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response corresponds to the topic name in the request. +- The `error_message` field contains a descriptive error message. +- The `num_partitions` and `replication_factor` fields are `-1`. + +## Notes + +- The official Kafka docs don't cover the structure of records inside the `__cluster_metadata` topic, but you can find the definitions in the Kafka source code [here](https://github.com/apache/kafka/tree/trunk/metadata/src/main/resources/common/metadata). \ No newline at end of file From 9de60d65a3d5ae0cdbd8c4d4965cb22c143e17f9 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:25:07 +0530 Subject: [PATCH 35/52] feat: add stage description for creating topics in CreateTopics API Introduce a new stage description that outlines the process for successfully creating topics with valid parameters in the CreateTopics API. This includes detailed steps for topic creation, validation of parameters, and the expected response structure. Additionally, links to interactive protocol inspectors for request and response structures are provided, along with testing procedures to ensure proper implementation. --- stage_descriptions/create-topics-05_rk2.md | 51 ++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/stage_descriptions/create-topics-05_rk2.md b/stage_descriptions/create-topics-05_rk2.md index e69de29..b435b52 100644 --- a/stage_descriptions/create-topics-05_rk2.md +++ b/stage_descriptions/create-topics-05_rk2.md @@ -0,0 +1,51 @@ +In this stage, you'll add support for successfully creating topics with valid parameters. + +## Single Topic Creation + +When a Kafka broker receives a CreateTopics request for a valid topic name that doesn't already exist, it needs to validate the parameters, create the topic metadata, persist it to the `__cluster_metadata` topic, create the topic directory structure, and return a successful response. + +The broker performs the following steps: +1. Validates the topic name (from previous stages) +2. Checks that the topic doesn't already exist (from previous stages) +3. Validates partition count and replication factor parameters (request will include valid values) +4. Creates a new topic UUID +5. Writes a `TOPIC_RECORD` to the `__cluster_metadata` topic's log file +6. Writes `PARTITION_RECORD`s to the `__cluster_metadata` topic's log file +7. Creates partition directories in the log directory (e.g., `/tmp/kraft-combined-logs/topic-name-0/`) +8. Nested structure inside the `__cluster_metadata` topic's log file: +TODO +9. Returns a successful response with error code `0` (NO_ERROR) + +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: +- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with valid parameters. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic response is `0` (NO_ERROR). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic response corresponds to the topic name in the request. +- The topic metadata is written to the `__cluster_metadata` topic. +- The topic directory structure is created in the log directory. +- The `num_partitions` and `replication_factor` fields in the topic response correspond to the request. + +## Notes + +- The topic metadata should be persisted in the `__cluster_metadata` topic as a `TOPIC_RECORD`. +- Topic directories should be created in the log directory (e.g., `/tmp/kraft-combined-logs/topic-name-0/`). \ No newline at end of file From 58514cbc521fa65369e47d9b76da8b14b55549fa Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:26:07 +0530 Subject: [PATCH 36/52] feat: add stage description for batch topic creation in CreateTopics API Introduce a new stage description that outlines the process for handling multiple topics in a single CreateTopics request. This includes detailed steps for validation, creation, and response handling for each topic, as well as testing procedures to ensure correct implementation. Links to interactive protocol inspectors for request and response structures are also provided. --- stage_descriptions/create-topics-06_fl3.md | 47 ++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/stage_descriptions/create-topics-06_fl3.md b/stage_descriptions/create-topics-06_fl3.md index e69de29..a2677de 100644 --- a/stage_descriptions/create-topics-06_fl3.md +++ b/stage_descriptions/create-topics-06_fl3.md @@ -0,0 +1,47 @@ +In this stage, you'll add support for handling multiple topics in a single CreateTopics request. + +## Batch Topic Creation + +When a Kafka broker receives a CreateTopics request containing multiple topics, it needs to process each topic independently, applying all validation rules and creation logic to each one. The broker processes each topic in the array and returns individual results for each topic, regardless of whether other topics succeed or fail. + +The broker performs the following steps for each topic: +1. Validates the topic name (from previous stages) +2. Checks that the topic doesn't already exist (from previous stages) +3. Validates partition count and replication factor parameters +4. Creates a new topic UUID (for successful topics) +5. Writes a `TOPIC_RECORD` to the `__cluster_metadata` topic's log file (for successful topics) +6. Creates partition directories in the log directory (for successful topics) +7. Returns individual results for each topic in the response array + +Mixed success/failure scenarios are handled gracefully - if one topic fails validation, it doesn't prevent other valid topics from being created. + +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with multiple topics. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The response contains individual results for each topic in the request. +- Each topic result has the correct error code (0 for success, appropriate error codes for failures). +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in each topic response corresponds to the topic name in the request. +- Successfully created topics have their metadata written to the `__cluster_metadata` topic. +- Failed topics have appropriate error messages. + +## Notes + +- The CreateTopics request can contain multiple topics, and each should be processed independently. +- The response should contain a result for each topic in the request, in the same order. \ No newline at end of file From bc5b677d64c118e1971b80ed02bc5fb3ede2552c Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:26:45 +0530 Subject: [PATCH 37/52] feat: add stage description for validation-only mode in CreateTopics API Introduce a new stage description that outlines the process for handling CreateTopics requests with the `validate_only` flag. This includes detailed validation steps, expected responses, and testing procedures to ensure correct implementation without side effects. Links to interactive protocol inspectors for request and response structures are also provided. --- stage_descriptions/create-topics-07_hh9.md | 50 ++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/stage_descriptions/create-topics-07_hh9.md b/stage_descriptions/create-topics-07_hh9.md index e69de29..0330dcf 100644 --- a/stage_descriptions/create-topics-07_hh9.md +++ b/stage_descriptions/create-topics-07_hh9.md @@ -0,0 +1,50 @@ +In this stage, you'll add support for handling CreateTopics requests with validation-only mode. + +## Validation-Only Mode + +When a Kafka broker receives a CreateTopics request with the `validate_only` flag set to `true`, it needs to perform all validation checks as if creating the topics but without actually creating them or modifying any persistent state. This allows clients to validate topic creation parameters without side effects. + +The broker performs the following steps: +1. Validates the topic name (from previous stages) +2. Checks that the topic doesn't already exist (from previous stages) +3. Validates partition count and replication factor parameters +4. Returns success/error responses as if creating the topics +5. Does NOT write any `TOPIC_RECORD` to the `__cluster_metadata` topic +6. Does NOT create any partition directories +7. Does NOT modify any persistent state + +The response should be identical to what would be returned for actual topic creation, but without any persistence side effects. + +We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: + +- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) +- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) + +In this stage, you'll need to check the `validate_only` flag in the request and modify your behavior accordingly. + +## Tests + +The tester will execute your program like this: + +```bash +./your_program.sh +``` + +It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with `validate_only` set to `true`. + +The tester will validate that: + +- The first 4 bytes of your response (the "message length") are valid. +- The correlation ID in the response header matches the correlation ID in the request header. +- The error code in the topic result is `0` (NO_ERROR) for valid topics. +- The `throttle_time_ms` field in the response is `0`. +- The `name` field in the topic result corresponds to the topic name in the request. +- NO topic metadata is written to the `__cluster_metadata` topic. +- NO topic directories are created in the log directory. +- The response is identical to what would be returned for actual topic creation. + +## Notes + +- This stage tests the ability to perform dry-run validation without side effects. +- All validation logic from previous stages should be applied. +- The response should be identical to actual topic creation, but without persistence. \ No newline at end of file From 799cd6eeeba4fa51799ac6c88bb68ff648fc9f0f Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:28:18 +0530 Subject: [PATCH 38/52] feat: add stage description for creating topics with valid parameters in CreateTopics API Introduce a new stage description that outlines the process for successfully creating topics with valid parameters in the CreateTopics API. This includes detailed steps for validation, metadata persistence, directory creation, and expected responses. Additionally, links to interactive protocol inspectors for request and response structures are provided, along with testing procedures to ensure proper implementation. --- ...create-topics-07_hh9.md => create-topics-05_hh9.md} | 0 ...create-topics-05_rk2.md => create-topics-07_rk2.md} | 0 stage_descriptions/final-stages.md | 10 +++++----- 3 files changed, 5 insertions(+), 5 deletions(-) rename stage_descriptions/{create-topics-07_hh9.md => create-topics-05_hh9.md} (100%) rename stage_descriptions/{create-topics-05_rk2.md => create-topics-07_rk2.md} (100%) diff --git a/stage_descriptions/create-topics-07_hh9.md b/stage_descriptions/create-topics-05_hh9.md similarity index 100% rename from stage_descriptions/create-topics-07_hh9.md rename to stage_descriptions/create-topics-05_hh9.md diff --git a/stage_descriptions/create-topics-05_rk2.md b/stage_descriptions/create-topics-07_rk2.md similarity index 100% rename from stage_descriptions/create-topics-05_rk2.md rename to stage_descriptions/create-topics-07_rk2.md diff --git a/stage_descriptions/final-stages.md b/stage_descriptions/final-stages.md index 3379eff..09f83dc 100644 --- a/stage_descriptions/final-stages.md +++ b/stage_descriptions/final-stages.md @@ -14,12 +14,12 @@ Stage CT4: CreateTopics with Existing Topic Name (Read __cluster_metadata) - Handle duplicate topic creation attempts - Error code: TOPIC_ALREADY_EXISTS (36) -Stage CT5: CreateTopics with Valid Parameters (Success Case) +Stage CT5: CreateTopics with Validation Only +- Handle all validations but don't persist any data + +Stage CT6: CreateTopics with Valid Parameters (Success Case) - Successfully create single topic with valid parameters - Core success functionality -Stage CT6: Multiple topics in single request +Stage CT7: Multiple topics in single request - Handle multiple topics in a single request - -Stage CT7: CreateTopics with Validation Only -- Handle validation only mode From 2634d96f563c9a74d4edcddcbf45d4a696efae78 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:28:54 +0530 Subject: [PATCH 39/52] fix: update CreateTopics API stage descriptions to use version 6 Update all stage descriptions for the CreateTopics API to reference version 6 instead of version 7. This includes adjustments to the interactive protocol inspector links and the testing procedures to ensure consistency across all stages. The changes clarify the expected request and response structures for the API. --- stage_descriptions/create-topics-01_yb1.md | 4 ++-- stage_descriptions/create-topics-02_ve7.md | 6 +++--- stage_descriptions/create-topics-03_cb8.md | 6 +++--- stage_descriptions/create-topics-04_us2.md | 6 +++--- stage_descriptions/create-topics-05_hh9.md | 6 +++--- stage_descriptions/create-topics-06_fl3.md | 6 +++--- stage_descriptions/create-topics-07_rk2.md | 6 +++--- stage_descriptions/createtopics.md | 10 +++++----- 8 files changed, 25 insertions(+), 25 deletions(-) diff --git a/stage_descriptions/create-topics-01_yb1.md b/stage_descriptions/create-topics-01_yb1.md index f1a58d0..3dd545d 100644 --- a/stage_descriptions/create-topics-01_yb1.md +++ b/stage_descriptions/create-topics-01_yb1.md @@ -6,8 +6,8 @@ The [CreateTopics API](https://kafka.apache.org/protocol#The_Messages_CreateTopi We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) In this stage, you'll only need to add an entry for the `CreateTopics` API to the APIVersions response you implemented in earlier stages. This will let the client know that the broker supports the `CreateTopics` API. We'll get to responding to `CreateTopics` requests in later stages. diff --git a/stage_descriptions/create-topics-02_ve7.md b/stage_descriptions/create-topics-02_ve7.md index 46975f1..852a753 100644 --- a/stage_descriptions/create-topics-02_ve7.md +++ b/stage_descriptions/create-topics-02_ve7.md @@ -14,8 +14,8 @@ If the topic name is invalid, the broker returns an error code of `17` (INVALID_ We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) In this stage, you'll need to implement basic topic name validation without needing to check against existing topics or system topics. You can hard code the error response for invalid topic names in this stage. @@ -27,7 +27,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with an invalid topic name. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with an invalid topic name. The tester will validate that: diff --git a/stage_descriptions/create-topics-03_cb8.md b/stage_descriptions/create-topics-03_cb8.md index e7b5afa..49f06da 100644 --- a/stage_descriptions/create-topics-03_cb8.md +++ b/stage_descriptions/create-topics-03_cb8.md @@ -14,8 +14,8 @@ If the topic name is reserved, the broker returns an error code of `42` (INVALID We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) @@ -30,7 +30,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with a reserved topic name. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with a reserved topic name. The tester will validate that: diff --git a/stage_descriptions/create-topics-04_us2.md b/stage_descriptions/create-topics-04_us2.md index 943c8fc..7c1bd40 100644 --- a/stage_descriptions/create-topics-04_us2.md +++ b/stage_descriptions/create-topics-04_us2.md @@ -10,8 +10,8 @@ If the topic already exists, the broker returns an error code of `36` (TOPIC_ALR We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) @@ -30,7 +30,7 @@ The tester will execute your program like this: # Then try to create the same topic again ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request for an existing topic. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request for an existing topic. The tester will validate that: diff --git a/stage_descriptions/create-topics-05_hh9.md b/stage_descriptions/create-topics-05_hh9.md index 0330dcf..593ca60 100644 --- a/stage_descriptions/create-topics-05_hh9.md +++ b/stage_descriptions/create-topics-05_hh9.md @@ -17,8 +17,8 @@ The response should be identical to what would be returned for actual topic crea We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) In this stage, you'll need to check the `validate_only` flag in the request and modify your behavior accordingly. @@ -30,7 +30,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with `validate_only` set to `true`. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with `validate_only` set to `true`. The tester will validate that: diff --git a/stage_descriptions/create-topics-06_fl3.md b/stage_descriptions/create-topics-06_fl3.md index a2677de..2b9ce96 100644 --- a/stage_descriptions/create-topics-06_fl3.md +++ b/stage_descriptions/create-topics-06_fl3.md @@ -17,8 +17,8 @@ Mixed success/failure scenarios are handled gracefully - if one topic fails vali We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) ## Tests @@ -28,7 +28,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with multiple topics. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with multiple topics. The tester will validate that: diff --git a/stage_descriptions/create-topics-07_rk2.md b/stage_descriptions/create-topics-07_rk2.md index b435b52..550fdef 100644 --- a/stage_descriptions/create-topics-07_rk2.md +++ b/stage_descriptions/create-topics-07_rk2.md @@ -18,8 +18,8 @@ TODO We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: -- 🔎 [CreateTopics Request (v7)](https://binspec.org/kafka-createtopics-request-v7) -- 🔎 [CreateTopics Response (v7)](https://binspec.org/kafka-createtopics-response-v7) +- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) +- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) @@ -32,7 +32,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with valid parameters. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with valid parameters. The tester will validate that: diff --git a/stage_descriptions/createtopics.md b/stage_descriptions/createtopics.md index 9043e29..407aa8b 100644 --- a/stage_descriptions/createtopics.md +++ b/stage_descriptions/createtopics.md @@ -52,7 +52,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with an invalid topic name. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with an invalid topic name. The tester will validate that: @@ -88,7 +88,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests with invalid parameters. +It'll then connect to your server on port 9092 and send `CreateTopics` (v6) requests with invalid parameters. The tester will validate that: @@ -125,7 +125,7 @@ The tester will execute your program like this: # Then try to create the same topic again ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request for an existing topic. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request for an existing topic. The tester will validate that: @@ -162,7 +162,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests with invalid manual assignments. +It'll then connect to your server on port 9092 and send `CreateTopics` (v6) requests with invalid manual assignments. The tester will validate that: @@ -197,7 +197,7 @@ The tester will execute your program like this: ./your_program.sh ``` -It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests without proper authorization. +It'll then connect to your server on port 9092 and send `CreateTopics` (v6) requests without proper authorization. The tester will validate that: From a01034e38609678a06ad7c4f8a954f3b1bfdbf9f Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:31:21 +0530 Subject: [PATCH 40/52] feat: add comprehensive stage descriptions for CreateTopics API Introduce new stage descriptions for the CreateTopics API, covering various scenarios including handling invalid topic names, partition values, replication factors, and authorization checks. This update enhances the documentation by providing clear guidelines for validation, error handling, and expected responses, ensuring a thorough understanding of the CreateTopics functionality. --- .../stages.yaml => all-stages.md | 14 +- .../final-stages.md => final-stages.md | 0 stage_descriptions/createtopics.md | 258 ------------------ 3 files changed, 7 insertions(+), 265 deletions(-) rename stage_descriptions/stages.yaml => all-stages.md (76%) rename stage_descriptions/final-stages.md => final-stages.md (100%) delete mode 100644 stage_descriptions/createtopics.md diff --git a/stage_descriptions/stages.yaml b/all-stages.md similarity index 76% rename from stage_descriptions/stages.yaml rename to all-stages.md index 67b1e76..b595461 100644 --- a/stage_descriptions/stages.yaml +++ b/all-stages.md @@ -12,17 +12,17 @@ - CreateTopics with Invalid Request (Duplicate topic names in single request) - 1. Topic Name Validation - INVALID_TOPIC_EXCEPTION (17) - - Empty topic name or "." or ".." - - Length > 249 characters - - Invalid characters (only ASCII alphanumerics, '.', '_', '-') - - Internal topic collision detection +1. Topic Name Validation - INVALID_TOPIC_EXCEPTION (17) +- Empty topic name or "." or ".." +- Length > 249 characters +- Invalid characters (only ASCII alphanumerics, '.', '_', '-') +- Internal topic collision detection -Stage CT8: CreateTopics with Invalid Parameters +Stage CTX: CreateTopics with Invalid Parameters - Handle negative/zero partitions, invalid replication factors - Duplicate topic names in single request - Error codes: INVALID_PARTITIONS (37), INVALID_REPLICATION_FACTOR (38) -- CT8: Topic-specific configurations +Stage CTX: Topic-specific configurations diff --git a/stage_descriptions/final-stages.md b/final-stages.md similarity index 100% rename from stage_descriptions/final-stages.md rename to final-stages.md diff --git a/stage_descriptions/createtopics.md b/stage_descriptions/createtopics.md deleted file mode 100644 index 407aa8b..0000000 --- a/stage_descriptions/createtopics.md +++ /dev/null @@ -1,258 +0,0 @@ -# Kafka / CreateTopics API - -## Stage 1: Include CreateTopics in APIVersions - -In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response. - -### APIVersions - -Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics API. - -### Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -``` - -It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the response body is `0` (No Error). -- The response body contains at least one entry for the API key `19` (CreateTopics). -- The `MaxVersion` for the CreateTopics API is at least 7. - -### Notes - -- You don't have to implement support for the `CreateTopics` request in this stage. We'll get to this in later stages. -- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. - -## Stage 2: CreateTopics with Invalid Topic Names - -In this stage, you'll add support for handling CreateTopics requests with invalid topic names and return proper error responses. - -### Handling Invalid Topic Names - -Your Kafka implementation should validate topic names and return appropriate error codes when a client tries to create topics with invalid names. Topic names must follow these rules: -- Cannot be empty string -- Cannot be "." or ".." -- Maximum length of 249 characters -- Only ASCII alphanumerics, '.', '_', and '-' are allowed -- Cannot have period/underscore collision patterns - -### Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -``` - -It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with an invalid topic name. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic result is `17` (INVALID_TOPIC_EXCEPTION). -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic result corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. - -### Notes - -- You'll need to parse the `CreateTopics` request in this stage to get the topic names. -- The official docs for the `CreateTopics` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_CreateTopics). -- Topic name validation logic is in the Kafka source code at `clients/src/main/java/org/apache/kafka/common/internals/Topic.java`. - -## Stage 3: CreateTopics with Invalid Partition/Replication Parameters - -In this stage, you'll add support for handling CreateTopics requests with invalid partition counts and replication factors. - -### Handling Invalid Parameters - -Your Kafka implementation should validate numeric parameters and return appropriate errors: -- Partition count must be greater than 0 (or -1 for defaults) -- Replication factor must be greater than 0 (or -1 for defaults) -- Both parameters cannot be 0 or negative (except -1) - -### Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -``` - -It'll then connect to your server on port 9092 and send `CreateTopics` (v6) requests with invalid parameters. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic result is `37` (INVALID_PARTITIONS) for invalid partition count. -- The error code in the topic result is `38` (INVALID_REPLICATION_FACTOR) for invalid replication factor. -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic result corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. - -### Notes - -- Test cases will include: partition count = 0, partition count = -2, replication factor = 0, replication factor = -2. -- Valid values are positive integers or -1 (for broker defaults). - -## Stage 4: CreateTopics with Existing Topics - -In this stage, you'll add support for detecting when a topic already exists and return the appropriate error response. - -### Handling Existing Topics - -Your Kafka implementation should check if a topic already exists before attempting to create it. You can check for existing topics by: -- Looking for topic directories in the log directory (e.g., `/tmp/kraft-combined-logs/topic-name-0/`) -- Reading the `__cluster_metadata` topic for TopicRecord entries - -### Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -# Create a topic first -# Then try to create the same topic again -``` - -It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request for an existing topic. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic result is `36` (TOPIC_ALREADY_EXISTS). -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic result corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. - -### Notes - -- This stage requires basic topic existence checking without full topic creation logic. -- You can use filesystem checks or metadata parsing to detect existing topics. - -## Stage 5: CreateTopics with Invalid Manual Assignment - -In this stage, you'll add support for validating manual partition assignments and return appropriate error responses. - -### Handling Invalid Manual Assignment - -Your Kafka implementation should validate manual partition assignments when provided: -- Manual assignment requires partition count and replication factor to be -1 -- Partition indices must be consecutive starting from 0 -- Replica lists cannot be empty -- Broker IDs must be valid (though you can mock this validation) -- No duplicate broker IDs within a partition assignment - -### Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -``` - -It'll then connect to your server on port 9092 and send `CreateTopics` (v6) requests with invalid manual assignments. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic result is `39` (INVALID_REPLICA_ASSIGNMENT). -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic result corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. - -### Notes - -- Test cases include: non-(-1) partition count with assignment, non-consecutive partitions, empty replica lists, duplicate broker IDs. -- Manual assignment validation is complex but can be implemented without broker state. - -## Stage 6: CreateTopics with Authorization Failure - -In this stage, you'll add support for authorization checks and return appropriate error responses. - -### Handling Authorization - -Your Kafka implementation should check authorization before creating topics: -- Require CREATE permission on CLUSTER resource or specific TOPIC resource -- You can implement a simple authorization check based on configuration or mock it -- Return authorization failure for unauthorized requests - -### Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -``` - -It'll then connect to your server on port 9092 and send `CreateTopics` (v6) requests without proper authorization. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic result is `29` (TOPIC_AUTHORIZATION_FAILED). -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic result corresponds to the topic name in the request. -- The `error_message` field contains "Authorization failed." - -### Notes - -- This stage focuses on authorization error handling rather than implementing full ACL logic. -- You can use a simple configuration-based approach to determine authorization. - ---- - -## Setup Commands - -```bash -# Clean environment -rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs - -# Generate cluster ID and format storage -KAFKA_CLUSTER_ID="$(/usr/local/kafka-latest/bin/kafka-storage.sh random-uuid)" -/usr/local/kafka-latest/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /usr/local/kafka-latest/config/kraft/server.properties - -# Start Kafka server -/usr/local/kafka-latest/bin/kafka-server-start.sh /usr/local/kafka-latest/config/kraft/server.properties - -# Test with Kafka CLI -/usr/local/kafka-latest/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 3 -``` - -## Testing Commands - -```bash -# Build tester -make build - -# Run individual test -go run main.go - -# Verify with Kafka CLI -/usr/local/kafka-latest/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 -``` - -## Error Code Reference - -| Code | Name | Stages | -|------|------|---------| -| 0 | NO_ERROR | Success cases | -| 17 | INVALID_TOPIC_EXCEPTION | 2 | -| 29 | TOPIC_AUTHORIZATION_FAILED | 6 | -| 36 | TOPIC_ALREADY_EXISTS | 4 | -| 37 | INVALID_PARTITIONS | 3 | -| 38 | INVALID_REPLICATION_FACTOR | 3 | -| 39 | INVALID_REPLICA_ASSIGNMENT | 5 | \ No newline at end of file From b8bdd4fd4438882777f2d844ef18703817eb1b1a Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:32:50 +0530 Subject: [PATCH 41/52] refactor: clarify stage description for multiple topics in CreateTopics API Update the stage description for handling multiple topics in a single CreateTopics request to improve clarity. The title has been modified to specify "single CreateTopics request," ensuring better understanding of the functionality being documented. --- final-stages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/final-stages.md b/final-stages.md index 09f83dc..7a07c28 100644 --- a/final-stages.md +++ b/final-stages.md @@ -21,5 +21,5 @@ Stage CT6: CreateTopics with Valid Parameters (Success Case) - Successfully create single topic with valid parameters - Core success functionality -Stage CT7: Multiple topics in single request +Stage CT7: Multiple topics in single CreateTopics request - Handle multiple topics in a single request From 23af5e72a07c1d6e604d6ecb96f74d4bb339ce90 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 00:51:29 +0530 Subject: [PATCH 42/52] feat: add new stages for creating topics in CreateTopics API Introduce multiple new stages for the CreateTopics API, focusing on various scenarios such as creating topics with valid, invalid, reserved, and existing names, as well as validation-only mode and batch creation. This update enhances the documentation and provides clear guidelines for implementing the CreateTopics functionality. --- course-definition.yml | 60 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/course-definition.yml b/course-definition.yml index 73cd241..e52f29f 100644 --- a/course-definition.yml +++ b/course-definition.yml @@ -80,6 +80,15 @@ extensions: [produce-api]: https://kafka.apache.org/protocol.html#The_Messages_Produce + - slug: "creating-topics" + name: "Creating Topics" + description_markdown: | + In this challenge extension you'll add support for creating topics by implementing the [CreateTopics][create-topics-api] API. + + Along the way you'll learn about how Kafka's CreateTopics API works, topic validation, the `__cluster_metadata` topic and more. + + [create-topics-api]: https://kafka.apache.org/protocol.html#The_Messages_CreateTopics + stages: - slug: "vi6" name: "Bind to a port" @@ -252,3 +261,54 @@ stages: difficulty: hard marketing_md: |- In this stage, you'll implement producing to multiple topics in a single request. + + # Creating Topics + + - slug: "yb1" + primary_extension_slug: "creating-topics" + name: "Include CreateTopics in APIVersions" + difficulty: easy + marketing_md: |- + In this stage, you'll add the CreateTopics API to the APIVersions response. + + - slug: "ve7" + primary_extension_slug: "creating-topics" + name: "CreateTopics with Invalid Topic Name" + difficulty: medium + marketing_md: |- + In this stage, you'll implement the CreateTopics response for invalid topic names. + + - slug: "cb8" + primary_extension_slug: "creating-topics" + name: "CreateTopics with Reserved Topic Name" + difficulty: medium + marketing_md: |- + In this stage, you'll implement the CreateTopics response for reserved topic names. + + - slug: "us2" + primary_extension_slug: "creating-topics" + name: "CreateTopics with Existing Topic Name" + difficulty: medium + marketing_md: |- + In this stage, you'll implement the CreateTopics response for existing topic names. + + - slug: "hh9" + primary_extension_slug: "creating-topics" + name: "CreateTopics with Validation Only" + difficulty: medium + marketing_md: |- + In this stage, you'll implement the CreateTopics response for validation-only mode. + + - slug: "rk2" + primary_extension_slug: "creating-topics" + name: "CreateTopics with a single topic" + difficulty: hard + marketing_md: |- + In this stage, you'll implement creating a single topic. + + - slug: "fl3" + primary_extension_slug: "creating-topics" + name: "CreateTopics with Multiple Topics" + difficulty: hard + marketing_md: |- + In this stage, you'll implement creating multiple topics in a single request. From 1b12f241aa1b1b2a41cdc827f0de3f673e418447 Mon Sep 17 00:00:00 2001 From: Ryan Date: Thu, 17 Jul 2025 12:16:59 +0530 Subject: [PATCH 43/52] chore: rename stage description files to match expected format --- .../{create-topics-01_yb1.md => creating-topics-01-yb1.md} | 0 .../{create-topics-02_ve7.md => creating-topics-02-ve7.md} | 0 .../{create-topics-03_cb8.md => creating-topics-03-cb8.md} | 0 .../{create-topics-04_us2.md => creating-topics-04-us2.md} | 0 .../{create-topics-05_hh9.md => creating-topics-05-hh9.md} | 0 .../{create-topics-06_fl3.md => creating-topics-06-fl3.md} | 0 .../{create-topics-07_rk2.md => creating-topics-07-rk2.md} | 0 7 files changed, 0 insertions(+), 0 deletions(-) rename stage_descriptions/{create-topics-01_yb1.md => creating-topics-01-yb1.md} (100%) rename stage_descriptions/{create-topics-02_ve7.md => creating-topics-02-ve7.md} (100%) rename stage_descriptions/{create-topics-03_cb8.md => creating-topics-03-cb8.md} (100%) rename stage_descriptions/{create-topics-04_us2.md => creating-topics-04-us2.md} (100%) rename stage_descriptions/{create-topics-05_hh9.md => creating-topics-05-hh9.md} (100%) rename stage_descriptions/{create-topics-06_fl3.md => creating-topics-06-fl3.md} (100%) rename stage_descriptions/{create-topics-07_rk2.md => creating-topics-07-rk2.md} (100%) diff --git a/stage_descriptions/create-topics-01_yb1.md b/stage_descriptions/creating-topics-01-yb1.md similarity index 100% rename from stage_descriptions/create-topics-01_yb1.md rename to stage_descriptions/creating-topics-01-yb1.md diff --git a/stage_descriptions/create-topics-02_ve7.md b/stage_descriptions/creating-topics-02-ve7.md similarity index 100% rename from stage_descriptions/create-topics-02_ve7.md rename to stage_descriptions/creating-topics-02-ve7.md diff --git a/stage_descriptions/create-topics-03_cb8.md b/stage_descriptions/creating-topics-03-cb8.md similarity index 100% rename from stage_descriptions/create-topics-03_cb8.md rename to stage_descriptions/creating-topics-03-cb8.md diff --git a/stage_descriptions/create-topics-04_us2.md b/stage_descriptions/creating-topics-04-us2.md similarity index 100% rename from stage_descriptions/create-topics-04_us2.md rename to stage_descriptions/creating-topics-04-us2.md diff --git a/stage_descriptions/create-topics-05_hh9.md b/stage_descriptions/creating-topics-05-hh9.md similarity index 100% rename from stage_descriptions/create-topics-05_hh9.md rename to stage_descriptions/creating-topics-05-hh9.md diff --git a/stage_descriptions/create-topics-06_fl3.md b/stage_descriptions/creating-topics-06-fl3.md similarity index 100% rename from stage_descriptions/create-topics-06_fl3.md rename to stage_descriptions/creating-topics-06-fl3.md diff --git a/stage_descriptions/create-topics-07_rk2.md b/stage_descriptions/creating-topics-07-rk2.md similarity index 100% rename from stage_descriptions/create-topics-07_rk2.md rename to stage_descriptions/creating-topics-07-rk2.md From 4e465b837df91e777adbd8ac8849be4512a3fcdf Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:51:40 +0530 Subject: [PATCH 44/52] fix: update MaxVersion for CreateTopics API to 6 in stage description Adjust the MaxVersion for the CreateTopics API in the stage description to reflect the correct version 6, ensuring consistency with previous updates and clarifying the expected API behavior. --- stage_descriptions/creating-topics-01-yb1.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stage_descriptions/creating-topics-01-yb1.md b/stage_descriptions/creating-topics-01-yb1.md index 3dd545d..800524d 100644 --- a/stage_descriptions/creating-topics-01-yb1.md +++ b/stage_descriptions/creating-topics-01-yb1.md @@ -16,7 +16,7 @@ In this stage, you'll only need to add an entry for the `CreateTopics` API to th The tester will execute your program like this: ```bash -./your_program.sh +./your_program.sh /tmp/server.properties ``` It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request. @@ -33,4 +33,4 @@ The tester will validate that: - You don't have to implement support for the `CreateTopics` request in this stage. We'll get to this in later stages. - You'll still need to include the entry for `APIVersions` in your response to pass previous stages. -- The `MaxVersion` for the `CreateTopics` API is 7. +- The `MaxVersion` for the `CreateTopics` API is 6. From f1ebdc4a9f0314dfc309aca2868f8173123a666a Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:51:46 +0530 Subject: [PATCH 45/52] docs: update topic name validation instructions in CreateTopics stage description Clarify the instructions for implementing basic topic name validation by specifying that error responses for invalid topic names can be hard-coded. Additionally, note that checks against existing topics and system topics will be addressed in later stages. Update the testing command to include the server properties file for better clarity. --- stage_descriptions/creating-topics-02-ve7.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/stage_descriptions/creating-topics-02-ve7.md b/stage_descriptions/creating-topics-02-ve7.md index 852a753..ac1f405 100644 --- a/stage_descriptions/creating-topics-02-ve7.md +++ b/stage_descriptions/creating-topics-02-ve7.md @@ -17,14 +17,14 @@ We've created an interactive protocol inspector for the request & response struc - 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) - 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) -In this stage, you'll need to implement basic topic name validation without needing to check against existing topics or system topics. You can hard code the error response for invalid topic names in this stage. +In this stage, you'll need to implement basic topic name validation without needing to check against existing topics or system topics. You can hard code the error response for invalid topic names in this stage. We'll get to checking against existing topics and system topics in later stages. ## Tests The tester will execute your program like this: ```bash -./your_program.sh +./your_program.sh /tmp/server.properties ``` It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with an invalid topic name. @@ -36,7 +36,7 @@ The tester will validate that: - The error code in the topic response is `17` (INVALID_TOPIC_EXCEPTION). - The `throttle_time_ms` field in the response is `0`. - The `name` field in the topic response corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. +- The `error_message` field contains `Topic name is invalid`. - The `num_partitions` and `replication_factor` fields are `-1`. ## Notes From e9bf2212aeadc66d6db824aa799d05a0532eb5ba Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:52:28 +0530 Subject: [PATCH 46/52] feat: add stage description for handling CreateTopics requests with existing topic names Introduce a new stage that outlines the process for validating topic existence in the CreateTopics API. This includes error handling for existing and reserved topic names, along with detailed testing procedures and links to interactive protocol inspectors for better understanding of the topic metadata structure. --- stage_descriptions/creating-topics-03-cb8.md | 48 -------------------- 1 file changed, 48 deletions(-) delete mode 100644 stage_descriptions/creating-topics-03-cb8.md diff --git a/stage_descriptions/creating-topics-03-cb8.md b/stage_descriptions/creating-topics-03-cb8.md deleted file mode 100644 index 49f06da..0000000 --- a/stage_descriptions/creating-topics-03-cb8.md +++ /dev/null @@ -1,48 +0,0 @@ -In this stage, you'll add support for handling CreateTopics requests with reserved topic names. - -## CreateTopics API response for reserved topics - -When a Kafka broker receives a CreateTopics request, it needs to validate that the topic name is not reserved for system use. If a client tries to create a topic with a reserved name, the broker returns an appropriate error code and response without attempting to create the topic. - -Reserved topic names include: -- `__cluster_metadata` - KRaft metadata topic -- Topics starting with `__` (double underscore) - System topics - -The broker validates this by checking if the topic name starts with `__` or matches specific reserved patterns. - -If the topic name is reserved, the broker returns an error code of `42` (INVALID_REQUEST). - -We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: - -- 🔎 [CreateTopics Request (v6)](https://binspec.org/kafka-createtopics-request-v6) -- 🔎 [CreateTopics Response (v6)](https://binspec.org/kafka-createtopics-response-v6) - -We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: -- 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) - -In this stage, you'll need to check if the topic name is reserved before allowing creation. - -## Tests - -The tester will execute your program like this: - -```bash -./your_program.sh -``` - -It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with a reserved topic name. - -The tester will validate that: - -- The first 4 bytes of your response (the "message length") are valid. -- The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic response is `42` (INVALID_REQUEST). -- The `throttle_time_ms` field in the response is `0`. -- The `name` field in the topic response corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. -- The `num_partitions` and `replication_factor` fields are `-1`. - -## Notes - -- This stage focuses on detecting reserved topic names, particularly `__cluster_metadata`. -- The `__cluster_metadata` topic is automatically created by the KRaft system and should not be manually created. \ No newline at end of file From ed96b98fed82440f4b61201b3d5e06cff26428be Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:52:34 +0530 Subject: [PATCH 47/52] chore: rename files --- ...cs-04-us2.md => creating-topics-03-us2.md} | 21 +++++++++++-------- ...cs-05-hh9.md => creating-topics-04-hh9.md} | 0 ...cs-06-fl3.md => creating-topics-05-fl3.md} | 0 ...cs-07-rk2.md => creating-topics-06-rk2.md} | 0 4 files changed, 12 insertions(+), 9 deletions(-) rename stage_descriptions/{creating-topics-04-us2.md => creating-topics-03-us2.md} (69%) rename stage_descriptions/{creating-topics-05-hh9.md => creating-topics-04-hh9.md} (100%) rename stage_descriptions/{creating-topics-06-fl3.md => creating-topics-05-fl3.md} (100%) rename stage_descriptions/{creating-topics-07-rk2.md => creating-topics-06-rk2.md} (100%) diff --git a/stage_descriptions/creating-topics-04-us2.md b/stage_descriptions/creating-topics-03-us2.md similarity index 69% rename from stage_descriptions/creating-topics-04-us2.md rename to stage_descriptions/creating-topics-03-us2.md index 7c1bd40..5ebffde 100644 --- a/stage_descriptions/creating-topics-04-us2.md +++ b/stage_descriptions/creating-topics-03-us2.md @@ -1,12 +1,17 @@ -In this stage, you'll add support for detecting when a topic already exists and return the appropriate error response. +In this stage, you'll add support for handling `CreateTopics` requests with existing topic names. ## CreateTopics API response for existing topics -When a Kafka broker receives a CreateTopics request, it needs to validate that the topic doesn't already exist before attempting to create it. If a topic already exists, it returns an appropriate error code and response without attempting to create the topic again. +When a Kafka broker receives a CreateTopics request, it needs to validate that the topic doesn't already exist and is not reserved for system use. If a topic already exists or is reserved, it returns an appropriate error code and response without attempting to create the topic again. To validate that a topic exists, the broker reads the `__cluster_metadata` topic's log file, located at `/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log`. Inside the log file, the broker finds the topic's metadata, which is a `record` (inside a RecordBatch) with a payload of type `TOPIC_RECORD`. If there exists a `TOPIC_RECORD` with the given topic name and the topic ID, the topic exists. -If the topic already exists, the broker returns an error code of `36` (TOPIC_ALREADY_EXISTS). +If the topic already exists, the broker returns an error code of `36` (TOPIC_ALREADY_EXISTS), with the `error_message` field containing `Topic '' already exists.`. + +Reserved topic names include: +- `__cluster_metadata` - KRaft metadata topic + +If the topic name is reserved, the broker returns an error code of `42` (INVALID_REQUEST), with the `error_message` field containing `Creation of internal topic __cluster_metadata is prohibited.`. We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: @@ -25,21 +30,19 @@ In this stage, you'll need to implement topic existence checking by reading the The tester will execute your program like this: ```bash -./your_program.sh -# Create a topic first -# Then try to create the same topic again +./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request for an existing topic. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request for an existing topic and a reserved topic name. The tester will validate that: - The first 4 bytes of your response (the "message length") are valid. - The correlation ID in the response header matches the correlation ID in the request header. -- The error code in the topic response is `36` (TOPIC_ALREADY_EXISTS). +- The error code in the topic response is `36` (TOPIC_ALREADY_EXISTS) or `42` (INVALID_REQUEST) depending on the request. - The `throttle_time_ms` field in the response is `0`. - The `name` field in the topic response corresponds to the topic name in the request. -- The `error_message` field contains a descriptive error message. +- The `error_message` field contains the expected error message. - The `num_partitions` and `replication_factor` fields are `-1`. ## Notes diff --git a/stage_descriptions/creating-topics-05-hh9.md b/stage_descriptions/creating-topics-04-hh9.md similarity index 100% rename from stage_descriptions/creating-topics-05-hh9.md rename to stage_descriptions/creating-topics-04-hh9.md diff --git a/stage_descriptions/creating-topics-06-fl3.md b/stage_descriptions/creating-topics-05-fl3.md similarity index 100% rename from stage_descriptions/creating-topics-06-fl3.md rename to stage_descriptions/creating-topics-05-fl3.md diff --git a/stage_descriptions/creating-topics-07-rk2.md b/stage_descriptions/creating-topics-06-rk2.md similarity index 100% rename from stage_descriptions/creating-topics-07-rk2.md rename to stage_descriptions/creating-topics-06-rk2.md From c0dec1ea8ac9127298cf25b151b2c1af0619239a Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:54:26 +0530 Subject: [PATCH 48/52] docs: refine validation-only mode description for CreateTopics requests Clarify the handling of CreateTopics requests in validation mode by updating the stage description. Changes include improved wording for validation steps, specifying that no topic metadata is written to the log file, and enhancing the testing command for better clarity. --- stage_descriptions/creating-topics-04-hh9.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/stage_descriptions/creating-topics-04-hh9.md b/stage_descriptions/creating-topics-04-hh9.md index 593ca60..1c1e86e 100644 --- a/stage_descriptions/creating-topics-04-hh9.md +++ b/stage_descriptions/creating-topics-04-hh9.md @@ -1,17 +1,17 @@ -In this stage, you'll add support for handling CreateTopics requests with validation-only mode. +In this stage, you'll add support for handling `CreateTopics` requests in validation mode. ## Validation-Only Mode When a Kafka broker receives a CreateTopics request with the `validate_only` flag set to `true`, it needs to perform all validation checks as if creating the topics but without actually creating them or modifying any persistent state. This allows clients to validate topic creation parameters without side effects. The broker performs the following steps: -1. Validates the topic name (from previous stages) -2. Checks that the topic doesn't already exist (from previous stages) +1. Validates the topic name +2. Checks that the topic doesn't already exist 3. Validates partition count and replication factor parameters 4. Returns success/error responses as if creating the topics -5. Does NOT write any `TOPIC_RECORD` to the `__cluster_metadata` topic -6. Does NOT create any partition directories -7. Does NOT modify any persistent state +5. Does NOT write any `TOPIC_RECORD` to the `__cluster_metadata` topic's log file +6. Does NOT create any partition directories in the log directory +7. Does NOT modify any persistent state (e.g. topic metadata, partition files, etc.) The response should be identical to what would be returned for actual topic creation, but without any persistence side effects. @@ -27,7 +27,7 @@ In this stage, you'll need to check the `validate_only` flag in the request and The tester will execute your program like this: ```bash -./your_program.sh +./your_program.sh /tmp/server.properties ``` It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with `validate_only` set to `true`. @@ -39,7 +39,7 @@ The tester will validate that: - The error code in the topic result is `0` (NO_ERROR) for valid topics. - The `throttle_time_ms` field in the response is `0`. - The `name` field in the topic result corresponds to the topic name in the request. -- NO topic metadata is written to the `__cluster_metadata` topic. +- NO topic metadata is written to the `__cluster_metadata` topic's log file. - NO topic directories are created in the log directory. - The response is identical to what would be returned for actual topic creation. From 0d01d5c2a7e34bb748131da3bdc185a07b1ea159 Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:58:52 +0530 Subject: [PATCH 49/52] feat: add stage descriptions for single and batch topic creation in CreateTopics API Introduce new stage descriptions for handling CreateTopics requests, detailing the processes for creating a single topic and multiple topics in a single request. The updates include validation steps, metadata handling, and testing procedures, along with links to interactive protocol inspectors for enhanced understanding of the request and response structures. --- .../{creating-topics-06-rk2.md => creating-topics-05-rk2.md} | 4 ++-- .../{creating-topics-05-fl3.md => creating-topics-06-fl3.md} | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) rename stage_descriptions/{creating-topics-06-rk2.md => creating-topics-05-rk2.md} (92%) rename stage_descriptions/{creating-topics-05-fl3.md => creating-topics-06-fl3.md} (96%) diff --git a/stage_descriptions/creating-topics-06-rk2.md b/stage_descriptions/creating-topics-05-rk2.md similarity index 92% rename from stage_descriptions/creating-topics-06-rk2.md rename to stage_descriptions/creating-topics-05-rk2.md index 550fdef..eac120a 100644 --- a/stage_descriptions/creating-topics-06-rk2.md +++ b/stage_descriptions/creating-topics-05-rk2.md @@ -1,8 +1,8 @@ -In this stage, you'll add support for successfully creating topics with valid parameters. +In this stage, you'll add support for handling `CreateTopics` requests for a single topic. ## Single Topic Creation -When a Kafka broker receives a CreateTopics request for a valid topic name that doesn't already exist, it needs to validate the parameters, create the topic metadata, persist it to the `__cluster_metadata` topic, create the topic directory structure, and return a successful response. +When a Kafka broker receives a CreateTopics request for a valid topic name that doesn't already exist, it needs to validate the parameters, create the topic metadata, persist it to the `__cluster_metadata` topic's log file, create the topic directory structure, and return a successful response. The broker performs the following steps: 1. Validates the topic name (from previous stages) diff --git a/stage_descriptions/creating-topics-05-fl3.md b/stage_descriptions/creating-topics-06-fl3.md similarity index 96% rename from stage_descriptions/creating-topics-05-fl3.md rename to stage_descriptions/creating-topics-06-fl3.md index 2b9ce96..db28747 100644 --- a/stage_descriptions/creating-topics-05-fl3.md +++ b/stage_descriptions/creating-topics-06-fl3.md @@ -1,4 +1,5 @@ In this stage, you'll add support for handling multiple topics in a single CreateTopics request. +In this stage, you'll add support for handling `CreateTopics` requests with multiple topics. ## Batch Topic Creation From b9791d91b479bb07780ec7bfba66a22d162f51d2 Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 12:59:04 +0530 Subject: [PATCH 50/52] docs: update stage descriptions for topic creation in CreateTopics API Enhance the documentation for the CreateTopics API by clarifying the handling of metadata. Specifically, update the descriptions to indicate that both `TOPIC_RECORD` and `PARTITION_RECORD` are not written to the `__cluster_metadata` log file during topic creation. Additionally, refine the steps for writing `PARTITION_RECORD`s to ensure clarity on their creation for each partition. Adjust the testing command details to reflect the correct parameters for topic creation requests. --- stage_descriptions/creating-topics-04-hh9.md | 2 +- stage_descriptions/creating-topics-05-rk2.md | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/stage_descriptions/creating-topics-04-hh9.md b/stage_descriptions/creating-topics-04-hh9.md index 1c1e86e..dcef5db 100644 --- a/stage_descriptions/creating-topics-04-hh9.md +++ b/stage_descriptions/creating-topics-04-hh9.md @@ -9,7 +9,7 @@ The broker performs the following steps: 2. Checks that the topic doesn't already exist 3. Validates partition count and replication factor parameters 4. Returns success/error responses as if creating the topics -5. Does NOT write any `TOPIC_RECORD` to the `__cluster_metadata` topic's log file +5. Does NOT write any `TOPIC_RECORD` or `PARTITION_RECORD` to the `__cluster_metadata` topic's log file 6. Does NOT create any partition directories in the log directory 7. Does NOT modify any persistent state (e.g. topic metadata, partition files, etc.) diff --git a/stage_descriptions/creating-topics-05-rk2.md b/stage_descriptions/creating-topics-05-rk2.md index eac120a..459b3de 100644 --- a/stage_descriptions/creating-topics-05-rk2.md +++ b/stage_descriptions/creating-topics-05-rk2.md @@ -10,11 +10,9 @@ The broker performs the following steps: 3. Validates partition count and replication factor parameters (request will include valid values) 4. Creates a new topic UUID 5. Writes a `TOPIC_RECORD` to the `__cluster_metadata` topic's log file -6. Writes `PARTITION_RECORD`s to the `__cluster_metadata` topic's log file +6. Writes `PARTITION_RECORD`s to the `__cluster_metadata` topic's log file for each partition 7. Creates partition directories in the log directory (e.g., `/tmp/kraft-combined-logs/topic-name-0/`) -8. Nested structure inside the `__cluster_metadata` topic's log file: -TODO -9. Returns a successful response with error code `0` (NO_ERROR) +8. Returns a successful response with error code `0` (NO_ERROR) We've created an interactive protocol inspector for the request & response structures for `CreateTopics`: @@ -29,10 +27,10 @@ We've also created an interactive protocol inspector for the `__cluster_metadata The tester will execute your program like this: ```bash -./your_program.sh +./your_program.sh /tmp/server.properties ``` -It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with valid parameters. +It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with a non-existent topic name. The tester will validate that: @@ -41,7 +39,9 @@ The tester will validate that: - The error code in the topic response is `0` (NO_ERROR). - The `throttle_time_ms` field in the response is `0`. - The `name` field in the topic response corresponds to the topic name in the request. -- The topic metadata is written to the `__cluster_metadata` topic. +TODO: details +- The topic metadata is written to the `__cluster_metadata` topic's log file. +TODO: details - The topic directory structure is created in the log directory. - The `num_partitions` and `replication_factor` fields in the topic response correspond to the request. From 7e5975551981b92d7c55199138b68de537e396bb Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 13:00:52 +0530 Subject: [PATCH 51/52] docs: enhance stage description for batch topic creation in CreateTopics API Refine the documentation for handling multiple topics in a CreateTopics request. Update the steps to clarify the writing of `PARTITION_RECORD`s to the `__cluster_metadata` log file for each partition and adjust the testing command to include the server properties file. Ensure that the handling of mixed success/failure scenarios is clearly articulated. --- stage_descriptions/creating-topics-06-fl3.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/stage_descriptions/creating-topics-06-fl3.md b/stage_descriptions/creating-topics-06-fl3.md index db28747..a88c06a 100644 --- a/stage_descriptions/creating-topics-06-fl3.md +++ b/stage_descriptions/creating-topics-06-fl3.md @@ -1,4 +1,3 @@ -In this stage, you'll add support for handling multiple topics in a single CreateTopics request. In this stage, you'll add support for handling `CreateTopics` requests with multiple topics. ## Batch Topic Creation @@ -11,8 +10,9 @@ The broker performs the following steps for each topic: 3. Validates partition count and replication factor parameters 4. Creates a new topic UUID (for successful topics) 5. Writes a `TOPIC_RECORD` to the `__cluster_metadata` topic's log file (for successful topics) -6. Creates partition directories in the log directory (for successful topics) -7. Returns individual results for each topic in the response array +6. Writes `PARTITION_RECORD`s to the `__cluster_metadata` topic's log file for each partition (for successful topics) +7. Creates partition directories in the log directory (for successful topics) +8. Returns individual results for each topic in the response array Mixed success/failure scenarios are handled gracefully - if one topic fails validation, it doesn't prevent other valid topics from being created. @@ -26,7 +26,7 @@ We've created an interactive protocol inspector for the request & response struc The tester will execute your program like this: ```bash -./your_program.sh +./your_program.sh /tmp/server.properties ``` It'll then connect to your server on port 9092 and send a `CreateTopics` (v6) request with multiple topics. @@ -39,10 +39,12 @@ The tester will validate that: - Each topic result has the correct error code (0 for success, appropriate error codes for failures). - The `throttle_time_ms` field in the response is `0`. - The `name` field in each topic response corresponds to the topic name in the request. +TODO: details - Successfully created topics have their metadata written to the `__cluster_metadata` topic. +TODO: details +- The topic directory structure is created in the log directory. - Failed topics have appropriate error messages. ## Notes -- The CreateTopics request can contain multiple topics, and each should be processed independently. -- The response should contain a result for each topic in the request, in the same order. \ No newline at end of file +- The CreateTopics request can contain multiple topics, and each should be processed independently. \ No newline at end of file From a3efc22e8e61302b3f6a15692a688c82b529028a Mon Sep 17 00:00:00 2001 From: Ryan Date: Sat, 19 Jul 2025 13:00:58 +0530 Subject: [PATCH 52/52] docs: update stage descriptions for CreateTopics API to reflect changes in topic creation handling Revise the documentation for the CreateTopics API stages, including updates to the handling of existing topic names and validation-only mode. Consolidate stages for clarity, ensuring that error codes and success cases are accurately described. Adjust the sequence of stages to improve the flow of topic creation scenarios. --- course-definition.yml | 13 +++---------- final-stages.md | 18 +++++++----------- 2 files changed, 10 insertions(+), 21 deletions(-) diff --git a/course-definition.yml b/course-definition.yml index e52f29f..9adfaee 100644 --- a/course-definition.yml +++ b/course-definition.yml @@ -278,24 +278,17 @@ stages: marketing_md: |- In this stage, you'll implement the CreateTopics response for invalid topic names. - - slug: "cb8" - primary_extension_slug: "creating-topics" - name: "CreateTopics with Reserved Topic Name" - difficulty: medium - marketing_md: |- - In this stage, you'll implement the CreateTopics response for reserved topic names. - - slug: "us2" primary_extension_slug: "creating-topics" name: "CreateTopics with Existing Topic Name" - difficulty: medium + difficulty: hard marketing_md: |- - In this stage, you'll implement the CreateTopics response for existing topic names. + In this stage, you'll implement the CreateTopics response for existing and reserved topic names. - slug: "hh9" primary_extension_slug: "creating-topics" name: "CreateTopics with Validation Only" - difficulty: medium + difficulty: hard marketing_md: |- In this stage, you'll implement the CreateTopics response for validation-only mode. diff --git a/final-stages.md b/final-stages.md index 7a07c28..5e3e908 100644 --- a/final-stages.md +++ b/final-stages.md @@ -6,20 +6,16 @@ Stage CT2: CreateTopics with Invalid Topic Name (Hard code error) - Handle invalid characters, reserved names, empty names - Error code: INVALID_TOPIC_EXCEPTION (17) -Stage CT3: CreateTopics with Reserved Topic Name (Check __cluster_metadata) -- Handle attempts to create system topics (__cluster_metadata) -- Error code: INVALID_REQUEST (42) +Stage CT3: CreateTopics with Existing Topic Name (Read __cluster_metadata) +- Handle attempts to create system topics (__cluster_metadata): INVALID_REQUEST (42) +- Handle duplicate topic creation attempts: TOPIC_ALREADY_EXISTS (36) -Stage CT4: CreateTopics with Existing Topic Name (Read __cluster_metadata) -- Handle duplicate topic creation attempts -- Error code: TOPIC_ALREADY_EXISTS (36) +Stage CT4: CreateTopics with Validation Only +- Success case without persisting any data -Stage CT5: CreateTopics with Validation Only -- Handle all validations but don't persist any data - -Stage CT6: CreateTopics with Valid Parameters (Success Case) +Stage CT5: CreateTopics with Valid Parameters (Success Case) - Successfully create single topic with valid parameters - Core success functionality -Stage CT7: Multiple topics in single CreateTopics request +Stage CT6: Multiple topics in single CreateTopics request - Handle multiple topics in a single request