dockerfile: install libzstd-dev for kafka zstd compression

[dbottini2]

Container images were built without libzstd-dev in the builder stage, so
librdkafka's find_package(ZSTD) failed and WITH_ZSTD was disabled. Kafka
producers in the shipped images could not use zstd compression.

Changes:

• dockerfiles/Dockerfile: add libzstd-dev to the builder and debug builder
  stages so librdkafka is compiled with zstd support.
• tests/runtime/out_kafka.c: add a runtime test asserting librdkafka accepts
  compression.codec=zstd (only true when WITH_ZSTD is compiled in).

Addresses #11366 (Packaging: Enable ZSTD compression by default).

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] local-build-all.sh builds the native distro packages under packaging/distros/. This PR only changes the container images (dockerfiles/Dockerfile), which that script does not build. The container image build was verified instead (see test evidence above).
• Set ok-package-test label to test for all targets (requires maintainer to do).

Test evidence

Example config (rdkafka.compression.codec=zstd):

[INPUT]
    Name    dummy
    Tag     test

[OUTPUT]
    Name                       kafka
    Match                      test
    Brokers                    127.0.0.1:9092
    Topics                     test
    rdkafka.compression.codec  zstd

Container build (dockerfiles/Dockerfile, --target=production) — zstd is now
detected so librdkafka is built with WITH_ZSTD:

-- Found ZSTD: /usr/lib/aarch64-linux-gnu/libzstd.so
WITH_ZSTD:BOOL=ON

Running the production image with the kafka zstd config — the output starts and
accepts the codec (only failure is the absent broker, as expected). Before this change
the same image rejected compression.codec=zstd at init:

[ info] [output:kafka:kafka.0] brokers='127.0.0.1:9092' topics='test'
[error] [output:kafka:kafka.0] ... Connect to ipv4#127.0.0.1:9092 failed: Connection refused

Runtime test (tests/runtime/out_kafka.c::zstd_compression_available) under Valgrind:

Test zstd_compression_available...              [ OK ]
SUCCESS: All unit tests have passed.
==13903== All heap blocks were freed -- no leaks are possible
==13903== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Notes on targets: amd64 and arm64 (the published distroless images) configure with
zstd and build successfully. The README's plain multi-arch example additionally lists
linux/arm/v7 and linux/s390x, which fail at CMakeLists.txt:310 because
-DFLB_SIMD=On is unsupported on those architectures; this is pre-existing and
unrelated to this change.

Documentation

• [N/A] Documentation required for this feature

Backporting

• Backport to latest stable release: dockerfile: install libzstd-dev for kafka zstd compression (4.2 backport)  #12004 (4.2 branch).

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[coderabbitai]

📝 Walkthrough

Walkthrough

Docker build images now install libzstd-dev, and the Kafka runtime test suite adds a check that compression.codec accepts zstd.

Changes

Zstd support updates

Layer / File(s)	Summary
Docker zstd packages dockerfiles/Dockerfile	The builder-base and debug stages install libzstd-dev with the existing development dependencies.
Kafka zstd availability test tests/runtime/out_kafka.c	A new runtime test includes rdkafka.h, sets compression.codec to zstd, asserts success, and registers the test in TEST_LIST.

Suggested reviewers

• edsiper
• fujimotos
• koleini

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A bunny hopped through Docker with a zstd carrot bright,
then poked the Kafka test to see compression take flight.
Two tiny tweaks went in, neat and spry,
now the fluffy build can zstd-ify.
🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately describes the main change: adding libzstd-dev for Kafka zstd compression support.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99acbc8514

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Gate the zstd check to zstd-enabled builds

In regular source builds where FLB_OUT_KAFKA=On but the host lacks a system libzstd-dev, librdkafka still builds with WITH_ZSTD=OFF because the CMake path only lets librdkafka's own find_package(ZSTD) decide; the Dockerfile install does not affect those builds. This unconditional assertion then makes flb-rt-out_kafka fail for an otherwise supported non-container build, so either wire zstd into the CMake Kafka build or only run this check when zstd support was actually required/enabled.

Useful? React with 👍 / 👎.