Kafka version and metadata.version compatibility

[asoldo11]

Hello everyone, I have trouble finding out which Kafka version is compatible with which metadata.version.

I have deployed Strimzi operator 0.48.0 and deployed Kafka and KafkaConnect 4.1.0, with metadata version 4.1.0-IV1.

While testing, I stumbled upon some errors with Openlineage and Debezium connector and saw that bug is present at version 4.1.0 but not on 4.0.0.

As Strimzi operator supports both 4.0.0 and 4.1.0, I tried to downgrade Kafka per documentation:

1. first set metadata version to 4.0.0-IV0 and waited for cluster-operator to apply it on status.metadataVersion on Kafka resource. when I applied this, I got the following error:

2025-10-29 12:16:38 INFO KRaftMetadataManager:165 - Reconciliation #698(timer) Kafka(REDACTED): Updating metadata version from 4.1-IV1 to 4.0-IV0 2025-10-29 12:16:38 WARN KRaftMetadataManager:124 - Reconciliation #698(timer) Kafka(REDACTED): Failed to update metadata version to 4.0-IV0 (the current version is 4.1-IV1) org.apache.kafka.common.errors.InvalidUpdateVersionException: The update failed for all features since the following feature had an error: Invalid metadata.version 22. Refusing to perform the requested downgrade because it might delete metadata information.

Can someone explain me or point me to where I can check which metadata.version values are supported by which Kafka version? What is the relation between each and how do they match?

Thank you in advance

[scholzj]

The existing metadata versions can be found here: https://github.com/apache/kafka/blob/2daf5be27ae81eda9955a9d8bdfcf15499d955cc/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L165. But if you are already on metadata 4.1, you will not be able to downgrade to 4.0. That is a Kafka limitation.

[asoldo11]

Ok, but the 0.48.0 documentation is showing downgrade procedure from 4.1.0 to 4.0.0, that's why I thought everything should work.

ok, so once I switch metadata.version to match Kafka version - there is no going back? for instance, if I currently have 4.0.0 Kafka with metadata 4.0-IV3 and I upgrade only Kafka to 4.1.0, I can switch back to 4.0.0 in case of problems, but if I bump the metadata to 4.1-IV0, then I'm locked for 4.1.0 version?

So, in essence - I can only downgrade to older version if and only if metadata.version has not been upgraded along with Kafka?

[scholzj]

It is (not so) funny, isn't it? You need to update the metadata version to complete the upgrade. But afterwards, you cannot go back anymore. So you might not find out problems before ou change it, but if you find them, you are screwed. But I was explained by the Kafka folks that this is really the way it works :-/

[asoldo11]

Great, that settles everything. Thank you very much for the replies, keep up the good work 😄

[scholzj]

Yeah, sorry for this. I know it sucks. But there is not much we can do about it in Strimzi :-/.

[asoldo11]

Yeah, I completely understand there's nothing you can do about it. But it sucks to find out that once you migrate to new version you can't go back in case of any error. I mean, you can always spin up new cluster with old version, replicate topics, replicate clients and so on, but that's not topic of this discussion 😄

[scholzj]

Yeah. Spinning up a new cluster is not really a solution unless you are dealing with some ephemeral data which are just flowing in and out of Kafka such as when you use Kafka as a transport for logs or metrics.

FYI: This is the Slack discussion we discussed this last time: https://cloud-native.slack.com/archives/CMH3Q3SNP/p1753711786335679