[improve][io] Support Protobuf schema for Kafka source connector by jiangpengcheng · Pull Request #23954 · apache/pulsar

jiangpengcheng · 2025-02-10T00:31:59Z

Motivation

The Kafka source connector support schema registered in schema registry, but it only support Avro for now, while the schema registry also supports Protobuf, it's better to support it too

Modifications

Update the Kafka source connector to make it support Protobuf schema

Verifying this change

Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Added integration tests for using KafkaProtobufDeserializer to create Kafka source connector

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: jiangpengcheng#39

jiangpengcheng · 2025-03-13T01:25:20Z

/pulsarbot rerun-failure-checks

codecov-commenter · 2025-03-13T01:56:13Z

Codecov Report

Attention: Patch coverage is 36.70886% with 50 lines in your changes missing coverage. Please review.

Project coverage is 74.20%. Comparing base (bbc6224) to head (ba65b6b).
Report is 959 commits behind head on master.

Files with missing lines	Patch %	Lines
...a/org/apache/pulsar/io/kafka/KafkaSchemaCache.java	16.32%	41 Missing ⚠️
...a/org/apache/pulsar/io/kafka/KafkaBytesSource.java	70.00%	6 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23954      +/-   ##
============================================
+ Coverage     73.57%   74.20%   +0.63%     
+ Complexity    32624    32438     -186     
============================================
  Files          1877     1863      -14     
  Lines        139502   144332    +4830     
  Branches      15299    16467    +1168     
============================================
+ Hits         102638   107104    +4466     
+ Misses        28908    28771     -137     
- Partials       7956     8457     +501

Flag	Coverage Δ
inttests	`26.74% <0.00%> (+2.16%)`	⬆️
systests	`23.16% <0.00%> (-1.16%)`	⬇️
unittests	`73.72% <36.70%> (+0.87%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...a/org/apache/pulsar/io/kafka/KafkaBytesSource.java	`71.96% <70.00%> (+1.50%)`	⬆️
...a/org/apache/pulsar/io/kafka/KafkaSchemaCache.java	`16.32% <16.32%> (ø)`

... and 1064 files with indirect coverage changes

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jiangpengcheng · 2025-03-25T03:04:46Z

Hi @lhotari, could u help to review this pr, thx~

nlu90

@freeznet could you also help take a look?

…r_kafka_source

lhotari · 2025-04-23T15:41:06Z

@jiangpengcheng I have resolved the merge conflict after #24201 changes which upgraded Confluent Platform version to 7.8.2 and Kafka client version to 3.8.1. The integration test seems to fail now. I'm not sure exactly what the problem is.

I guess you are already aware that the way to run the integration test locally is to first build the docker image with this command:

mvn -B -am -pl distribution/io,distribution/offloaders,distribution/server,distribution/shell,tests/docker-images/latest-version-image install \
          -Pmain,docker -Dmaven.test.skip=true \
          -Dspotbugs.skip=true -Dlicense.skip=true -Dcheckstyle.skip=true -Drat.skip=true

And then run the test

mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -DintegrationTests -pl tests/integration -Dtest=KafkaSourceTest#testProtobuf

lhotari

There seems to be even more to resolving the complete protobuf schema as can be seen in the documentation at https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/serdes-protobuf.html .

For example:

support for multiple message types defined in the same schema
support for schema references

lhotari · 2025-04-23T15:48:47Z

+                    // the kafka protobuf serializer encodes the MessageIndexes in the payload, we need to skip them
+                    if (schemaType == SchemaType.PROTOBUF_NATIVE) {
+                        MessageIndexes.readFrom(buffer);
+                    }


There seems to be a meaning why the MessageIndexes exists. Each protobuf schema can include multiple message definitions and the MessageIndexes contains a solution for referencing a specific message.

wire format docs

Once the license issue is resolved, we could come back to this detail. If the code for KafkaProtobufDeserializer would be Apache 2.0 licensed, we could rather safely look at the code and see how the proper schema resolution can be handled for protobuf encoded messages. As long as we have the license issue, we better not copy-paste code due to IPR violation risk.

lhotari · 2025-04-23T16:14:12Z

One potential problem is the Confluent schema registry licenses. It says

The project is licensed under the Confluent Community License, except for the client-* and avro-* libs, which are under the Apache 2.0 license. See LICENSE file in each subfolder for detailed license agreement.

for example, https://github.com/confluentinc/schema-registry/blob/master/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/KafkaProtobufDeserializer.java doesn't have Apache 2.0 license.

https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java is under Apache 2.0 license.

lhotari · 2025-04-23T16:28:45Z


+    <dependency>
+      <groupId>io.confluent</groupId>
+      <artifactId>kafka-protobuf-serializer</artifactId>


According to the source repository, https://github.com/confluentinc/schema-registry, this library has a license of Confluent Community License. This didn't change between the previous
https://github.com/confluentinc/schema-registry/tree/v6.2.8 version and the currently used https://github.com/confluentinc/schema-registry/tree/v7.8.2 version.

The information provided by Confluent about the license is slightly conflicting since it says that client libraries are with Apache 2.0 license, but at the same time, it says "See LICENSE file in each subfolder for detailed license agreement". However, the license headers for protobuf classes are very explicit about the license. For Avro related classes, there's an Apache 2.0 license header.

In the pom file https://packages.confluent.io/maven/io/confluent/kafka-protobuf-serializer/7.8.2/kafka-protobuf-serializer-7.8.2.pom, there's Apache 2.0 license. This is also the case for 6.2.8 version, https://packages.confluent.io/maven/io/confluent/kafka-protobuf-serializer/6.2.8/kafka-protobuf-serializer-6.2.8.pom . I wonder if they just have forgotten to make things consistent in the repository.

I found an existing issue about the license for protobuf libraries: confluentinc/schema-registry#1558

there's even a branch to fix the issue, dating back to 2021, but it was never merged: https://github.com/confluentinc/schema-registry/compare/protobuf-licensing

Confluent employee replied in 2021: confluentinc/schema-registry#1558 (comment)

oh, I didn't check the library license, will close this then

lhotari · 2025-04-24T07:20:52Z

We cannot add a dependency on the io.confluent:kafka-protobuf-serializer library since it is under the Confluent Community License, as briefly commented in an issue discussion.

This is how Claude AI explained the licensing issue:

The Confluent Community License (CCL) and Apache 2.0 license have important compatibility considerations:

The main challenge is that the Confluent Community License is not an open source license according to the Open Source Initiative's definition. It contains field-of-use restrictions that prevent using the software to build competing services.

Specifically:

Apache 2.0 is a permissive open source license that allows code to be used in proprietary software

The CCL restricts using the licensed software to create "Competitive Offerings" (competing SaaS products)

This creates asymmetric compatibility:

You can include Apache 2.0 code in a CCL-licensed project

You generally cannot include CCL-licensed code in an Apache 2.0 project without the resulting work becoming subject to CCL restrictions

If you're developing software that combines components with these licenses, you should:

Carefully review the specific license texts

Consider the direction of code sharing between components

Consult with legal counsel for your specific use case

If you need to maintain Apache 2.0 compliance, you'll typically need to avoid incorporating CCL-licensed code in your Apache project.

Unless io.confluent:kafka-protobuf-serializer gets relicensed under Apache 2.0 license, we cannot use it in Apache Pulsar. We should also avoid looking at the code to avoid any IPR issues and instead implement a "clean room" implementation from scratch using the documentation of the wire format, since one could assume that there's no IPR on the wire format itself.

We could also consider supporting protobuf encoded messages that aren't encoded using Confluent's schema registry client libraries and don't use the schema registry for dynamically retrieving the schema. In those cases, the schema would have to be provided in the source connector config. That would be a different feature than what this PR implements.
@jiangpengcheng WDYT?

lhotari

We cannot add a dependency on the io.confluent:kafka-protobuf-serializer library since it is under the Confluent Community License, as briefly commented in an issue discussion.

lhotari · 2025-04-25T07:51:44Z

It looks like AWS Glue Schema Registry Library contains the classes that we'd need for parsing the Confluent Schema Registry protobuf wire format.

Under https://github.com/awslabs/aws-glue-schema-registry/tree/master/serializer-deserializer/src/main/java/com/amazonaws/services/schemaregistry there are both serializers and deserializers which we could use as a replacement for the Confluent Community Licensed libraries. We could continue to use the Apache 2.0 licensed Confluent Schema Registry libraries.

There are limitations in the AWS Glue libs, such as not supporting imported schemas like Confluent Schema Registry supports: https://github.com/awslabs/aws-glue-schema-registry/blob/72cbca0b05a758f0a775c39e580a15e7f19613fb/serializer-deserializer/src/main/java/com/amazonaws/services/schemaregistry/serializers/protobuf/MessageIndexFinder.java#L71-L72

lhotari · 2025-04-25T08:01:02Z

I also found Apicurio Schema registry libs which are Apache 2.0 licensed, https://github.com/Apicurio/apicurio-registry/tree/main/serdes/kafka/protobuf-serde/src/main/java/io/apicurio/registry/serde/protobuf . However, it looks like the wire format is Apicurio specific and not compatible with Confluent Schema Registry.

lhotari · 2025-04-25T08:10:57Z

More OSS schema registry clients with protobuf support:

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Feb 10, 2025

nlu90 assigned jiangpengcheng Mar 12, 2025

nlu90 added the ready-to-test label Mar 12, 2025

nlu90 approved these changes Apr 23, 2025

View reviewed changes

freeznet approved these changes Apr 23, 2025

View reviewed changes

jiangpengcheng added 11 commits April 23, 2025 14:47

[improve][io] Support protobuf for kafka source

e2af31f

fix style

1277a2b

Fix ci

b1098dc

Remove unused imports

cc78e3b

Fix ci

243f77a

Fix ci

16c10dc

Fix ci

8da7388

Debug

d30d08d

Fix ci

0159c1e

Remove debug

c364e54

Update

07d4919

jiangpengcheng force-pushed the support_protobuf_for_kafka_source branch from ba65b6b to 07d4919 Compare April 23, 2025 06:47

lhotari reviewed Apr 23, 2025

View reviewed changes

Comment thread pulsar-io/kafka/pom.xml

lhotari added 2 commits April 23, 2025 17:14

Merge remote-tracking branch 'origin/master' into support_protobuf_fo…

339d49e

…r_kafka_source

Remove duplication

3e546e5

lhotari reviewed Apr 23, 2025

View reviewed changes

lhotari requested changes Apr 24, 2025

View reviewed changes

jiangpengcheng closed this Apr 24, 2025

Conversation

jiangpengcheng commented Feb 10, 2025

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

jiangpengcheng commented Mar 13, 2025

Uh oh!

codecov-commenter commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jiangpengcheng commented Mar 25, 2025

Uh oh!

nlu90 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lhotari commented Apr 23, 2025

Uh oh!

lhotari left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhotari commented Apr 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhotari Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhotari commented Apr 24, 2025

Uh oh!

lhotari left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhotari commented Apr 25, 2025

Uh oh!

lhotari commented Apr 25, 2025

Uh oh!

lhotari commented Apr 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Mar 13, 2025 •

edited

Loading

lhotari Apr 23, 2025 •

edited

Loading

lhotari left a comment •

edited

Loading