Skip to content

Add JSON and Protobuf native schema support#279

Merged
blankensteiner merged 1 commit into
apache:masterfrom
Thorbenl:feature/json-protobuf-schemas
Feb 11, 2026
Merged

Add JSON and Protobuf native schema support#279
blankensteiner merged 1 commit into
apache:masterfrom
Thorbenl:feature/json-protobuf-schemas

Conversation

@Thorbenl
Copy link
Copy Markdown
Contributor

@Thorbenl Thorbenl commented Feb 10, 2026

Fixes #254

Description

Implements ISchema<T> for JSON and Protobuf message types, bringing DotPulsar closer to feature parity with the Java and Go Pulsar clients.

New schemas:

  • JsonSchema<T>: Serializes/deserializes using System.Text.Json. Supports optional JsonSerializerOptions for customization and an optional JSON schema definition string for broker-side schema compatibility checking. Uses SchemaType.Json.
  • ProtobufSchema<T>: Constrained to Google.Protobuf.IMessage<T>. Uses native protobuf serialization with SchemaType.ProtobufNative (type 20). Extracts the file descriptor from the message type as schema data. No new dependencies — Google.Protobuf is already a project dependency.

Other changes:

  • Schema.Json<T>() and Schema.Protobuf<T>() factory methods added to Schema.cs, following the existing pattern for AvroISpecificRecord<T>() and AvroGenericRecord<T>().
  • Added inner exception constructor to SchemaSerializationException to properly wrap serialization errors (matching SchemaException which already has this overload).
  • Added System.Text.Json package reference for netstandard2.0 and netstandard2.1 targets only (already part of the BCL for net8.0+).

Design decisions worth discussing:

  1. ProtobufNative (type 20) vs Protobuf (type 3): We chose ProtobufNative which uses the native protobuf file descriptor as schema data. Type 3 uses Avro-encoded schema representation which would require an Avro dependency. Happy to adjust if maintainers prefer type 3.
  2. JSON schema data: The default constructor sends empty schema data (matching how primitive schemas like BooleanSchema work). An overload accepting a schema definition string is available for broker-side schema evolution checking.
  3. IMessage<T> disambiguation: Google.Protobuf.IMessage<T> is fully qualified in the constraint to avoid ambiguity with DotPulsar.Abstractions.IMessage<T>.

Regression

No. This is a new feature adding two new schema types. No existing functionality is modified beyond adding an inner exception constructor to SchemaSerializationException.

Testing

  • 12 unit tests for JsonSchema<T> covering: construction, schema info, encode, decode, round-trip, custom JsonSerializerOptions (camelCase), schema definition passthrough, and empty bytes error handling.
  • 9 unit tests for ProtobufSchema<T> covering: construction, schema info (type, name, data), encode, decode, round-trip with StringValue and Int32Value, empty message, and empty bytes.
  • All 21 new tests pass.
  • All 224 existing unit tests still pass (1 pre-existing ZstdCompressionTests failure unrelated to this change).
  • Solution builds with 0 warnings, 0 errors across all 5 target frameworks (netstandard2.0, netstandard2.1, net8.0, net9.0, net10.0).

Implement ISchema<T> for JSON and Protobuf message types,
closing the gap with Java and Go Pulsar clients.

- JsonSchema<T>: uses System.Text.Json for serialization with
  optional JsonSerializerOptions and schema definition string
- ProtobufSchema<T>: uses Google.Protobuf (already a dependency)
  with native protobuf descriptor for schema data
- Add factory methods Schema.Json<T>() and Schema.Protobuf<T>()
- Add inner exception constructor to SchemaSerializationException
- Add System.Text.Json dependency for netstandard2.0/2.1 targets
- Include 21 unit tests covering round-trip, edge cases, and
  error handling

Fix apache#254
@blankensteiner blankensteiner merged commit 9e01e5c into apache:master Feb 11, 2026
3 of 4 checks passed
@blankensteiner
Copy link
Copy Markdown
Contributor

Thanks for the PR :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON, and Protobuf schemas

2 participants