[GitHub Request] Create new repository opensearch-protobufs

[karenyrx]

What is the type of request?

Repository Management

Details of the request

Purpose of the new repository

For the currently undergoing Client-Server GRPC project, (targeting to be released as part of the April OS 3.0 release), we would like to have a separate repository to store the Protobufs and generated code used for client <> server GRPC API. This repository would also include all tooling/scripts/CI to generate .proto files from the API spec and to convert the .proto files into protobuf generated code. We may consider linters and validation for the protobufs and other CI/CD, similar to the spec repo, to enforce some standards for protobufs which must be followed.

Separation from API spec repo

This repo would be separate from the existing opensearch-api-specification repo for several reasons:

1. Protobufs will require a separate expertise from JSON API spec, and therefore we propose the 2 repos have a different set of maintainers.
2. The requirements for merging to the opensearch-api-specification repo are much tighter and will slow down initial development speed of the Protobufs/GRPC features.
3. The API spec will continue to be the source of truth, and these protobufs will mostly be a downstream consumer of the spec, so decoupling them similar to opensearch-java-client/other language clients seem to make sense.
4. It will be easier to combine the two repos into one in the future, than to separate them out.

Details on intended usage of the repo

The repo will consist of:

1. Protobufs
   a) Raw *.proto files based on the API spec

        i) In the short term: we will have both handrolled protos and auto-generated protos in the repo. These will be stored in 2 separate folders. The former handrolled protos will be used for the GRPC endpoint development in the OpenSearch core repo, while the auto-generated protos will be used to test the development of the tooling. 
        ii) Long term: Once the tooling is fully developed, we will ensure these 2 sets of protobufs fully converge. 
   

   b) Build files/tooling to compile the protobufs

2. Generated code:
   a) The generated code for Java/Go/Python/etc languages, which can be imported as jars/packages into the downstream repos that need them. Having already packaged generated protobuf code makes it easy to import into the various repos (e.g. OpenSearch core, opensearch-java-client, opensearch-python, opensearch-benchmark, etc) and avoids duplicate efforts to regenerate them in every single repository.

3. Tooling and CI
   a) Tooling to auto generate the *.proto files from the opensearch-api-specification and GHAs to trigger the conversion scripts
   b) Tooling (i.e Bazel files / scripts) to produce the protobuf generated code using protoc, and CI to trigger it automatically upon .proto file changes

4. Linters/Validators (TBD)
   a) Tooling to validate and lint the generated *.proto files, to ensure they conform to Google's protobuf best practices, as well as conventions established within the OpenSearch org (more important for any portions that are hand-rolled)

Addendum

While we are aware that there is a new process being rolled out for repository creation in #296, in the interest of time, it would be preferable if we could continue using the old process for repo creation requests until the new process is approved and rolled out.

cc @peterzhuamazon @dblock

Additional information to support your request

We would ideally like a separate maintainer list for this repo, that consists of individuals who are closely working on the GRPC/Protobuf efforts, which may not be the same maintainers as the API Spec repo.

When does this request need to be completed?

Due to GRPC feature's planned release in April, we would like this repository as soon as possible (ideally within 1 week), to unblock PRs int he OpenSearch core repo for GRPC API support to be merged.

Notes

Track the progress of your request here: https://github.com/orgs/opensearch-project/projects/208/views/33.
Member of @opensearch-project/admin will take a look at the request soon.
Thanks!

[karenyrx]

Example repository setup: https://github.com/karenyrx/opensearch-protos

[dblock]

+1 from me

[peternied]

protos could be misconstrued to mean prototypes instead of protobuf. @karenyrx what do you think about an alternative name?

[karenyrx]

Thanks @peternied, good point, how about opensearch-protobufs?

[peternied]

@karenyrx Thanks!

[andrross]

+1 from me

[sam-herman]

I like the analogy to the various clients we have today (opesearch-py, opensearch-java etc..) and I agree that it's consistent with the existing methodology which works well and allows for multiple extension implementations.
so overall +1

[anastead]

The TSC voted with majority in favor for this repo creation. Please proceed
VOTE please
⬆️ = yes ⬇️ = no (edited)
⬆️
10

[peterzhuamazon]

Pending @karenyrx reply on a few repo details per offline chat.
Will update once it is ready and proceed with the ticket.

Thanks.

[karenyrx]

Repo Name/Project Name: opensearch-protobufs

Project license: Apache License 2.0

Project description and business value:

GRPC/Protobuf support within OpenSearch will offer better performance and support for a wider variety of clients. 

What customer problem are we trying to solve with this repo?

A unified place to store protobufs and generated protobuf code used for Client <> Server APIs, to avoid duplicate efforts of multiple repositories having to generate their own protobuf code.

Should this project be in OpenSearch Core/OpenSearch Dashboard Core? If no, why not?

No. These protobufs will be used in multiple repositories, including OpenSearch core, opensearch-java (along with other language clients), opensearch-benchmark, etc, so it does not make sense to put OpenSearch core as a depenedency of all the other repositories.

Why should we create a new repo at this time?

It is needed for the upcoming GRPC feature, planned to be released in April as part of the OS 3.0 launch.

Please also link to the feature review, the RFC/Feature brief in github and any PR/FAQs or other working backwards documents.

* https://github.com/opensearch-project/OpenSearch/issues/16787
* https://github.com/opensearch-project/opensearch-api-specification/issues/677

Source code and code review

https://github.com/karenyrx/opensearch-protos/blob/main/LICENSE 
This is a WIP and not production ready. 

Third-party code
N/A

Similar projects

https://github.com/opensearch-project/opensearch-api-specification
Similar to the API spec repo, there will be CI/scripts for API spec to downstream client generation (e.g. opensearch-java), as well as validation/linting scripts, and versioning. 

Support Expectations

Initial setup requires help configuring permissions to upload artifacts to the official OSS Maven repository.

Who will be supporting this repo going forward?

Individuals working closely on the GRPC/Protobuf effort. 

What is your plan (including staffing) to be responsive to the community (at a minimum, this should include reviewing PRs, responding to issues, answering forum questions?)

Review PRs, respond to issues, answering forum questions, and anything else required. 

Targeted OpenSearch Release

April 2025

Maintainer access

@karenyrx 
@amberzsy
@lucy66hw
@philiplhchan

[peterzhuamazon]

Thanks everyone, I have created a ticket with LF on creating the repo and invite initial maintainers:
https://jira.linuxfoundation.org/plugins/servlet/desk/portal/2/IT-27894

Thanks.

[peterzhuamazon]

Hi guys, would you please accept your invite to the repo?
@karenyrx
@amberzsy
@lucy66hw
@philiplhchan

Thanks.

[peterzhuamazon]

We still have a few tasks need to complete before we will officially deliver the repo.
Running a few automations on settings up rules, secret, etc. next week.

Thanks.

[peterzhuamazon]

Update:

We are on track to deliver the repo either by EOD Thursday or early Friday.

Thanks.

[peterzhuamazon]

Hi @karenyrx @amberzsy @lucy66hw @philiplhchan

https://github.com/opensearch-project/opensearch-protobufs

The above repo has been created.

Next steps:

• Please make sure you copy over the relevant files from https://github.com/opensearch-project/.github
• If you are a plugin repository for OpenSearch or OpenSearch-Dashboards, make sure your github workflows/ci checks are running on top of the production docker images used for production distribution build on Jenkins (Example), then follow this (Example2) as a few changes needs to be made since first Example.
• Please make sure to update the MAINTAINERS.md and CODEOWNERS before adding any code.
• All new maintainers after the initial ones need to go through the public process to add a maintainer
• Read about triage. Create an "untriaged" label with the official color of "#FBCA04" in your repo. Copy the untriaged label workflow that auto-labels all new issues as "untriaged" to ".github/workflows"
• Make sure you've read and understand the branching strategies for the OpenSearch Project.
• Add code coverage to your GitHub actions workflow. You can view the example here.
• Update the “About” of the project.

Happy Repo’ing!

Thanks,
Peter

[peterzhuamazon]

Will close this issue as the repo is public now.
Please feel free to reopen if needed.

Thanks.