continue prost by jokemanfire · Pull Request #286 · containerd/ttrpc-rust

jokemanfire · 2025-02-24T06:03:08Z

work continue with #173

Tim-Zhang · 2025-02-25T06:29:14Z

@jokemanfire I appreciate you taking on this work very much.
Why don't we directly reopen #173 and continue the work?
Just let me know if there are any difficulties.

Thanks agian.

jokemanfire · 2025-02-25T06:42:20Z

@jokemanfire I appreciate you taking on this work very much. Why don't we directly reopen #173 and continue the work? Just let me know if there are any difficulties.

Thanks agian.

There are some conflicts and API changes that need to be resolved and some code needs to be packaged to adapt to the current API. It's not easy to operate with the previous version of PR. I need to add some commit to complete this feature. I have communicated with the original author.

jokemanfire · 2025-03-13T07:05:43Z

@Tim-Zhang @justxuewei Local test all pass but i don't know how to install protoc in CI.

Tim-Zhang · 2025-03-31T02:55:36Z

@jokemanfire FYI
https://github.com/marketplace/actions/setup-protoc
https://www.andreamedda.com/posts/go-buf-github-actions/

jokemanfire · 2025-04-09T09:53:13Z

Basically enough @Tim-Zhang @justxuewei just take a look . :)

justxuewei · 2025-04-10T02:09:42Z

Nice work, thanks @jokemanfire! Could you cleanup your commits? It makes us easier to review and maintain.

Use git rebase instead of git merge.
Modify the commit title to {subsystem}: {brief title for your changes}, and give it a brief introduction in the commit message.
Split your commits, and don't mix bugfixes, refactoring and new features.
Please modify this PR title to something like "codegen: Introduce a new codegen using prost".

Thanks!

This commit refactors the ttrpc-codegen and the compiler, and merges the two crates into a single crate, named "codegen". The codegen uses prost crate, a protobuf compiler for Rust. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

The ttrpc provides a "prost" feature to support the new version of codegen. An "example2" has been added to demonstrate how to use the codegen. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

Install protoc when executing `make deps` of the ttrpc. Add codegen's check and build, and example2' build to the ci testing. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

Fix all issues reported by cargo clippy to make ci testing pass. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

merge from remote branch. ref: containerd#173 Co-authored-by: Xuewei Niu <niuxuewei.nxw@antgroup.com> Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

fix ci for protoc Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

because of the api change 1. fix the marco in prost 2. fix the test Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

fix rebase error. Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

jokemanfire · 2025-04-10T06:18:28Z

Nice work, thanks @jokemanfire! Could you cleanup your commits? It makes us easier to review and maintain.好的工作，谢谢！你能清理一下你的提交吗？这让我们更容易审查和维护。

Use git rebase instead of git merge.使用 git rebase 代替 git merge 。

Modify the commit title to {subsystem}: {brief title for your changes}, and give it a brief introduction in the commit message.修改提交标题为 {subsystem}: {brief title for your changes} ，并在提交信息中简要介绍。

Split your commits, and don't mix bugfixes, refactoring and new features.拆分您的提交，不要混合修复错误、重构和新功能。

Please modify this PR title to something like "codegen: Introduce a new codegen using prost".请将此 PR 标题修改为类似于“codegen：引入使用 prost 的新代码生成器”的标题。

Thanks! 谢谢！

I have changed it.

skip build example2 in windows Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

justxuewei · 2025-04-10T09:15:52Z

Once this feature gets merged, I prefer to bump a major version to 2 to avoid some compatibility issues, as it nearly changes the entire codegen engine. WDYT? @jokemanfire @Tim-Zhang

jokemanfire · 2025-04-10T11:21:09Z

Once this feature gets merged, I prefer to bump a major version to 2 to avoid some compatibility issues, as it nearly changes the entire codegen engine. WDYT? 一旦这个功能合并，我更倾向于将主版本号提升到 2，以避免一些兼容性问题，因为它几乎改变了整个代码生成引擎。你怎么看？@jokemanfire @Tim-Zhang

This modification is indeed quite significant, and I agree. It depends on the maintainer's thoughts

Tim-Zhang · 2025-04-16T06:25:16Z

    let path: PathBuf = [out_dir.clone(), "mod.rs".to_string()].iter().collect();
    fs::write(path, "pub mod ttrpc;").unwrap();

+    generate_ttrpc(&out_dir);


To get unified internal APIs which was defined in ttrpc.proto, how about keeping build script untouched?

I think we can try to avoid so many condition compiling sentences like #[cfg(not(feature = "prost"))] in code.

right ,I think adding a transformation to the camel hump naming convention can reduce this conditional compilation code

while use protoc,rename the Code enum. Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

fix marco Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

jokemanfire · 2025-05-08T06:25:28Z

I tried to minimize conditional compilation as much as possible @Tim-Zhang @justxuewei

jokemanfire · 2025-06-24T07:51:15Z

@Tim-Zhang Can you take some time to advance this feature? In the previous puncture, it was found that the prost library uses less memory during operation than the Rust Protobuf library

Tim-Zhang

Several generator edge cases, size-limit regressions, and CI issues remain. The inline comments also include refactoring and idiomatic-Rust feedback focused on containing backend-specific behavior, reducing duplication, and avoiding unnecessary allocations and API divergence. Please address the inline findings before merge and add focused fixtures for imported types, multiple services with overlapping method names, and package-less schemas.

Tim-Zhang · 2026-08-05T13:09:44Z

+1. The protoc should be installed.
+2. Enabling "prost" feature for the ttrpc-rust.
+3. The Rust files are named based on their package name, rather than the proto
+   filename, e.g. `ttrpc = { version = "1.0", features = ["prost"] }`.


This example enables prost while keeping default features, which also enables rustprotobuf; the mutual-exclusion check then aborts compilation. Please document default-features = false and include the desired runtime feature explicitly, for example features = ["sync", "prost"].

Tim-Zhang · 2026-08-05T13:09:44Z

+    () => {};
+    ($first:tt $(,$rest:tt)*) => {
+        $(
+            #[cfg(all(feature = $first, feature = $rest))]


docs.rs is configured with all-features = true, so it activates both protobuf backends and always reaches this compile error. Configure docs.rs with one explicit compatible backend feature set instead of all features.

Tim-Zhang · 2026-08-05T13:09:44Z

+    fn method_handler_impl_sync_token(&self, struct_name: &Ident, method: &Method) -> TokenStream {
+        let mod_path = ttrpc_mod();
+        let context = self.ttrpc_context(false);
+        let input_type = format_ident!("{}", method.input_type);


method.input_type can be a qualified path such as super::google::protobuf::Empty. Passing it to format_ident! panics because it is not a single identifier. Use the path-aware type_token(&method.input_type) conversion here and in the async handler.

Tim-Zhang · 2026-08-05T13:09:44Z

+    fn finalize_package(&mut self, _package: &str, _buf: &mut String) {}
+    /// Generate services
+    fn generate(&mut self, service: Service, buf: &mut String) {
+        self.generate_type_aliases(buf);


prost-build calls generate once per service and appends services from a package into the same module. Emitting HashMap, Arc, Message, and async_trait imports here for every service causes E0252 on packages with multiple services. Emit package-level imports once, for example from finalize_package.

Tim-Zhang · 2026-08-05T13:09:44Z

+    }
+
+    fn method_handler_token(&self, service: &Service, method: &Method) -> TokenStream {
+        let struct_name = format_ident!("{}Method", to_camel_case(method.proto_name.as_str()));


The handler type is derived only from the method name, so Foo.Get and Bar.Get in one package both generate GetMethod. Include the service name in this type, for example FooGetMethod.

Tim-Zhang · 2026-08-05T13:09:44Z

+    }
+}
+
+#[derive(Default)]


Idiomatic Rust: the boolean and enum settings can store their defaults directly rather than using Option, and build should normally consume self. The P: Default bound is unrelated to paths, while borrowing all inputs makes this builder unnecessarily restrictive. Consider owned PathBuf/Vec<PathBuf> fields and build(self).

Tim-Zhang · 2026-08-05T13:09:44Z

 # lock home to avoid conflict with latest version
 home = "=0.5.9"
 protobuf-codegen = "3.1.0"
+prost-build = { version = "0.13", optional = true }


Refactor/dependencies: the runtime uses prost 0.11 here while the root build uses prost-build 0.13, and the new codegen crate pins the Prost family to 0.11. Keep the Prost crates on one compatible version and centralize the versions to avoid duplicate dependency trees and generated/runtime drift.

Tim-Zhang · 2026-08-05T13:09:44Z

+    let mut camel_case_name = String::with_capacity(name.len());
+    for s in NameSpliter::new(name) {
+        let mut chs = s.chars();
+        camel_case_name.extend(chs.next().unwrap().to_uppercase());


Refactor/correctness: this custom name splitter can yield an empty segment after leading underscores and then panic at this unwrap; it also reimplements casing already handled by established Rust casing utilities and by normalized prost-build names. Prefer those normalized names or a well-tested casing helper.

Tim-Zhang · 2026-08-05T13:09:44Z

@@ -0,0 +1,55 @@
+[package]


Scope/refactor: example2 duplicates the existing examples and brings in large third-party proto trees solely to switch protobuf backends, so future fixes and tests can diverge. Prefer running the existing example under a backend feature matrix and add small focused fixtures only for Prost-specific codegen behavior.

Tim-Zhang · 2026-08-05T13:09:44Z

+example2/protocols/**/*.rs
+!example2/protocols/**/mod.rs
 src/ttrpc.rs
+example2/protocols/**/*.rs


These example2 ignore/negation rules duplicate lines 10-11. Please remove the redundant entries and keep the file's final newline.

Superseded by the inline review with additional refactoring feedback.

Tim-Zhang · 2026-08-05T14:00:54Z

@jokemanfire I have left a few review comments, and btw please resolve the conflicts, thanks.

jokemanfire · 2026-08-06T01:29:43Z

@jokemanfire I have left a few review comments, and btw please resolve the conflicts, thanks.

thanks , I will take some time to resolve it.

jokemanfire changed the title ~~continue prost~~ continue prost (not complete) Feb 24, 2025

jokemanfire force-pushed the prost branch from ee4481d to 2bad764 Compare February 28, 2025 14:14

jokemanfire force-pushed the prost branch 2 times, most recently from dcf4304 to 781c816 Compare March 13, 2025 07:00

jokemanfire changed the title ~~continue prost (not complete)~~ continue prost Mar 13, 2025

jokemanfire force-pushed the prost branch 3 times, most recently from 21ebbd3 to 4a76b7b Compare April 1, 2025 12:19

jokemanfire force-pushed the prost branch 3 times, most recently from a29312f to 5fadae0 Compare April 9, 2025 09:50

jokemanfire force-pushed the prost branch from 5fadae0 to b74a116 Compare April 10, 2025 01:58

justxuewei and others added 7 commits April 10, 2025 11:19

codegen: Introduce a new codegen using prost

e777fc9

This commit refactors the ttrpc-codegen and the compiler, and merges the two crates into a single crate, named "codegen". The codegen uses prost crate, a protobuf compiler for Rust. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

ttrpc-rust: Support the new version of codegen

c8bacbd

The ttrpc provides a "prost" feature to support the new version of codegen. An "example2" has been added to demonstrate how to use the codegen. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

ci: Add codegen-related ci testing

da8166f

Install protoc when executing `make deps` of the ttrpc. Add codegen's check and build, and example2' build to the ci testing. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

codegen: Fix issues reported by cargo clippy

8e32522

Fix all issues reported by cargo clippy to make ci testing pass. Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>

fix: prost compile faild.

0ff5ede

merge from remote branch. ref: containerd#173 Co-authored-by: Xuewei Niu <niuxuewei.nxw@antgroup.com> Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

ci: change protoc install script in ci

5237576

fix ci for protoc Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

fix: fix test in protoc async

23256d3

because of the api change 1. fix the marco in prost 2. fix the test Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

jokemanfire force-pushed the prost branch from b74a116 to 23256d3 Compare April 10, 2025 05:53

fix: fix the rebase caused the error

eadbd7b

fix rebase error. Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

jokemanfire force-pushed the prost branch 2 times, most recently from aa000cc to 4da8148 Compare April 10, 2025 06:54

jokemanfire force-pushed the prost branch 2 times, most recently from b25b1a0 to 00fce61 Compare April 10, 2025 07:24

fix(example2): skip build example2 in windows

735a10e

skip build example2 in windows Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

jokemanfire force-pushed the prost branch from 00fce61 to 735a10e Compare April 10, 2025 07:40

Tim-Zhang reviewed Apr 16, 2025

View reviewed changes

jokemanfire added 2 commits May 8, 2025 12:47

fix(build.rs): reduce the Code which is enum's difference

ce3c169

while use protoc,rename the Code enum. Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

fix(macro): change marco's enum to same

26b0146

fix marco Signed-off-by: jokemanfire <hu.dingyang@zte.com.cn>

This comment was marked as outdated.

Sign in to view

Tim-Zhang requested changes Aug 5, 2026

View reviewed changes

Uh oh!

Conversation

jokemanfire commented Feb 24, 2025

Uh oh!

Tim-Zhang commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jokemanfire commented Feb 25, 2025

Uh oh!

jokemanfire commented Mar 13, 2025

Uh oh!

Tim-Zhang commented Mar 31, 2025

Uh oh!

jokemanfire commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justxuewei commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jokemanfire commented Apr 10, 2025

Uh oh!

justxuewei commented Apr 10, 2025

Uh oh!

jokemanfire commented Apr 10, 2025

Uh oh!

Tim-Zhang Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

jokemanfire Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

jokemanfire commented May 8, 2025

Uh oh!

jokemanfire commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Tim-Zhang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Tim-Zhang commented Aug 5, 2026

Uh oh!

jokemanfire commented Aug 6, 2026

Tim-Zhang commented Feb 25, 2025 •

edited

Loading

jokemanfire commented Apr 9, 2025 •

edited

Loading

justxuewei commented Apr 10, 2025 •

edited

Loading

jokemanfire commented Jun 24, 2025 •

edited

Loading

Tim-Zhang left a comment •

edited

Loading

Tim-Zhang Aug 5, 2026 •

edited

Loading

Tim-Zhang Aug 5, 2026 •

edited

Loading

Tim-Zhang Aug 5, 2026 •

edited

Loading

Tim-Zhang Aug 5, 2026 •

edited

Loading

Tim-Zhang Aug 5, 2026 •

edited

Loading