Implement a schema language to support generic tools and other languages

[yasammez]

Consider adding a schema language, like Capnp, Flatbuffers, and the like. From there, a code generator could, in a first step, generate Rust types with the known bitcode (and/or serde) annotations. In a further step, reliance on proc macros could be eliminated for users who wish to do so, but that is not the main benefit. The main benefits would be two-fold:

• Allow a generic cli tool to accept a schema file and an input stream to translate between the bitcode and a human friendly format for debugging purposes (introspection is a powerful tool when debugging complex problems in distributed systems). Also, versioned APIs could advertise their schema so that not-yet updated clients could fall back to a "dynamically typed" scheme (i.e. (de)serialise to/from a HashMap instead of a native type) to continue working with degraded performance until they can be patched.
• Allow interoperational implementations in other languages to exist.

I am willing to give this a shot myself, but if you stalk me, you'll notice that my own attempts at data serialisation have been lackluster so far, to say the least. Just let me know if you're interested!

[finnbear]

I'm curious what you are trying to do with bitcode. Porting it to other languages sounds time consuming and error prone (especially since the format is unstable) and you might be better served by using the Rust type system to implement your schema and derive bitcode traits on top of that.

[yasammez]

I am mostly interested in server to server communication with small-ish message sizes in scenarios where I don't control all sides. As it stands now, bitcode is automatically out if we want to give the teams free choice of technology.

[finnbear]

Thanks for the additional clarification. As far as I can tell, most of what you want is outside the scope of bitcode (the library) but within the scope of a new library/CLI tool (possibly within bitcode the repository but feel free to "give this a shot" yourself!).

For transcoding between serde formats, there is https://docs.rs/serde-transcode/latest/serde_transcode/. This could be used out-of-the-box to transcode between bitcode's serde::{Serialize, Deserialize} implementations and a human friendly format like JSON via serde_json. bitcode's Encode and Decode (which are incompatible format) could be handled by first deserializing and then serializing (less efficient but also practical). That translation could be wrapped in a CLI tool. The schema might as well be a subset of Rust's type system.

I am mostly interested ... communication ... where I don't control all sides.

This is honestly not a good use-case for bitcode, as the format is unstable between versions and it would be tedious to port to other programming languages.