Use and effects of @JsonIdentityInfo for recursive json schema definitions

[scottslewis]

I'm a little new to the Jackson API, although very experienced in Java.

I'm working on a proposal for adding Groups to the MCP protocol. The mcp-java-sdk uses Jackson (2.19.something I think).

In the Groups proposal I've suggested introducing a nested Group structure, by adding a 'parent' Group attribute. This provides for arbitrary hierarchy of groups.

With normal serialization/deserialization of json with this schema the Group.parent recursive/nested structure could produce a lot of duplication of groups (for example if the same nested hierarchy of Groups was assigned to multiple Tools).

I've heard that the @JsonIdentityInfo annotation can be used for this situation, to replace all but the first given Group instance with a json reference, so that on deserialization the Group tree structure is reconstituted by clients.

I've read the docs for @JsonIdentityInfo and it says this:

...In practice this is done by serializing the first instance as full object and object identity, and other references to the object as reference values....

My question is: Is adding @JsonIdentityInfo to the Group type...e.g. here all that's necessary to get this 'in practice' behavior? (i.e. serialize to json the first reference to a given Group, and send a json reference for all other references?). Or are there other options required for getting this in practice behavior (and/or changing/customizing that behavior)?

[cowtowncoder]

Without knowing full details/intents I could be wrong, but I think use of @JsonIdentityInfo with proper settings should be sufficient, yes.
(I couldn't see what "here" is meant to link to fwtw)

Also quick note on "in practice": that's clumsy way of saying "this is how Jackson implements it" (but implying other implementations could produce output that is not necessarily ordered this way).

[scottslewis]

Relative pointers would be even more complicated, I suspect, since now references need to be resolved more dynamically/incrementally. Either way, much more state-keeping/access is needed when references have semantic info instead of just being opaque ids. So the problem is not the absolute nature of references.

So could you summarize the actual problem? Given the use case of representing object graphs efficiently in json messages (streaming), that will work across impl language/libraries, what approach do you think would would work best?

Global (per JSON doc) scope

I suppose I don't know what an impl of this would look like...in Jackson specifically, but more generally in other lang/json libs as well.

[cowtowncoder]

So could you summarize the actual problem?

The problem/my concern is that Jackson may not be able to support use of Json References as the Object identifiers (via @JsonIdentityInfo and functionality that supports it). As far as I understand these definitions.

Global (per JSON doc) scope
I suppose I don't know what an impl of this would look like..

I think that can be ignored, it's more about what @JsonIdentityInfo on Jackson-handled POJOs would look like; and would not something other impls would support or expose (ability to have different scopes (value domains) within which Object Id must be unique -- but possibly overlapping with ids in other scopes).

[scottslewis]

So could you summarize the actual problem?

The problem/my concern is that Jackson may not be able to support use of Json References as the Object identifiers (via @JsonIdentityInfo and functionality that supports it). As far as I understand these definitions.

Ok. Actually my use case (mcp protocol addition) doesn't require full json references as object identifiers support. Let me detail a little:

I'm proposing adding a new primitive called Group to the mcp specification

To experiment with this, I've implemented this in the mcp java sdk, which uses Jackson. Here's the java/Jackson code for the new Group type. Note the Group 'parent' attribute. This is a recursive attribute that allows hierarchies of groups.

Using the @JsonIdentityInfo annotation and the "@id" works perfectly for hierarchical Groups...the first one in json serialization is complete and the subsequent ones are referenced as 1, 2, 3 (integers) as per the @JsonIdentifyInfo.

If JsonIdentityInfo is not present, then each Group instance is serialized as a copy...including all the other attribute values (description, name, title). As you can imagine with the recursive Group parent definition this can be very inefficient.

All I'm looking to do is to figure out a strategy that will interoperate with the other languages/json libs for the Group type only. There are no other recursive references in the mcp spec, so any solution (across languages) could be applied just for the Group type (e.g. custom serialization for that type only).

My initial thought was to try to use the json ref spec/standards....as I thought this would be more likely to already exist in mature json libs like Jackson.

But I'm not married to that strategy...it could very well be something custom to the Group type that is just implemented the same in all langs/libraries.

Does that ease your concerns at all?

[cowtowncoder]

Ok so my hesitation wrt json-ref is solely based on whether Jackson's @JsonIdentityInfo can produce them at all. And to make this concrete, it may help to look at ObjectIdGenerator:

public abstract class ObjectIdGenerator<T> {
       // .... other methods

       public abstract T generateId(Object forPojo);
}

so Object Id to use can only be based on Java Object being serialized, and not on serialization state (like location/path of Object being serialized). So it needs to be sort of "pure" Object Id.

This can be considered an unfortunate limitation in ObjectIdGenerator interface -- if json-ref (and similar) schemes were considered, method should be taking JsonGenerator as input. But changing that is not something that is likely to happen (being major compatibility-breaking change).

Does this make more sense?

[scottslewis]

Ok so my hesitation wrt json-ref is solely based on whether Jackson's @JsonIdentityInfo can produce them at all. And to make this concrete, it may help to look at ObjectIdGenerator:

public abstract class ObjectIdGenerator<T> {
       // .... other methods

       public abstract T generateId(Object forPojo);
}

so Object Id to use can only be based on Java Object being serialized, and not on serialization state (like location/path of Object being serialized). So it needs to be sort of "pure" Object Id.

This can be considered an unfortunate limitation in ObjectIdGenerator interface -- if json-ref (and similar) schemes were considered, method should be taking JsonGenerator as input. But changing that is not something that is likely to happen (being major compatibility-breaking change).

Does this make more sense?

Yes it does, thank you.

I understand it's not your problem to fix, but I would like to find some answer for recursive object definitions (e.g. Group) that could work across languages/json libs without copying/re-serializing every object (Group) in the graph multiple times and without a gigantic effort for every json library.

What about a new annotation (rather than @JsonIdentityInfo) that is specifically designed for using object refs? (e.g. @JsonObjectRefInfo)?

It could reuse existing classes/code as much as possible, and introduce new classes/API where required.

Another approach could be do implement the equivalent of @JsonIdentityInfo in each language/json library. I'm somewhat hesitant to do this, however, because it's not standardized at the json spec level afaik.

[scottslewis]

@cowtowncoder I've just discovered the JavaScript Object Graph project: https://github.com/jsog/jsog

Interestingly, it has a Jackson plugin implementation

https://github.com/jsog/jsog-jackson

that uses the @JsonIdentityInfo generator property to customize the generator. I'm looking over this code myself, but wanted to make you aware of it's existence too. Looks like it was produced a while ago, so I'm hoping it still functions as described with recent Jackson.

[cowtowncoder]

I vaguely recall JSOG. It got things almost right; too bad @ref marker Object was used (instead of plain Scalar id).
But yet, Jackson can actually handle that case (1-property JSON Object).

And yes, @JsonIdentityInfo handling has been stable for years, no breaking changes AFAIK.