FEATURE REQUEST: option to ignore unrecognized attributes when creating a typed document from a JSON string

[rodw]

Suppose we have registered a person document type, with the following model:

{
  "uuid": "{\"type\":\"string\"}",
  "name": {
    "given": "{\"type\":\"string\"}",
    "family": "{\"type\":\"string\"}"
  }
}

Then we can easily parse a JSON string such as:

{
  "uuid": "404b9ff1-fb45-4276-9665-e6f87f725a9e",
  "name": {
    "given": "John",
    "family": "Doe"
  }
}

or (omitting some attributes this time):

{
  "name": {
    "given": "Jane"
  }
}

into a typed JDocument by calling something like:

Document person = new JDocument("person", json);
System.out.println(person.getType()); // yields "person"

However if the input JSON includes some extra attributes, say:

{
  "uuid": "404b9ff1-fb45-4276-9665-e6f87f725a9e",
  "dob": "1999-12-31", // <~ not defined in type spec
  "name": {
    "given": "John",
    "family": "Doe"
  }
}

then the new JDocument("person", json) call will trigger an exception because the unrecognized attributes (dob here) cause the document to fail validation.

In the spirit of Postel's Law - "be liberal in what you accept and conservative in what you send" - it would sometimes be helpful to simply ignore the unrecognized attributes and extract the otherwise valid person object "embedded" within that document.

One way to achieve this behavior is to first parse the input JSON as an untyped JDocument, strip out (using deletePath) any attributes not defined in the model and only then assign the type or attempt to validate the remaining attributes.

but for sake of discussion here's a utility method that implements that strategy:

/**
  * Create a valid, typed JDocument from a JSON string, ignoring any attributes
  * not defined in the specified model (but validating the recognized attributes
  * as we otherwise would).
  *
  * @param type name of the (previously loaded) document model to apply
  * @param json the stringified JSON data to be parsed
  *
  * @return a JDocument with of specified type that only contains the attributes
  *         defined in the model
  *
  * @throws UnifyException when the named type definition cannot be found, the input
  *                        json is not well-formed, or the resulting JDocument is
  *                        otherwise invalid even after any unrecognized attributes
  *                        are removed
  */
JDocument parseTypedJsonIgnoringUnrecognizedAttributes(final String type, final String json) throws UnifyException {
  // Grab the specified model definition...
  final Document model = JDocument.getDocumentModel(type);
  if (model == null) { // ...aborting with an exception if not found.
    throw new UnifyException("jdoc_err_29", type);
  } else {
    // Otherwise parse the input json as an untyped document...
    final JDocument doc = new JDocument(json);
    // ...and strip out any attributes that aren't defined in the model.
    for (String path : doc.flatten()) {
      // (note that arrays in the model will only contain a single element, so we must
      // change any array index references found in the untyped document path to index
      // 0 to create the corresponding path in the model document)
      final String modelPath = path.replaceAll("\\[\\d+\\]", "[0]");
      if (!model.pathExists(modelPath)) {
        doc.deletePath(path);
      }
    }
    // Now we can assign the type and validate as we usually would.
    doc.setType(type);
    doc.validate(type); // <~~ I'm not sure if this call is strictly necessary
    return doc;
  }
}

There may be a more elegant or efficient way to go about this - and I think it might be more appropriate to expose this as an optional ignoreUnrecognizedAttributes parameter to the JDocument constructor - but unless there's some nuanced edge case I haven't accounted for this general strategy seems to work.

If this is a feature you'd be interested in I'm happy to submit a full PR for it, but I thought it best to open it up for discussion/guidance/feedback first.

[deepakarora3]

Hi,

Thanks for taking out the time to write out the problem you are facing and also proposing a solution for the same. I really do appreciate the fact that you took time to write out the problem and solution in detail and very clearly too.

I wanted to give you the good news that we internally faced this very problem here in Amex in the project that we developed JDocs for and had already solved this a couple of years back. You will find the reference to it in the documentation on the main page itself (https://github.com/americanexpress/unify-jdocs), though I have to admit it is a bit buried and not very self-evident. Please search for the heading "Validating typed documents" to know the usage. I am reproducing the contents from the documentation below. You can have what you are looking for at per document level or at JDocs level. I hope this helps you. Happy to help if you have any further questions. Thanks and regards.

---

By default, typed documents are validated when they are created and when any read / write operation is performed on them. However there may be cases, when we do not want to validate the document at the time of creation but only at the time of reading and writing. This is a typical scenario in the use of APIs which can return extra blocks and paths which may not be present in the model document and which we may not be interested in. If we were to validate at the time of creating the document, the model validations would fail unless we kept the model document in sync with all the changes happening on the API side. Most of the times this is not possible as the teams are separate and there is no reason that adding fields to the response which we are not interested in should cause a failure. To take care of this scenario, we have two alternatives:

Use the overloaded JDocument constructor to specify that the validation should only be done while reading / writing paths. This way the default behaviour will not change but can be set at a per document level
Specify the default behaviour of JDocs to do the validation only while reading / writing paths. This can be set using the public static init method while initializing JDocs. Of course this can be overridden as desired by using the JDocument overloaded constructor
The two methods are shown below:

public JDocument(String type,String json,boolean validateAtReadWriteOnly);
public static void init(boolean defaultValidateAtReadWriteOnly);

---

[rodw]

I'm not sure if that's solving the same problem (or if it is it's not clear to me how).

My use case is something like this:

1. I have a REST API POST endpoint that accepts a JSON document that matches a particular schema in the request body.
2. I want to parse and validate the JSON input according to that schema, so (i) I can reject invalid request payloads with an HTTP 400 response and (ii) I can "safely" assume the typed doc.getString("$.known_attr") calls are going to work for the rest of the processing.
3. However I don't want to reject an input request payload simply because an additional attribute has been added.

Essentially this is same function as the @JsonIgnoreProperties(ignoreUnknown = true) annotation in Jackson.

As I understand it, setting validateAtReadWriteOnly to true will allow the initial parse to work, but:

A. won't the unrecognzied attributes still trigger a validation error when I ultimately do the validation?
B. until I've done the validation, I can't assume I'll be able to get the expected types for the known attributes.

I don't understand how to achieve the 2.i and 2.ii objectives above using validateAtReadWriteOnly. Doesn't that just defer the validation until later?

[rodw]

Re-reading your initial reply again, are you saying when validateAtReadWriteOnly == true the validation only happens on per-path (per-attribute) basis, on demand, as the paths are used in the getValue/setValue and related calls?

If yes, I guess I can see how that's pretty close to what I'm looking for (though it might be more convenient to (1) be able to detect the invalid payloads earlier in the workflow and (2) to know that if later choose to serialize that payload back to JSON to send it somewhere it won't contain any invalid attributes, even for paths I may not have directly touched yet).

For what it's worth, over several readings of the README I interpreted "read" and "write" in that context to mean "parsing" and "stringifing" the JSON document as a whole, not accessing individual attributes. Maybe it's just me, but (assuming the "per-path, on-use validation" interpretation is correct) it may be worth trying to call that out more explicitly.

[rodw]

A related question: assuming validateAtReadWriteOnly is set, will the document be validated when getJSON is called? I.e., is getJSON the equivalent of "reading" every path in the document (or maybe in the model), or does validateAtReadWriteOnly literally mean that the only time validation occurs is when getter/setter methods are invoked for a particular path?

[deepakarora3]

I see your point and agree - there needs to be a way to validate the document upfront (minus the fields not in the model) - I am just thinking of the most efficient and intuitive way to do this - give me a couple of days to think through and I will let you know - will implement in a week at max - hope that helps. BTW will update the documentation as well to make it more explicit - thanks!

[deepakarora3]

Hi - have released 1.6.0. Please go through the release notes / documentation and let me know if you have any questions. Thanks.