How do we present errors when the error is that the schema(s) passed?

[jdesrosiers]

The oneOf keyword fails if more than one of the alternatives passes. Right now, the best we can do is tell the user that more than one of the schemas failed, but that's not terribly helpful. The user shouldn't have to go look at the schema to figure out what was wrong.

We have the same problem when not and contains.

In our first draft of this library, we were working on creating a description of what a schema describes. We could use something like that to create a message of some kind. The biggest problem with that approach is that schemas could easily get too complex for that to be feasible. I'll try to share some more examples later to help think about the problem.

What do you think? What strategies can we use to present meaningful error messages in these situations?

[jdesrosiers]

The test failures are unrelated to this issue. It was a mistake that has now been corrected.

[jdesrosiers]

For the oneOf case, I think we can use the "alternatives" field in the response, but instead of presenting things that failed validation (there weren't any failures), we present what's expected from each alternative.

{
  "oneOf": [
    { "required": ["a"] },
    { "required": ["b"] },
    { "required": ["c"] }
  ]
}

{ "a": 1, "b": 2 }

{
  "message": "More than one of the alternatives match",
  "alternatives": [
    [
      {
        "message": "The 'a' property is required",
        "instanceLocation": "#",
        "schemaLocation": "#/oneOf/0/required"
      }
    ],
    [
      {
        "message": "The 'b' property is required",
        "instanceLocation": "#",
        "schemaLocation": "#/oneOf/1/required"
      }
    ],
  ],
  "instanceLocation": "#",
  "schemaLocation": "#/oneOf"
}

We only need to show messages for the alternatives that passed. One optimization we could do is to remove any messages that are the same in every alternative. Anything that's the same in every alternative isn't useful in determining what needs to change to make only one option match.

[jdesrosiers]

The "not" case could work similarly, except there's always only one alternative.

{
  "not": { "pattern": "^a" }
}

"apple"

{
  "message": "Expected a value that doesn't match the alternative",
  "alternatives": [
    [
      {
        "message": "The string matches the regular expression /^a/",
        "instanceLocation": "#",
        "schemaLocation": "#/not/pattern"
      }
    ]
  ],
  "instanceLocation": "#",
  "schemaLocation": "#/not"
}

[jdesrosiers]

contains can be handled normally if there are not enough items matching. It's the case where there are too many matches where we run into the problem.

{
  "contains": { "type": "string" },
  "maxContains": 1
}

[42, "foo", "bar"]

{
  "message": "Expected an array that contains one item that passes the 'contains' schema",
  "alternatives": [
    [
      {
        "message": "A string",
        "instanceLocation": "#/1",
        "schemaLocation": "#/contains/type"
      }
    ],
    [
      {
        "message": "A string",
        "instanceLocation": "#/2",
        "schemaLocation": "#/contains/type"
      }
    ]
  ],
  "instanceLocation": "#",
  "schemaLocation": "#/contains"
}

[jdesrosiers]

The messaging in these examples needs work, but I think this approach should work. The challenge will be refactoring so that we can generate "passing" messages as well as "failing" messages. Hopefully, it doesn't end up meaning we need both an error handler and a passing handler for every keyword. I'm going to start working on a proof of concept, but I still want feedback on if you think this will work and more importantly, what you think won't work well.

[adityayadav-dev]

--just little off topic --> Hey! I've been exploring this repo for the past few days, and while running npm test, I noticed 6 tests were failing due to pluralization issues in the en-US.ftl translation file.
After investigating the error messages and test expectations, I identified the root cause in the Fluent translation syntax and fixed it. All 114 tests now pass. if you allow me then i can raise a issue on this and Happy to provide more details if needed!

[jdesrosiers]

As I said before, the problem has already been corrected. Try pulling down the most recent changes and try running the tests again. You should see them all passing.

In the future, please don't comment on issues that aren't related to you're comment. The best place for this question would have been the #gsoc-better-errors channel on Slack. Creating a new issue would have been ok too.

[RitoG09]

Thanks for sharing the examples, that clears my concept of this issue.
My understanding is that the core issue with oneOf not and contains keyword is not regular validation failure, but failure caused by unexpected success. This approach doesn’t appear to require passing handlers for all keywords, but only for a some of the logic keywords (oneOf not contains). And I think the method you mentioned will work great.

Yet I have a different approach (may be silly)->
we can logically separate keywords into two explanation patterns:

1. Invertible keywords ->[ required type pattern ]-> where a passing explanation can be derived by inverting the existing failure message.
2. Non-invertible keywords ->[ oneOf not contains ]-> which don’t meaningfully explain success themselves and instead need to aggregate explanations from the subschemas that matched.

For example, in a schema like,

oneOf: [{ required: ["a"] }, { required: ["b"] }],

messages such as “The 'a' property is required” are technically derived from the right keyword, but semantically misleading in a success context. Rephrasing these as something like “The 'a' property is present” makes it clear why the alternative matched and avoids confusion.

Invertible keywords (required) can produce passing explanations by inverting or reusing their existing failure messages, while Non-invertible keywords (oneOf) don’t generate passing explanations themselves and instead just aggregate explanations from the subschemas that matched. If that holds, it might not need parallel passing handlers for every keyword while still producing clearer messages.

[jdesrosiers]

Thanks for trying to think through the problem. I think there's actually three categories.

Some keywords are just simple applicators and don't have assertions of their own (allOf, properties, items). Those don't have error message of their own and they won't need success message either.

Other keywords are simple assertions (type, required). These have an error message and will need a success message as well.

The third category is keywords that are both applicators and assertions (oneOf, not). They apply a schema, but can also fail by themselves. For example, oneOf will fail if none of the alternatives match. That's the keyword's error message. It doesn't come from any of the alternative schemas.

However, I don't think any of that categorization matters. In the end, anything with an error message needs a success message. So, oneOf/not/contains will need success messages just like type/required/etc do.

So far, the only idea I have to make this work is to pass a boolean flag to the error handlers saying whether we're collecting error messages or success messages. The Localization message functions would also need to take that flag and switch between messages accordingly. That feels messy and complicated to me. I'm hoping to figure out a cleaner way, but I think the alternative is a lot of duplication, which isn't great either.