Testing stage's SCHEMA validation

[thozza]

@teg's comment on my pull request osbuild/osbuild#555 (comment) got me thinking, whether it would be worth to test stage's SCHEMA. Specifically whether it (dis)allows object properties as it was intended. Presumably allowed SCHEMA objects should be already tested by the "tree diff" testing. However disallowed SCHEMA properties are not tested. Specifically for invalid manifests, which specify additional disallowed properties in stage's section, we could have additional manifest in test/data/stages. This manifest could be passed to osbuild --inspect and the result checked against known failed result of JSON schema validation stored in additional .json file in the respective stage directory in test/data/stages.

Would be such thing useful and worth the effort?

[teg]

Sounds reasonable to me. We could also consider having checks that verify that all our schemas set the disallow additional prpoerties where they should (without explicit failing test-cases as they would be cumbersome to keep up-to-date for all the combinations). I think we always want to enforce that (?).

[gicmo]

I think enhancing the stage tests is a very good idea. They are quite inflexible right now. But I think I would wait until we got the new format done.

[mvo5]

Sorry, I only now stumbled upon this - I wonder if this can now be closed? With https://github.com/osbuild/osbuild/pull/1440/files there is a relatively straightforward way to write such tests now and this is used extensively in the "org.osbuild.kickstart" stage. It's implemented slightly differently from the suggested approach here though so maybe it shouldn't be closed(?).

[thozza]

Hey Michael. I think that your contribution covers the schema validation sufficiently. Thanks for it, I'll close this discussion.