Schema test coverage: split tests into versioned directories

[karenetheridge]

I noticed that all the schema tests are in tests/schema/fail and tests/schema/pass. These directories are going to get mighty huge as more and more tests are added for different spec versions.

• could these be split up by one more subdir, by openapi spec version? That will make it easier to spot (e.g. by inspecting filenames) whether a particular keyword has any coverage or not.
• this involves a bit more scripting, but instead of cloning all the passing 3.1 tests into 3.2, the test runner could do s/3.1/3.2 for all tests in the 3.1 dir and also run them against the 3.2 schema -- because all valid 3.1 schemas should be valid 3.2 schemas. The fail tests would have to be copied manually though as some 3.1/fail tests could be in 3.2/pass (we SHOULD copy every new 3.2/pass test into 3.1/fail -- e.g. openapi: 3.1 combined with additionalOperations should be a fail).

---

...addendum that is probably of zero interest to anyone but me: I was originally thinking about writing up a request that the 3.2 schema make this change:

-    pattern: '^3\.2\.\d+(-.+)?$'
+    pattern: '^3\.[12]\.\d+(-.+)?$'`

...so as to allow the 3.2 schema (and implementations defaulting to 3.2) to parse 3.1 OADs, but then I realized there would be no way to do the above: that is ensure that 3.1 documents don't include 3.2-only constructs by mistake (which would then fail to work on a 3.1-only implementation).

I was wanting to have to avoid my implementation having to parse the openapi line to determine the version if it was going to happily support 3.1 and 3.2 anyway -- since 3.2 is backwards compatible, so any implementation that supports 3.2 should be able to support everything and all OAD constructs in 3.1 so it seemed unnecessary. It was only when I started thinking about negative testing scenarios that I realized I was being dumb, and I'll have to bifurcate bits of my application for 3.2 after all, boo!

[ralfhandl]

We didn't add any "pass" test cases in 3.2, so this may be an idea for the future.

We did add a few lines to ten of the "pass" test cases to cover newly introduced properties, for example defaultMapping in a discriminator example, or summary in a Response Object.

The second bullet point of the proposal would have led to copying those ten pass/3.2 cases to the fail/3.1 folder, contributing to the size of the test case folders. With version 3.3 we would have to copy the new 3.3/pass test cases to both the 3.1/fail and 3.2/fail folders, and so on.

How would we have dealt with the ten additions to existing "pass" test cases?

1. Add the lines and move the file to the new version subdir?
2. Copy the file to the new version subdir and then add the lines?
   • This seems redundant as the existing file will also be tested.
3. Add a minimal file for the new feature (for example a stripped-down copy of the existing file)?

Options 2 and 3 would further contribute to the size of the test case folders.

I liked the idea at first, but it seems to worsen the problem that it is trying to solve.

[karenetheridge]

It just seemed weird to me that all the existing test cases were s/3.1/3.2/ in the main branch. Do we not want to be able to run code coverage tests on 3.1 anymore? Are they going to be run periodically in the v3.1-dev branch, which is the only place they live now? When edits/fixes are made to the tests in 3.2-dev (and merged to main), will they be backported to the v3.1-dev branch? It seemed to me it would be easier to keep these in sync if they were duplicated into 3.1 and 3.2 directories.

The second bullet point of the proposal would have led to copying those ten pass/3.2 cases to the fail/3.1 folder, contributing to the size of the test case folders. With version 3.3 we would have to copy the new 3.3/pass test cases to both the 3.1/fail and 3.2/fail folders, and so on.

Yes, that was intentional. New features in 3.x should always constitute failures in 3.x-1, so it seems reasonable to test for that, surely? But this could even be automated -- the test suite could automatically consider any file in 3.2/pass as virtually existing as a 3.1/fail test if that file did not already exist in 3.1/pass.

[ralfhandl]

Are they going to be run periodically in the v3.1-dev branch, which is the only place they live now?

Yes, the 3.1 schema tests run on each PR for the v3.1-dev branch, from which the 3.1 schemas are published.

[ralfhandl]

When edits/fixes are made to the tests in 3.2-dev (and merged to main), will they be backported to the v3.1-dev branch?

Schema tests are not merged back to main. They exclusively live in their version dev branch vX.Y-dev. On release only the src/oas.md file is merged back to main after being renamed to versions/X.Y.Z.md. See CONTRIBUTING.md, section Publishing for details.

[ralfhandl]

it would be easier to keep these in sync

Currently we don’t keep them in sync. We haven’t yet had a schema bug in 3.1 that also needed to be fixed in 3.2. If that happens, we‘d need fix PRs for each version dev branch with (almost) the same changes.

We also currently plan to create a new version dev branch v3.3-dev from dev, after which we‘d have to somehow update the spec markdown file, the schema files, and the schema tests from v3.2-dev.

@handrews Maybe we should create v3.3-dev from v3.2-dev? I've created

• How to create a new version dev branch such as v3.3-dev? #4880

to discuss and decide this.