CI: Store Passes in results.json + various improvements.

[IanButterworth]

• Adds option to enable saving Test.Pass results to DefaultTestSet (started as Test: add option to record passes and enable it for CI #57690 but more complete here) and enable it when Base.runtests is called when CI=true, to match when reports are saved
• Stops attributing testset duration to tests from that testset, it doesn't make sense.
• Make the test name resemble (as far as possible) the original test call. Making the name independent of test outcome should make it easier to group tests and identify flaky ones.
• store repeat counts as tags, rather than in the name
• moves the save to inside the test workers, where all test info is available, and the save can be done in parallel to save time , then jsons are made at the end and uploaded as an artifact in a tarball
• speed improvements to make saving time acceptable
• fixes a couple of tests that seem to get a little flakier with this

Note on the WeakRef test fix: If a test like @test x == y passes, the Pass object retains x. Previously to this the Pass object was still made, but must have been GC-ed immediately. This PR remakes the Pass object when appending it to DefaultTestSet without the captured data. I think that just delayed the GC a little to make the test flaky, so I moved the @test to outside the function to avoid capturing x.

Also buildkite had to fix an issue to make tests show up properly on our jobs:
https://forum.buildkite.community/t/test-failures-showing-up-in-test-digest-but-not-on-jobs/4306

[vchuravy]

Stop attributing testset duration to tests from that testset, it doesn't make sense.

Can you expand on this a little bit?

[IanButterworth]

Currently we report that a test takes the duration of its parent testset. So @test x == 1 could be reported to take like 10 minutes. I think it's misleading.

[vchuravy]

Oh yeah, that's silly. Can we capture useful duration times?

[IanButterworth]

Our Result types don't currently store timing information. We could add it, but tbh I'm not sure it's any more useful. The useful information is somewhere inbetween the testset and the @test duration. If testsets were strictly a single test setup + test then it'd probably be fair to report testset duration. But testsets are very commonly a group of many sequential setups and tests.

[IanButterworth]

It seems like dropping the value from the Pass does help memory usage but the weakref test is failing again. I guess that test just was dependent on the Pass object with the value captured being freed immediately and this somehow slightly delays it.

Also, it seems like bk considers tests with 0.0 duration to not have run.. these last few runs report no tests, so I think I'll just set all tests to 1.0 duration.

[IanButterworth]

Hitting a buildkite issue: https://forum.buildkite.community/t/test-failures-showing-up-in-test-digest-but-not-on-jobs/4306

(buildkite are looking into it)