SwiftTesting: Enhance Event Stream JSON ABI #3040
Conversation
bkhouri
force-pushed
the
t/main/augment_event_stream_json
branch
from
ad4afed to
bf363fb
Compare
stmontgomery
added
workgroup: needs pitch
| ### Enhanced Capabilities | ||
| With access to this metadata, tools can now offer features like: | ||
| - Filtering test runs by tags for faster feedback cycles | ||
| - Generating reports that group results by test categories | ||
| - Tracking performance regressions against defined time limits | ||
| - Automatically linking test failures to relevant bug reports |
There was a problem hiding this comment.
This section feels very repetitive with the "Immediate Benefits for Tool Developers" section, can they be consolidated or remove this section if it's redundant?
|
|
||
| - **ABI Version Protection**: New fields are conditionally included based on ABI | ||
| version checks, ensuring older tools continue to function without modification | ||
| - **Experimental Feature Migration**: The existing experimental `_tag` field is |
There was a problem hiding this comment.
The current experimental field is named _tags, plural
| replaced with the `tags` array. Since experimental features don't provide | ||
| stability guarantees, this replacement doesn't constitute a breaking change | ||
| - **Graceful Degradation**: Tools that don't expect the new fields will simply ignore | ||
| them, while updated tools can leverage the enhanced metadata |
| - **Flattened vs Structured Bug Information**: We chose a structured approach for bug | ||
| metadata to accommodate various bug tracking systems while maintaining extensibility | ||
|
|
||
| ### Implementation Approaches |
There was a problem hiding this comment.
| ### Implementation Approaches | |
| ### Unconditionally include optional fields |
|
|
||
| ### Potential Extensions | ||
| - **Additional Metadata**: Other test traits could be exposed as the ecosystem evolves | ||
| - **Enhanced Bug Integration**: More sophisticated bug tracking integration with status updates |
There was a problem hiding this comment.
This one feels pretty tentative to me. IMO "Future directions" don't need to be planned or expected; they may never happen. But I think they should be plausible and feel like a natural evolution of the product. It's certainly subjective, but if something like this were to be added I would expect it to be part of some related tool rather than an evolution of Swift Testing itself. If you agree, I'd suggest removing this bullet
|
|
||
| The new fields are conditionally included in the JSON output based on test | ||
| configuration: | ||
| - Fields are only included when the test actually uses the corresponding traits |
There was a problem hiding this comment.
Fields are only included when the test actually uses the corresponding traits
The word "uses" is a bit confusing to me, since some traits can be applied to a test but may not be used by that test in any meaningful way. I think a better word choice would be "applied" or "applied to", as in:
| - Fields are only included when the test actually uses the corresponding traits | |
| - The field corresponding to each trait is only included for a given test if the | |
| trait has been applied to the test |
There was a problem hiding this comment.
Reviewing the subsequent bullets, I think this section could become a (shorter) paragraph without a title:
The field corresponding to each trait is only included for a given test conditionally, if the
trait has been applied to the test.
| - Empty or unused traits result in omitted fields, keeping the JSON clean and | ||
| efficient |
There was a problem hiding this comment.
This bullet doesn't seem to add much to the previous one, especially if you take my suggestion.
Plus, I don't know of a way to specify an "empty" version of any of these traits; they all require at least one value (at least one tag, for example)
| - Fields are only included when the test actually uses the corresponding traits | ||
| - Empty or unused traits result in omitted fields, keeping the JSON clean and | ||
| efficient | ||
| - This approach maintains backward compatibility while providing rich information |
There was a problem hiding this comment.
I think it's the versioning of the event stream which ultimately preserves backwards compatibility; clients that need to maintain compatibility can choose to continue using an older version until they're ready to upgrade.
I commented separately, but I think this bullet could be removed and you could consolidate this group of bullets into a sentence
2 participants
This proposal adds test metadata (
tags,bugsandtimeLimit) to Swift Testing's event JSON ABI, enabling richer external tooling capabilities.Links: Implementation | Forum Pitch