Store test content in a custom metadata section.

See also: swiftlang/swift#76698

Resolves #735.

Documentation/ABI/TestContent.md

@@ -0,0 +1,175 @@
+# Runtime-discoverable test content
+
+<!--
+This source file is part of the Swift.org open source project
+
+Copyright (c) 2024 Apple Inc. and the Swift project authors
+Licensed under Apache License v2.0 with Runtime Library Exception
+
+See https://swift.org/LICENSE.txt for license information
+See https://swift.org/CONTRIBUTORS.txt for Swift project authors
+-->
+
+This document describes the format and location of test content that the testing
+library emits at compile time and can discover at runtime.
+
+## Basic format
+
+Swift Testing uses the [ELF Note format](https://man7.org/linux/man-pages/man5/elf.5.html)
+to store individual records of test content. Records created and discoverable by
+the testing library are stored in dedicated platform-specific sections:
+
+| Platform | Binary Format | Section Name |
+|-|:-:|-|
+| macOS, iOS, watchOS, tvOS, visionOS | Mach-O | `__DATA_CONST,__swift5_tests` |
+| Linux, FreeBSD, Android | ELF | `PT_NOTE`[^1] |
+| WASI | Statically Linked | `swift5_tests` |
+| Windows | PE/COFF | `.sw5test` |
+
+[^1]: On platforms that use the ELF binary format natively, test content records
+      are stored in ELF program headers of type `PT_NOTE`. Take care not to
+      remove these program headers (for example, by invoking [`strip(1)`](https://www.man7.org/linux/man-pages/man1/strip.1.html).)
+
+### Record headers
+
+Regardless of platform, all test content records created and discoverable by the
+testing library have the following structure:
+
+```c
+struct SWTTestContentHeader {
+  int32_t n_namesz;
+  int32_t n_descsz;
+  int32_t n_type;
+  char n_name[n_namesz];
+  // ...
+};
+```
+
+This structure can be represented in Swift as a heterogenous tuple:
+
+```swift
+typealias SWTTestContentHeader = (
+  n_namesz: Int32,
+  n_descsz: Int32,
+  n_type: Int32,
+  n_name: (CChar, CChar, /* ... */),
+  // ...
+)
+```
+
+The size of `n_name` is dynamic and cannot be statically computed. The testing
+library always generates the name `"Swift Testing"` and specifies an `n_namesz`
+value of `20` (the string being null-padded to the correct length), but other
+content may be present in the same section whose header size differs. For more
+information about this structure such as its alignment requirements, see the
+documentation for the [ELF format](https://man7.org/linux/man-pages/man5/elf.5.html).
+
+Each record's _kind_ (stored in the `n_type` field) determines how the record
+will be interpreted at runtime:
+
+| Type Value | Interpretation |
+|-:|-|
+| `< 0` | Undefined (**do not use**) |
+| `0 ... 99` | Reserved |
+| `100` | Test or suite declaration |
+| `101` | Exit test |
+
+<!-- When adding cases to this enumeration, be sure to also update the
+corresponding enumeration in Discovery.h and TestContentGeneration.swift. -->
+
+### Record contents
+
+For all currently-defined record types, the header structure is immediately
+followed by the actual content of the record. A test content record currently
+contains an `accessor` function to load the corresponding Swift content and a
+`flags` field whose value depends on the type of record. The overall structure
+of a record therefore looks like:
+
+```c
+struct SWTTestContent {
+  SWTTestContentHeader header;
+  bool (* accessor)(void *outValue);
+  uint32_t flags;
+  uint32_t reserved;
+};
+```
+
+Or, in Swift as a tuple:
+
+```swift
+typealias SWTTestContent = (
+  header: SWTTestContentHeader,
+  accessor: @convention(c) (_ outValue: UnsafeMutableRawPointer) -> Bool,
+  flags: UInt32,
+  reserved: UInt32
+)
+```
+
+This structure may grow in the future as needed. Check the `header.n_descsz`
+field to determine if there are additional fields present. Do not assume that
+the size of this structure will remain fixed over time or that all discovered
+test content records are the same size.
+
+#### The accessor field
+
+The function `accessor` is a C function. When called, it initializes the memory
+at its argument `outValue` to an instance of some Swift type and returns `true`,
+or returns `false` if it could not generate the relevant content. On successful
+return, the caller is responsible for deinitializing the memory at `outValue`
+when done with it.
+
+The concrete Swift type of the value written to `outValue` depends on the type
+of record:
+
+| Type Value | Return Type |
+|-:|-|
+| `..< 0` | Undefined (**do not use**) |
+| `0 ... 99` | Reserved (**do not use**) |
+| `100` | `@Sendable () async -> Test`[^2] |
+| `101` | `ExitTest` (consumed by caller) |
+
+[^2]: This signature is not the signature of `accessor`, but of the Swift
+      function reference it writes to `outValue`. This level of indirection is
+      necessary because loading a test or suite declaration is an asynchronous
+      operation, but C functions cannot be `async`.
+
+#### The flags field
+
+For test or suite declarations (type `100`), the following flags are defined:
+
+| Bit | Description |
+|-:|-|
+| `1 << 0` | This record contains a suite declaration |
+| `1 << 1` | This record contains a parameterized test function declaration |
+
+For exit test declarations (type `101`), no flags are currently defined.
+
+#### The reserved field
+
+This field is reserved for future use. Always set it to `0`.
+
+## Third-party test content
+
+Testing tools may make use of the same storage and discovery mechanisms by
+emitting their own test content records into the test record content section.
+
+Third-party test content should use the same value for the `n_name` field
+(`"Swift Testing"`). The `n_type` field should be set to a unique value only
+used by that tool, or used by that tool in collaboration with other compatible
+tools. At runtime, Swift Testing ignores test content records with unrecognized
+`n_type` values. To reserve a new unique `n_type` value, open a [GitHub issue](https://github.com/swiftlang/swift-testing/issues/new/choose)
+against Swift Testing.
+
+The layout of third-party test content records must be compatible with that of
+`SWTTestContentHeader` as specified above. For the actual content of a test
+record, you do not need to use the same on-disk/in-memory layout as is specified
+by `SWTTestContent` above, but it is preferred. Third-party tools are ultimately
+responsible for ensuring the values they emit into the test content section are
+correctly aligned and have sufficient padding; failure to do so may render
+downstream test code unusable.
+
+<!--
+TODO: elaborate further, give examples
+TODO: standardize a mechanism for third parties to produce `Test` instances
+      since we don't have a public initializer for the `Test` type.
+-->

Documentation/Porting.md

@@ -113,13 +113,14 @@ Once the header is included, we can call `GetDateTime()` from `Clock.swift`:
 ## Runtime test discovery
 
 When porting to a new platform, you may need to provide a new implementation for
-`enumerateTypeMetadataSections()` in `Discovery.cpp`. Test discovery is
-dependent on Swift metadata discovery which is an inherently platform-specific
-operation.
+`enumerateTestContentSections()` in `Discovery.cpp`. Test discovery is dependent
+on Swift metadata discovery which is an inherently platform-specific operation.
 
-_Most_ platforms will be able to reuse the implementation used by Linux and
-Windows that calls an internal Swift runtime function to enumerate available
-metadata. If you are porting Swift Testing to Classic, this function won't be
+_Most_ platforms in use today use the ELF image format and will be able to reuse
+the implementation used by Linux. That implementation calls `dl_iterate_phdr()`
+in the GNU C Library to enumerate available metadata.
+
+If you are porting Swift Testing to Classic, `dl_iterate_phdr()` won't be
 available, so you'll need to write a custom implementation instead. Assuming
 that the Swift compiler emits section information into the resource fork on
 Classic, you could use the [Resource Manager](https://developer.apple.com/library/archive/documentation/mac/pdf/MoreMacintoshToolbox.pdf)
@@ -132,16 +133,21 @@ to load that information:
  // ...
 +#elif defined(macintosh)
 +template <typename SectionEnumerator>
-+static void enumerateTypeMetadataSections(const SectionEnumerator& body) {
++static void enumerateTestContentSections(const SectionEnumerator& body) {
 +  ResFileRefNum refNum;
 +  if (noErr == GetTopResourceFile(&refNum)) {
 +    ResFileRefNum oldRefNum = refNum;
 +    do {
 +      UseResFile(refNum);
-+      Handle handle = Get1NamedResource('swft', "\p__swift5_types");
++      Handle handle = Get1NamedResource('swft', "\p__swift5_tests");
 +      if (handle && *handle) {
-+        size_t size = GetHandleSize(handle);
-+        body(*handle, size);
++        auto imageAddress = reinterpret_cast<const void *>(static_cast<uintptr_t>(refNum));
++        SWTSectionBounds sb = { imageAddress, *handle, GetHandleSize(handle) };
++        bool stop = false;
++        body(sb, &stop);
++        if (stop) {
++          break;
++        }
 +      }
 +    } while (noErr == GetNextResourceFile(refNum, &refNum));
 +    UseResFile(oldRefNum);
@@ -150,7 +156,7 @@ to load that information:
  #else
  #warning Platform-specific implementation missing: Runtime test discovery unavailable
  template <typename SectionEnumerator>
- static void enumerateTypeMetadataSections(const SectionEnumerator& body) {}
+ static void enumerateTestContentSections(const SectionEnumerator& body) {}
  #endif
 ```
 

Package.swift

@@ -128,6 +128,8 @@ extension Array where Element == PackageDescription.SwiftSetting {
       .enableExperimentalFeature("AccessLevelOnImport"),
       .enableUpcomingFeature("InternalImportsByDefault"),
 
+      .enableExperimentalFeature("SymbolLinkageMarkers"),
+
       .define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
 
       .define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),
@@ -162,6 +164,15 @@ extension Array where Element == PackageDescription.CXXSetting {
   static var packageSettings: Self {
     var result = Self()
 
+    result += [
+      .define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
+
+      .define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),
+      .define("SWT_NO_SNAPSHOT_TYPES", .when(platforms: [.linux, .windows, .wasi])),
+      .define("SWT_NO_DYNAMIC_LINKING", .when(platforms: [.wasi])),
+      .define("SWT_NO_PIPES", .when(platforms: [.wasi])),
+    ]
+
     // Capture the testing library's version as a C++ string constant.
     if let git = Context.gitInformation {
       let testingLibraryVersion = if let tag = git.currentTag {

Sources/Testing/ExitTests/ExitTest.swift

@@ -17,22 +17,44 @@ private import _TestingInternals
 
 /// A type describing an exit test.
 ///
-/// Instances of this type describe an exit test defined by the test author and
-/// discovered or called at runtime.
-@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
+/// An instance of this type describes an exit test defined in a test target and
+/// discovered or called at runtime. You do not create instances of this type.
+///
+/// You don't usually need to interact with an instance of this type. To create
+/// an exit test, use the ``expect(exitsWith:_:sourceLocation:performing:)``
+/// or ``require(exitsWith:_:sourceLocation:performing:)`` macro.
+@_spi(Experimental)
 public struct ExitTest: Sendable, ~Copyable {
-  /// The expected exit condition of the exit test.
+  /// This exit test's expected exit condition.
   public var expectedExitCondition: ExitCondition
 
-  /// The body closure of the exit test.
-  fileprivate var body: @Sendable () async throws -> Void = {}
-
-  /// The source location of the exit test.
+  /// The source location of this exit test.
   ///
   /// The source location is unique to each exit test and is consistent between
   /// processes, so it can be used to uniquely identify an exit test at runtime.
   public var sourceLocation: SourceLocation
 
+  /// The body closure of the exit test.
+  ///
+  /// Do not invoke this closure directly. Instead, invoke ``callAsFunction()``
+  /// to run the exit test. Running the exit test will always terminate the
+  /// current process.
+  fileprivate var body: @Sendable () async throws -> Void
+
+  /// Initialize an exit test at runtime.
+  ///
+  /// - Warning: This initializer is used to implement the `#expect(exitsWith:)`
+  ///   macro. Do not use it directly.
+  public init(
+    __expectedExitCondition expectedExitCondition: ExitCondition,
+    sourceLocation: SourceLocation,
+    body: @escaping @Sendable () async throws -> Void = {}
+  ) {
+    self.expectedExitCondition = expectedExitCondition
+    self.sourceLocation = sourceLocation
+    self.body = body
+  }
+
   /// Disable crash reporting, crash logging, or core dumps for the current
   /// process.
   private static func _disableCrashReporting() {
@@ -83,6 +105,7 @@ public struct ExitTest: Sendable, ~Copyable {
   /// to terminate the process; if it does not, the testing library will
   /// terminate the process in a way that causes the corresponding expectation
   /// to fail.
+  @_spi(ForToolsIntegrationOnly)
   public consuming func callAsFunction() async -> Never {
     Self._disableCrashReporting()
 
@@ -102,44 +125,24 @@ public struct ExitTest: Sendable, ~Copyable {
 
 // MARK: - Discovery
 
-/// A protocol describing a type that contains an exit test.
-///
-/// - Warning: This protocol is used to implement the `#expect(exitsWith:)`
-///   macro. Do not use it directly.
-@_alwaysEmitConformanceMetadata
-@_spi(Experimental)
-public protocol __ExitTestContainer {
-  /// The expected exit condition of the exit test.
-  static var __expectedExitCondition: ExitCondition { get }
-
-  /// The source location of the exit test.
-  static var __sourceLocation: SourceLocation { get }
-
-  /// The body function of the exit test.
-  static var __body: @Sendable () async throws -> Void { get }
-}
-
 extension ExitTest {
-  /// A string that appears within all auto-generated types conforming to the
-  /// `__ExitTestContainer` protocol.
-  private static let _exitTestContainerTypeNameMagic = "__🟠$exit_test_body__"
-
   /// Find the exit test function at the given source location.
   ///
   /// - Parameters:
   ///   - sourceLocation: The source location of the exit test to find.
   ///
   /// - Returns: The specified exit test function, or `nil` if no such exit test
   ///   could be found.
+  @_spi(ForToolsIntegrationOnly)
   public static func find(at sourceLocation: SourceLocation) -> Self? {
     var result: Self?
 
-    enumerateTypes(withNamesContaining: _exitTestContainerTypeNameMagic) { _, type, stop in
-      if let type = type as? any __ExitTestContainer.Type, type.__sourceLocation == sourceLocation {
+    enumerateTestContent(ofKind: .exitTest, as: ExitTest.self) { _, exitTest, _, stop in
+      if exitTest.sourceLocation == sourceLocation {
         result = ExitTest(
-          expectedExitCondition: type.__expectedExitCondition,
-          body: type.__body,
-          sourceLocation: type.__sourceLocation
+          __expectedExitCondition: exitTest.expectedExitCondition,
+          sourceLocation: exitTest.sourceLocation,
+          body: exitTest.body
         )
         stop = true
       }
@@ -183,7 +186,7 @@ func callExitTest(
 
   let actualExitCondition: ExitCondition
   do {
-    let exitTest = ExitTest(expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
+    let exitTest = ExitTest(__expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
     actualExitCondition = try await configuration.exitTestHandler(exitTest)
   } catch {
     // An error here would indicate a problem in the exit test handler such as a
@@ -295,7 +298,7 @@ extension ExitTest {
     // External tools authors should set up their own back channel mechanisms
     // and ensure they're installed before calling ExitTest.callAsFunction().
     guard var result = find(at: sourceLocation) else {
-      return nil
+      fatalError("Could not find an exit test that should have been located at \(sourceLocation).")
     }
 
     // We can't say guard let here because it counts as a consume.