[Vision OS] Unit test cannot be attached to vision os simulator correctly on CI (macos-15 with Xcode 16.4)

[kaisong1990]

Description

Our framework runs unit tests across iOS, macOS, and visionOS. The CI setup is solid for iOS and macOS, but visionOS is a bit tricky—builds usually go through fine, but the pipeline often fails to attach the simulator during testing. Interestingly, I’ve had very few successful runs using the exact same YAML and script, though it’s more like 1 out of 10. I have downloaded the vision os sdk to CI befroe running the script FYI

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• macOS 26 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

Runner Image
Image: macos-15
Version: 20250917.1898

Those successful ones are also having the same image version

Is it regression?

Likely, as the vision os simulator is working fine on macos-14 and SDK 1.2

Expected behavior

vision os should attach testHostApp to vision os simulator on Azure pipeline

Actual behavior

vision os failed attach testHostApp to vision os simulator on Azure pipeline. Example:

Total running time: 7164.77 seconds
FINAL STATUS = 1
POSSIBLE ERRORS: 2025-10-02 00:50:51.557 xcodebuild[15261:406453] IDETestOperationsObserverDebug: Failure collecting diagnostics from simulator: Timed out after 600.0 seconds while waiting for a response from the invoked process
2025-10-02 00:50:53.775 xcodebuild[15261:276449] [MT] IDETestOperationsObserverDebug: 2096.031 elapsed -- Testing started completed.
2025-10-02 00:50:53.777 xcodebuild[15261:276449] [MT] IDETestOperationsObserverDebug: 0.008 sec, +0.008 sec -- start
2025-10-02 00:50:53.777 xcodebuild[15261:276449] [MT] IDETestOperationsObserverDebug: 2096.031 sec, +2096.023 sec -- end
Testing failed:
UnitTestsHostApp (64221) encountered an error (Test runner never began executing tests after launching)

** TEST FAILED **
Build & Testing Failed! \n 2025-10-02 00:50:51.557 xcodebuild[15261:406453] IDETestOperationsObserverDebug: Failure collecting diagnostics from simulator: Timed out after 600.0 seconds while waiting for a response from the invoked process
2025-10-02 00:50:53.775 xcodebuild[15261:276449] [MT] IDETestOperationsObserverDebug: 2096.031 elapsed -- Testing started completed.
2025-10-02 00:50:53.777 xcodebuild[15261:276449] [MT] IDETestOperationsObserverDebug: 0.008 sec, +0.008 sec -- start
2025-10-02 00:50:53.777 xcodebuild[15261:276449] [MT] IDETestOperationsObserverDebug: 2096.031 sec, +2096.023 sec -- end
Testing failed:
UnitTestsHostApp (64221) encountered an error (Test runner never began executing tests after launching)

Repro steps

1. Run on vmimage as macos-15, Xcode 16.4 and manually download the vision os sdk
2. Run a unit test.

[erik-bershel]

Hey @kaisong1990! 👋

This is most likely part of the general performance degradation we're investigating here and here, and the simulator simply isn't starting up fast enough, as strange as that may sound. But perhaps you've stumbled upon another change related to additional security requirements for macOS 15. That would be easier to fix. However, to do this, it would be great to either access your code or get a full system log from the moment Xcode launched. Otherwise, we won't be able to exit RCA with the information provided. There could be many reasons why the simulator didn't start.
Just in case, I should clarify which image we're talking about here. Are you using the macos-latest-internal YAML tag or something like that?