Local E2E Tests #751
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Local E2E Tests | |
| concurrency: | |
| group: e2e-local-tests-${{ github.ref }} | |
| cancel-in-progress: true | |
| on: | |
| schedule: | |
| - cron: '00 1 * * *' | |
| workflow_dispatch: | |
| inputs: | |
| jobs: | |
| description: >- | |
| Comma-separated job names to run, no spaces (e.g. "Windows_Chrome_Local" or | |
| "Windows_Chrome_Local,MacOSX_Safari_Local"). Defaults to every job -- today's | |
| behaviour -- and is always every job on the scheduled nightly run. | |
| required: false | |
| default: 'all' | |
| tests: | |
| description: >- | |
| Maven -Dtest= selector, threaded into every selected job's Run | |
| tests step. Leave empty to keep each selected job's own default | |
| selector unchanged. | |
| required: false | |
| default: '' | |
| permissions: | |
| contents: read | |
| env: | |
| # Environment-independent unit tests belong to pr-gate.yml's unit-tests job, not to each | |
| # local browser/OS job. Keep this trailer aligned with e2eTests.yml; grid-only Playwright | |
| # and LazyLoadingFixtureLiveTest exclusions deliberately remain absent here because those | |
| # tests require local browser coverage. | |
| GLOBAL_TESTING_SCOPE: "!%regex[.*DatabaseActions.*], !%regex[.*CLI.*], !%regex[.*DB.*], !%regex[.*Db.*], !%regex[.*db.*], !%regex[.*API.*], !%regex[.*Api.*], !%regex[.*api.*], !%regex[.*uestBuilder.*], !%regex[.*Rest.*], !%regex[.*Json.*], !%regex[.*JSON.*], !%regex[.*json.*], !%regex[.*ndroid.*], !%regex[.*IOS.*], !%regex[.*MobileWeb.*], !%regex[.*appium.Mobile.*], !%regex[.*properties.Mobile.*], !%regex[.*CucumberTests.*], !%regex[.*LT.*], !%regex[.*Flutter.*], !%regex[.*MoonTests.*], !%regex[.*testPackage.unitTests.*], !%regex[.*DriverFactoryHelperAugmentationUnitTest.*], !%regex[.*DriverFactoryHelperCoverageUnitTest.*], !%regex[.*DriverFactoryHelperStorageStateUnitTest.*], !%regex[.*OptionsManagerCoverageUnitTest.*], !%regex[.*RemoteGridPreflightUnitTest.*], !%regex[.*ActionsCoverageUnitTest.*], !%regex[.*ActionsExceptionDetailsUnitTest.*], !%regex[.*ElementInformationCoverageUnitTest.*], !%regex[.*ElementsHelperCoverageUnitTest.*], !%regex[.*AriaSnapshotHelperUnitTest.*], !%regex[.*MobileAccessibilityTreeConverterUnitTest.*], !%regex[.*AnimatedGifManagerCoverageUnitTest.*], !%regex[.*ImageProcessingActionsReportingUnitTest.*], !%regex[.*ImageProcessingActionsScreenshotBaselineUnitTest.*], !%regex[.*ScreenshotHelperCoverageUnitTest.*], !%regex[.*ScreenshotManagerCoverageUnitTest.*], !%regex[.*LocatorHealthReporterUnitTest.*], !%regex[.*RecordManagerCoverageUnitTest.*], !%regex[.*PlaywrightSessionFactoryStorageStateUnitTest.*], !%regex[.*PlaywrightNaturalActionExecutorTest.*], !%regex[.*ShardPartitionerUnitTest.*], !%regex[.*UpdateCheckerUnitTest.*], !%regex[.*LightHouseGenerateReportCoverageUnitTest.*], !%regex[.*PropertiesHelperDownloadRetryUnitTest.*], !%regex[.*PropertyFileManagerInternalUnitTest.*], !%regex[.*AllureManagerCoverageUnitTest.*], !%regex[.*BrowserPerformanceExecutionReportUnitTest.*], !%regex[.*FlakeProfilerUnitTest.*], !%regex[.*ProgressBarLoggerCoverageUnitTest.*], !%regex[.*ValidationsHelperCoverageUnitTest.*], !%regex[.*ValidationsHelperNewPatternCoverageUnitTest.*], !%regex[.*VisualValidationsBuilderUnitTest.*], !%regex[.*WebDriverElementValidationsBuilderAriaSnapshotUnitTest.*], !%regex[.*WebDriverElementValidationsBuilderCoverageUnitTest.*], !%regex[.*AssertionEvidenceReporterTest.*], !%regex[.*FailureTraceReporterTest.*]" | |
| FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true | |
| jobs: | |
| Workflow_Summary: | |
| if: always() | |
| needs: | |
| - Windows_Edge_Local | |
| - MacOSX_Safari_Local | |
| - Windows_Chrome_Local | |
| - Windows_Managed_Lighthouse | |
| - Ubuntu_Managed_Android | |
| - Windows_SikuliX_Local | |
| - Windows_Appium_Desktop_Local | |
| - MacOSX_Chrome_Local | |
| - Windows_Edge_Cucumber_Local | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| permissions: | |
| contents: read | |
| actions: read | |
| steps: | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Consolidate Test Results | |
| uses: ./.github/actions/consolidate-test-results | |
| with: | |
| needs-json: ${{ toJSON(needs) }} | |
| Windows_Edge_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Windows_Edge_Local,') | |
| runs-on: windows-latest | |
| timeout-minutes: 90 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| # .memory relation filenames exceed MAX_PATH with the runner workspace prefix; | |
| # without long paths, git checkout itself fails on Windows runners. | |
| - name: Enable Windows long paths | |
| run: git config --global core.longpaths true | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '20' | |
| # See the note in e2eTests.yml: a Central refusal in this step, which has | |
| # no continue-on-error, cascades into a missing Allure summary (#4445). | |
| # `shell: bash` is explicit because these jobs run on Windows and macOS. | |
| - name: Install optional visual test runtime | |
| shell: bash | |
| run: >- | |
| bash scripts/ci/build_retry.sh 2 60 | |
| mvn -pl shaft-visual -am -e install "-DskipTests" "-Dgpg.skip" | |
| - name: Print Edge version | |
| continue-on-error: true | |
| run: (Get-Item (Get-Command msedge).Source).VersionInfo.ProductVersion | |
| - name: Run tests | |
| continue-on-error: true | |
| run: mvn -pl shaft-engine -am -e test "-DincludeVisualTestRuntime" "-DdefaultElementIdentificationTimeout=5" "-DsetParallelMode=NONE" "-DretryMaximumNumberOfAttempts=1" "-DexecutionAddress=local" "-DtargetOperatingSystem=WINDOWS" "-DtargetBrowserName=MicrosoftEdge" "-DheadlessExecution=true" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || env.GLOBAL_TESTING_SCOPE }}" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Windows_Edge_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| MacOSX_Safari_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',MacOSX_Safari_Local,') | |
| runs-on: macos-latest | |
| timeout-minutes: 120 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '20' | |
| # See the note in e2eTests.yml: a Central refusal in this step, which has | |
| # no continue-on-error, cascades into a missing Allure summary (#4445). | |
| # `shell: bash` is explicit because these jobs run on Windows and macOS. | |
| - name: Install optional visual test runtime | |
| shell: bash | |
| run: >- | |
| bash scripts/ci/build_retry.sh 2 60 | |
| mvn -pl shaft-visual -am -e install "-DskipTests" "-Dgpg.skip" | |
| - name: Run tests | |
| continue-on-error: true | |
| run: mvn -pl shaft-engine -am -e test "-DincludeVisualTestRuntime" "-DdefaultElementIdentificationTimeout=5" "-DretryMaximumNumberOfAttempts=1" "-DexecutionAddress=local" "-DtargetOperatingSystem=MAC" "-DtargetBrowserName=safari" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || format('{0}, !%regex[.*MobileEmulation.*]', env.GLOBAL_TESTING_SCOPE) }}" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: MacOSX_Safari_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| Windows_Chrome_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Windows_Chrome_Local,') | |
| runs-on: windows-latest | |
| timeout-minutes: 75 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| # .memory relation filenames exceed MAX_PATH with the runner workspace prefix; | |
| # without long paths, git checkout itself fails on Windows runners. | |
| - name: Enable Windows long paths | |
| run: git config --global core.longpaths true | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '20' | |
| # See the note in e2eTests.yml: a Central refusal in this step, which has | |
| # no continue-on-error, cascades into a missing Allure summary (#4445). | |
| # `shell: bash` is explicit because these jobs run on Windows and macOS. | |
| - name: Install optional visual test runtime | |
| shell: bash | |
| run: >- | |
| bash scripts/ci/build_retry.sh 2 60 | |
| mvn -pl shaft-visual -am -e install "-DskipTests" "-Dgpg.skip" | |
| - name: Run tests | |
| continue-on-error: true | |
| run: mvn -pl shaft-engine -am -e test "-DincludeVisualTestRuntime" "-DdefaultElementIdentificationTimeout=5" "-DsetParallelMode=NONE" "-DretryMaximumNumberOfAttempts=1" "-DexecutionAddress=local" "-DtargetOperatingSystem=WINDOWS" "-DtargetBrowserName=chrome" "-DheadlessExecution=true" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || env.GLOBAL_TESTING_SCOPE }}" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Windows_Chrome_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| # Issue #4884: opt-in real acceptance for the release-pinned managed Node/Lighthouse | |
| # toolchain. This stays manual because it downloads a full npm dependency tree and | |
| # audits a live local Chrome session; PR/unit gates cover the deterministic seams. | |
| Windows_Managed_Lighthouse: | |
| if: github.event_name == 'workflow_dispatch' && (github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Windows_Managed_Lighthouse,')) | |
| runs-on: windows-latest | |
| timeout-minutes: 30 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| - name: Enable Windows long paths | |
| run: git config --global core.longpaths true | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '24' | |
| - name: Package the setup CLI | |
| run: mvn --batch-mode -pl shaft-cli -am package "-DskipTests" "-Dallure.automaticallyOpen=false" "-Dgpg.skip=true" | |
| - name: Plan, approve, install, and verify managed Lighthouse | |
| shell: pwsh | |
| run: | | |
| $cli = Get-ChildItem -Path shaft-cli/target -Filter 'shaft-cli-*.jar' | | |
| Where-Object { $_.Name -match '^shaft-cli-[0-9][0-9.]*\.jar$' } | | |
| Select-Object -First 1 -ExpandProperty FullName | |
| if (-not $cli) { throw 'Packaged shaft-cli executable JAR was not found.' } | |
| $cacheRoot = Join-Path $env:RUNNER_TEMP 'shaft-lighthouse-cache' | |
| $dataRoot = Join-Path $cacheRoot 'data' | |
| $plan = Join-Path $env:RUNNER_TEMP 'lighthouse-plan.json' | |
| $planOutput = & java -jar $cli setup plan --profile LIGHTHOUSE --mode MANAGED --output $plan --cache-root $cacheRoot --data-root $dataRoot | |
| if ($LASTEXITCODE -ne 0) { throw "Lighthouse planning failed: $planOutput" } | |
| $digest = (($planOutput | Select-String '^Digest: ').Line -replace '^Digest:\s*', '') | |
| if (-not $digest) { throw 'Lighthouse plan did not emit an approval digest.' } | |
| & java -jar $cli setup install --plan $plan --approve $digest --cache-root $cacheRoot --data-root $dataRoot | |
| if ($LASTEXITCODE -ne 0) { throw 'Managed Lighthouse installation failed.' } | |
| & java -jar $cli setup verify --profile LIGHTHOUSE --cache-root $cacheRoot --data-root $dataRoot | |
| if ($LASTEXITCODE -ne 0) { throw 'Managed Lighthouse verification failed.' } | |
| - name: Run real local-Chrome Lighthouse report | |
| continue-on-error: true | |
| run: >- | |
| mvn --batch-mode -pl shaft-engine -am test | |
| "-Dtest=testPackage.ManagedLighthouseE2ETest" | |
| "-DrunManagedLighthouseE2E=true" | |
| "-Dinfrastructure.mode=MANAGED" | |
| "-Dinfrastructure.profile=LIGHTHOUSE" | |
| "-Dinfrastructure.cacheDirectory=${{ runner.temp }}\shaft-lighthouse-cache" | |
| "-DlightHouseExecution=true" | |
| "-DopenLighthouseReportWhileExecution=false" | |
| "-DexecutionAddress=local" | |
| "-DtargetBrowserName=chrome" | |
| "-DheadlessExecution=true" | |
| "-Dallure.automaticallyOpen=false" | |
| "-Dgpg.skip=true" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Windows_Managed_Lighthouse | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| Windows_SikuliX_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Windows_SikuliX_Local,') | |
| runs-on: windows-latest | |
| timeout-minutes: 10 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| # .memory relation filenames exceed MAX_PATH with the runner workspace prefix; | |
| # without long paths, git checkout itself fails on Windows runners. | |
| - name: Enable Windows long paths | |
| run: git config --global core.longpaths true | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '20' | |
| - name: Run SikuliX Windows desktop flow | |
| continue-on-error: true | |
| run: mvn -pl shaft-sikulix -am -e test "-DrunWindowsDesktopE2E=true" "-DdefaultElementIdentificationTimeout=5" "-DretryMaximumNumberOfAttempts=1" "-DautomaticallyOpen=false" "-DopenLighthouseReportWhileExecution=false" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || 'SikuliWindowsDesktopE2ETest' }}" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Windows_SikuliX_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| module-directory: shaft-sikulix | |
| Windows_Appium_Desktop_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Windows_Appium_Desktop_Local,') | |
| runs-on: windows-latest | |
| timeout-minutes: 15 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| # .memory relation filenames exceed MAX_PATH with the runner workspace prefix; | |
| # without long paths, git checkout itself fails on Windows runners. | |
| - name: Enable Windows long paths | |
| run: git config --global core.longpaths true | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '24' | |
| - name: Install Appium Windows driver | |
| run: | | |
| npm install -g appium@3.5.2 | |
| appium driver install windows | |
| appium driver run windows install-wad | |
| - name: Verify WinAppDriver installation | |
| shell: powershell | |
| run: | | |
| # install-wad launches the WinAppDriver MSI via an unawaited elevated | |
| # Start-Process and returns immediately, so the binary may not exist yet. | |
| $winAppDriverPath = Join-Path ${env:ProgramFiles(x86)} 'Windows Application Driver\WinAppDriver.exe' | |
| $installed = $false | |
| for ($i = 0; $i -lt 60; $i++) { | |
| if (Test-Path $winAppDriverPath) { | |
| $installed = $true | |
| break | |
| } | |
| Start-Sleep -Seconds 1 | |
| } | |
| if (-not $installed) { | |
| throw "WinAppDriver did not finish installing at $winAppDriverPath after 60s." | |
| } | |
| - name: Start Appium server and run Windows desktop flow | |
| continue-on-error: true | |
| shell: powershell | |
| run: | | |
| # The Appium server and the test MUST run in the same step. When the | |
| # server was started via Start-Job in a *separate* step, that job's | |
| # child process was torn down as soon as the start step's pwsh exited, | |
| # so nothing was listening on 4723 by the time the test step ran: every | |
| # POST /session failed instantly with java.nio.channels.ClosedChannelException | |
| # and SHAFT retried until the 300s remote-session timeout. The server | |
| # --log confirmed it -- it stopped right after the in-step /status probe | |
| # with no POST /session ever logged server-side (run 29717278042, #3806). | |
| # (That same cross-step death, not stream buffering, is why an earlier | |
| # Receive-Job at cleanup surfaced zero server output.) Co-locating start | |
| # + test keeps the server alive for the whole session-creation window; | |
| # the --log file still feeds the diagnostics artifact. | |
| $appiumLog = Join-Path $env:RUNNER_TEMP 'appium-server.log' | |
| Start-Job -Name shaft-appium -ScriptBlock { | |
| param($logPath) | |
| appium --address 127.0.0.1 --port 4723 --log-level info --log $logPath | |
| } -ArgumentList $appiumLog | Out-Null | |
| $ready = $false | |
| for ($i = 0; $i -lt 60; $i++) { | |
| try { | |
| Invoke-RestMethod -Uri "http://127.0.0.1:4723/status" -TimeoutSec 2 | Out-Null | |
| $ready = $true | |
| break | |
| } catch { | |
| Start-Sleep -Seconds 1 | |
| } | |
| } | |
| if (-not $ready) { | |
| Receive-Job -Name shaft-appium -Keep | |
| if (Test-Path $appiumLog) { | |
| Get-Content $appiumLog | |
| } | |
| throw "Appium server did not become ready." | |
| } | |
| mvn -pl shaft-engine -am -e test "-DrunWindowsDesktopE2E=true" "-DdefaultElementIdentificationTimeout=5" "-DretryMaximumNumberOfAttempts=1" "-DexecutionAddress=http://127.0.0.1:4723" "-DtargetOperatingSystem=WINDOWS" "-DtargetBrowserName=WindowsApp" "-DheadlessExecution=false" "-DplatformName=Windows" "-Dmobile_automationName=Windows" "-Dmobile_app=C:\Windows\System32\notepad.exe" "-DautomaticallyOpen=false" "-DopenLighthouseReportWhileExecution=false" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || 'WindowsDesktopAppiumE2ETest' }}" | |
| - name: Dump Appium server log | |
| if: always() | |
| shell: powershell | |
| run: | | |
| # The server ran inside the previous step's shell, so its Start-Job is | |
| # already gone here -- only the on-disk --log file survives to inspect. | |
| $appiumLog = Join-Path $env:RUNNER_TEMP 'appium-server.log' | |
| Write-Output "=== Appium server log ($appiumLog) ===" | |
| if (Test-Path $appiumLog) { | |
| Get-Content $appiumLog | |
| } else { | |
| Write-Output "(log file was not created)" | |
| } | |
| - name: Upload Appium server log | |
| if: always() | |
| uses: actions/upload-artifact@v7 | |
| with: | |
| name: Windows_Appium_Desktop_Local_server_log | |
| path: ${{ runner.temp }}\appium-server.log | |
| if-no-files-found: ignore | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Windows_Appium_Desktop_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| MacOSX_Chrome_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',MacOSX_Chrome_Local,') | |
| runs-on: macos-latest | |
| timeout-minutes: 90 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '20' | |
| # See the note in e2eTests.yml: a Central refusal in this step, which has | |
| # no continue-on-error, cascades into a missing Allure summary (#4445). | |
| # `shell: bash` is explicit because these jobs run on Windows and macOS. | |
| - name: Install optional visual test runtime | |
| shell: bash | |
| run: >- | |
| bash scripts/ci/build_retry.sh 2 60 | |
| mvn -pl shaft-visual -am -e install "-DskipTests" "-Dgpg.skip" | |
| - name: Run tests | |
| continue-on-error: true | |
| run: mvn -pl shaft-engine -am -e test "-DincludeVisualTestRuntime" "-DdefaultElementIdentificationTimeout=5" "-DsetParallelMode=NONE" "-DretryMaximumNumberOfAttempts=1" "-DexecutionAddress=local" "-DtargetOperatingSystem=MAC" "-DtargetBrowserName=chrome" "-DheadlessExecution=true" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || env.GLOBAL_TESTING_SCOPE }}" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: MacOSX_Chrome_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| Windows_Edge_Cucumber_Local: | |
| if: github.event_name != 'workflow_dispatch' || github.event.inputs.jobs == '' || github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Windows_Edge_Cucumber_Local,') | |
| runs-on: windows-latest | |
| timeout-minutes: 10 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| # .memory relation filenames exceed MAX_PATH with the runner workspace prefix; | |
| # without long paths, git checkout itself fails on Windows runners. | |
| - name: Enable Windows long paths | |
| run: git config --global core.longpaths true | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: 'lts/*' | |
| - name: Print Edge version | |
| continue-on-error: true | |
| run: (Get-Item (Get-Command msedge).Source).VersionInfo.ProductVersion | |
| - name: Run tests | |
| continue-on-error: true | |
| run: mvn -pl shaft-engine -am -e test "-DdefaultElementIdentificationTimeout=5" "-DsetParallelMode=NONE" "-DretryMaximumNumberOfAttempts=1" "-DexecutionAddress=local" "-DtargetOperatingSystem=WINDOWS" "-DmaximumPerformanceMode=2" "-DtargetBrowserName=MicrosoftEdge" "-DgenerateAllureReportArchive=true" "-Dtest=${{ github.event.inputs.tests != '' && github.event.inputs.tests || '%regex[.*CucumberTests.*]' }}" | |
| - name: Set up Python | |
| uses: actions/setup-python@v7 | |
| with: | |
| python-version: '3.13' | |
| # The %regex[.*CucumberTests.*] selector also matches unrelated unit tests (e.g. | |
| # CucumberTestsHelperCoverageUnitTest, testPackage.properties.CucumberTests). Their | |
| # passing runs previously kept the aggregate Allure total above zero even during | |
| # #4079's original defect, when the real Cucumber runner found zero scenarios and | |
| # every job on this selector reported false-green. #4078 fixed the underlying | |
| # auto-wiring (verified: cucumberTestRunner.CucumberTests now finds and runs all 4 | |
| # scenarios both locally and in CI), but this check remains as the regression guard -- | |
| # a future break in that auto-wiring must fail loudly and can never hide behind the | |
| # unrelated classes' passing counts. Checked via the runner's own Surefire report so | |
| # this can't be fooled by that padding the way the aggregate Allure total can. | |
| # | |
| # Path note: when more than one class matches -Dtest (as here), Surefire's TestNG | |
| # provider writes each class's JUnit-style report under surefire-reports/junitreports/, | |
| # not a top-level surefire-reports/TEST-<Class>.xml (that top-level path is only | |
| # produced when a single class is selected directly, e.g. -Dtest=<FQCN> with no other | |
| # matches -- confirmed empirically both ways). | |
| - name: Verify Cucumber scenarios actually executed | |
| run: >- | |
| python3 scripts/ci/assert_tests_executed.py | |
| shaft-engine/target/surefire-reports/junitreports/TEST-cucumberTestRunner.CucumberTests.xml | |
| --min-executed 1 | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Windows_Edge_Cucumber_Local | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| # Opt-in real acceptance for #4898. The packaged CLI performs the only installation | |
| # mutation; a packaged CLI process starts the receipt-bound AVD/Appium lease, the engine | |
| # reuses it for a real UiAutomator2 session, and a later CLI process owns final shutdown. | |
| Ubuntu_Managed_Android: | |
| if: github.event_name == 'workflow_dispatch' && (github.event.inputs.jobs == 'all' || contains(format(',{0},', github.event.inputs.jobs), ',Ubuntu_Managed_Android,')) | |
| runs-on: ubuntu-22.04 | |
| timeout-minutes: 60 | |
| outputs: | |
| job: ${{ steps.post_test_report.outputs.job }} | |
| total: ${{ steps.post_test_report.outputs.total }} | |
| passed: ${{ steps.post_test_report.outputs.passed }} | |
| failed: ${{ steps.post_test_report.outputs.failed }} | |
| broken: ${{ steps.post_test_report.outputs.broken }} | |
| skipped: ${{ steps.post_test_report.outputs.skipped }} | |
| duration: ${{ steps.post_test_report.outputs.duration }} | |
| steps: | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: Setup Test Environment | |
| uses: ./.github/actions/setup-test-env | |
| with: | |
| node-version: '24' | |
| - name: Provision the Android emulator host runtime | |
| run: | | |
| if [ -e /dev/kvm ]; then sudo chmod 666 /dev/kvm; fi | |
| sudo apt-get update | |
| sudo apt-get install --yes libpulse0 | |
| - name: Package the setup CLI | |
| run: mvn --batch-mode -pl shaft-cli -am package -DskipTests '-Dallure.automaticallyOpen=false' -Dgpg.skip | |
| - name: Install the reviewed managed Android plan | |
| shell: bash | |
| run: | | |
| cli_jar="$(find shaft-cli/target -maxdepth 1 -name 'shaft-cli-*[0-9].jar' -print -quit)" | |
| test -n "$cli_jar" | |
| cache_root="$RUNNER_TEMP/shaft-android-cache" | |
| data_root="$cache_root/data" | |
| plan="$RUNNER_TEMP/shaft-android-plan.json" | |
| java -jar "$cli_jar" setup plan --profile MOBILE_ANDROID --mode MANAGED \ | |
| --output "$plan" --cache-root "$cache_root" --data-root "$data_root" \ | |
| --startup-timeout PT8M --shutdown-timeout PT1M | |
| digest="$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['digest'])" "$plan")" | |
| if ! java -jar "$cli_jar" setup install --plan "$plan" --approve "$digest" \ | |
| --accept-license android-sdk-license --cache-root "$cache_root" --data-root "$data_root" \ | |
| --startup-timeout PT8M --shutdown-timeout PT1M; then | |
| install_log="$data_root/state/logs/mobile-android-install.log" | |
| test ! -f "$install_log" || cat "$install_log" | |
| exit 1 | |
| fi | |
| - name: Start the owned runtime from the packaged CLI | |
| shell: bash | |
| run: | | |
| cli_jar="$(find shaft-cli/target -maxdepth 1 -name 'shaft-cli-*[0-9].jar' -print -quit)" | |
| cache_root="$RUNNER_TEMP/shaft-android-cache" | |
| data_root="$cache_root/data" | |
| plan="$RUNNER_TEMP/shaft-android-plan.json" | |
| digest="$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['digest'])" "$plan")" | |
| java -jar "$cli_jar" setup start --plan "$plan" --approve "$digest" \ | |
| --accept-license android-sdk-license --cache-root "$cache_root" --data-root "$data_root" \ | |
| --startup-timeout PT8M --shutdown-timeout PT1M | |
| test -s "$data_root/state/mobile-android-runtime.json" | |
| - name: Run real emulator, UiAutomator2, and aapt2 acceptance | |
| continue-on-error: true | |
| run: >- | |
| mvn --batch-mode -pl shaft-engine -am test | |
| '-Dtest=testPackage.ManagedAndroidE2ETest' | |
| -DrunManagedAndroidE2E=true | |
| -Dinfrastructure.mode=MANAGED | |
| -Dinfrastructure.profile=MOBILE_ANDROID | |
| -Dinfrastructure.cacheDirectory=${{ runner.temp }}/shaft-android-cache | |
| -Dinfrastructure.startupTimeout=PT8M | |
| -Dinfrastructure.shutdownTimeout=PT1M | |
| -Dallure.automaticallyOpen=false | |
| -Dgpg.skip=true | |
| - name: Stop the owned runtime from a later packaged CLI process | |
| if: always() | |
| shell: bash | |
| run: | | |
| cli_jar="$(find shaft-cli/target -maxdepth 1 -name 'shaft-cli-*[0-9].jar' -print -quit)" | |
| cache_root="$RUNNER_TEMP/shaft-android-cache" | |
| data_root="$cache_root/data" | |
| test -s "$data_root/state/mobile-android-runtime.json" | |
| java -jar "$cli_jar" setup stop --profile MOBILE_ANDROID \ | |
| --cache-root "$cache_root" --data-root "$data_root" --shutdown-timeout PT1M | |
| test ! -e "$data_root/state/mobile-android-runtime.json" | |
| - name: Post-Test Report and Check | |
| id: post_test_report | |
| if: always() | |
| uses: ./.github/actions/post-test-report | |
| with: | |
| job-name: Ubuntu_Managed_Android | |
| codecov-token: ${{ secrets.CODECOV_TOKEN }} | |
| notify_local_e2e_tests_failure: | |
| name: Notify on nightly failure | |
| needs: [ Windows_Edge_Local, MacOSX_Safari_Local, Windows_Chrome_Local, Windows_Managed_Lighthouse, Ubuntu_Managed_Android, Windows_SikuliX_Local, Windows_Appium_Desktop_Local, MacOSX_Chrome_Local, Windows_Edge_Cucumber_Local ] | |
| if: always() | |
| runs-on: ubuntu-22.04 | |
| timeout-minutes: 5 | |
| permissions: | |
| contents: read | |
| issues: write | |
| steps: | |
| - name: Checkout Code | |
| uses: actions/checkout@v7 | |
| - name: File or recover the nightly tracking issue | |
| uses: ./.github/actions/notify-nightly-failure | |
| with: | |
| # A targeted dispatch (github.event.inputs.jobs set to something other than | |
| # '' / 'all') only runs a subset of jobs on purpose (#4115) -- most needs.*.result | |
| # entries are 'skipped', not real passes, so this must never be read as a | |
| # full-suite recovery/failure signal. Only a schedule run or a full ('all'/empty) | |
| # dispatch feeds the real outcome into the tracking-issue logic below. | |
| outcome: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.jobs != '' && github.event.inputs.jobs != 'all') && 'skip' || (contains(needs.*.result, 'failure') && 'failure') || (contains(needs.*.result, 'cancelled') && 'skip') || 'success' }} | |
| workflow-name: ${{ github.workflow }} | |
| label: 'nightly-failure:local-e2e-tests' |