Skip to content

E2e reliability and performance#2201

Open
samuelreichert wants to merge 34 commits into
mainfrom
e2e-reliability-and-performance
Open

E2e reliability and performance#2201
samuelreichert wants to merge 34 commits into
mainfrom
e2e-reliability-and-performance

Conversation

@samuelreichert
Copy link
Copy Markdown
Contributor

@samuelreichert samuelreichert commented May 5, 2026 •
edited
Loading

Pull request type

No code changes (changes to documentation, CI, metadata, etc.)
Test related change (New E2E test, test automation, etc.)

Description

This PR improves E2E test reliability and reduces nightly run duration through a set of infrastructure, stability, and parallelization improvements.

Shared fixtures & Mendix helpers (run-e2e)

  • Added worker-scoped session fixture — each Playwright worker holds one Mendix session, eliminating per-test login/logout overhead and staying within the 5-session developer license limit
  • Added waitForMendixApp(page) helper to replace waitForLoadState("networkidle"), which was unreliable and caused flaky tests
  • Added checkAccessibility shared helper
  • Added smoke suite support via E2E_SUITE=smoke env var and @smoke tag

Flakiness fixes

  • Replaced all page.waitForTimeout() fixed delays with event-based waits
  • Removed waitForLoadState("networkidle") across all specs
  • Fixed race conditions in datagrid filter specs
  • Fixed racy nth(1) assertion in badge-button close page test
  • Migrated remaining specs that the codemod missed

Screenshot baseline hardening

  • Set global threshold: 0.1 and animations: "disabled" in playwright config
  • Removed per-test loose threshold/maxDiffPixels overrides (0.2–0.5, up to 15000px) that were masking real pixel differences

ESLint rules

  • Added eslint-plugin-playwright rules to prevent flaky patterns from creeping back: no-wait-for-timeout (error), no-networkidle (warn), prefer-web-first-assertions (warn)

Parallelization

  • Nightly workflow now runs 4 parallel matrix runners instead of a single sequential job
  • Tests split using weighted bin-packing (FFD algorithm) for balanced runner load
  • Expected wall-clock time: ~80 min → ~20–25 min

Turbo

  • Fixed stale Cypress input globs (cypress/**, cypress.config.*) replaced with Playwright equivalents (e2e/**, playwright.config.*) for correct cache invalidation

What should be covered while testing?

  • Trigger the nightly workflow manually and verify all 4 matrix runners complete without infrastructure errors

@samuelreichert samuelreichert requested a review from a team as a code owner May 5, 2026 21:09
@samuelreichert samuelreichert force-pushed the e2e-reliability-and-performance branch from 05040a8 to e7a39b8 Compare May 5, 2026 21:14
@samuelreichert samuelreichert marked this pull request as draft May 6, 2026 07:48
samuelreichert and others added 25 commits May 21, 2026 17:07
@samuelreichert @claude
feat(run-e2e): add smoke suite support via E2E_SUITE env var and @smoke…
8cd87af 
… tag

When E2E_SUITE=smoke is set, only tests tagged with @smoke will run.
This enables fast PR feedback by running a minimal subset of tests.

Usage in test files:
  test("renders widget @smoke", async ({ page }) => { ... });

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
feat(run-e2e): add shared checkAccessibility helper
487a68d 
Adds checkAccessibility(page, selector, options) to mendix-helpers.
Wraps @axe-core/playwright with sensible defaults (wcag21aa tags).

Usage:
  import { checkAccessibility } from "@mendix/run-e2e/mendix-helpers";
  await checkAccessibility(page, ".mx-name-widget1");

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(run-e2e): add exports field for fixtures and mendix-helpers
0f3b7f2 
Without explicit exports, Node cannot resolve @mendix/run-e2e/fixtures
or @mendix/run-e2e/mendix-helpers from widget spec files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(run-e2e): waitForMendixApp must wait for page render and network …
f05e629 
…idle

The previous implementation resolved too early — mx.session exists before
sub-page widgets render. Now waits for:
1. domcontentloaded
2. mx.session + no progress indicator + .mx-page exists
3. networkidle (ensures widget data fetches complete)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(e2e): migrate remaining specs that codemod missed (expect,test or…
51def71 
…der)

Codemod regex only matched `{ test, expect }` ordering. Six specs used
`{ expect, test }` and were left importing from @playwright/test — no
fixture cleanup meant session exhaustion after ~5 tests.

Also fix codemod regex to handle both orderings for future runs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(run-e2e): worker-scoped session and waitForFunction timeout fix
067a909 
- Make mendixSession worker-scoped so each worker holds 1 Mendix session,
  preventing session exhaustion with parallel workers (4 workers < 5 license limit).
- Fix waitForFunction call: pass timeout as 3rd arg (options), not 2nd (arg).
  Previously actionTimeout (10s) was used instead of intended 60s.
- Set local workers to 4 (safe with worker-scoped sessions).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(datagrid-web): eliminate race conditions in filter e2e tests
06f13f9 
- FilteringSingle: replace allTextContents() after toHaveText() with
  single toHaveText([...array]) call that retries atomically.
- FilteringMulti: remove waitForTimeout(300) and allTextContents() pattern,
  use toHaveText()/toContainText() for auto-retry assertions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(e2e): remove flaky patterns from video-player and checkbox-radio …
6320d33 
…specs

- video-player-web: remove redundant waitForMendixReady (fixture handles it).
- checkbox-radio-selection-web: replace networkidle waits with waitForMendixApp.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
fix(run-e2e): remove networkidle from waitForMendixApp, add opt-in wa…
ffb0e6d 
…itForDataReady

networkidle hangs forever on pages with streaming content (video embeds,
websockets). The mx.session + .mx-page check is sufficient for readiness.
Add waitForDataReady helper for tests that explicitly need networkidle.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@samuelreichert @claude
chore(run-e2e): remove migrate-spec.mjs one-time migration script
3880ce3 
Migration to shared fixtures and waitForMendixApp is complete across all specs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@samuelreichert
fix(badge-button-web): remove racy nth(1) assertion in close page test
07e51cb
@samuelreichert
fix(run-e2e): reduce local workers from 4 to 2
4d7319f
@samuelreichert
docs(e2e): add Playwright test guidelines and update AGENTS.md
385fc32
@samuelreichert
test(e2e): update chromium-linux snapshots
0bfcb33
@samuelreichert
test(gallery-web): ensure text filter input has focus before screenshot
89f81c2
@samuelreichert
test(line-chart-web): wait for Plotly render before screenshot to fix…
3272620 
… blank snapshots
@samuelreichert
test(e2e): update snapshots for badge button, slider, and tooltip com…
8d71b46 
…ponents
@samuelreichert
test(e2e): update SkipLink tests for focus context and visibility checks
dc8d88a
@samuelreichert
test(e2e): update DataGridDropDownFilter tests to target correct elem…
a7d57cd 
…ent selectors
@samuelreichert
test(e2e): enhance heatmap chart tests to verify additional elements …
78080a0 
…visibility
@samuelreichert
test(e2e): fix skiplink visibility checks using getBoundingClientRect
3320a87
@samuelreichert
test(e2e): use toBeInViewport and drop stale count assertion
3acb9a6
@samuelreichert
test(e2e): enhance external video poster rendering checks and add scr…
5e46aa1 
…eenshot validation
@samuelreichert @claude
test(e2e): fix race condition and locator iteration in number filter …
854a2f9 
…test

Replace textContent()+toBe() with toHaveText() locator assertion so the
paging status check retries until the filter is applied. Fix row loop to
use expected.length and .nth(i) — rows.length on a Playwright Locator is
undefined, so the loop was silently never executing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@samuelreichert
docs(e2e): consolidate guidelines, remove duplication
6d43298 
- Replace inline rules in AGENTS.md with pointer to guidelines file
- Remove redundant waitForMendixApp calls from template (fixture wraps goto)
- Drop checklist section — rules already covered inline
- Drop Session Management section — merged into Imports & Setup
- Expand Locator Patterns: composed selectors, text-only = false positive
@samuelreichert samuelreichert force-pushed the e2e-reliability-and-performance branch from 8d6f6a3 to 6d43298 Compare May 21, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment