Setup playwright tests for accessibility testing by radmorecameron · Pull Request #8664 · FreeTubeApp/FreeTube

radmorecameron · 2026-02-14T01:26:46Z

Pull Request Type

Other

Related issue

related to: #6661 (a static analyzer of some sort might be better for this use case though?)

Description

This PR sets up initial accessibility testing through playwright and axe.

The tests are currently failing due to accessibility issues, I fixed one of the issues but there's still 2 accessibility issues for mobile/tablet and 4 for desktop.

Testing

To run the tests, you can use yarn run test:e2e, you may need to run npx playwright install first

Future

This is just an initial implementation but in the future it'd probably be worth it to:

run these tests in CI (only if UI code is touched? ie: not dependabot)
add more tests (ex: tests that change theme)
mock API requests so more can be tested

Desktop

OS: Fedora Linux
OS Version: 43
FreeTube version: latest nightly

radmorecameron · 2026-02-14T01:45:21Z

Oh yeah for added context these are the failures:

For mobile/tablet it doesn't like:

no h1 on page
multiple nav elements (/ role = navigation) but no aria-label

[
  {
    "description": "Ensure landmarks are unique",
    "help": "Landmarks should have a unique role or role/label/title (i.e. accessible name) combination",
    "helpUrl": "https://dequeuniversity.com/rules/axe/4.11/landmark-unique?application=playwright",
    "id": "landmark-unique",
    "impact": "moderate",
    "nodes": [
      {
        "all": [],
        "any": [
          {
            "data": { "accessibleText": null, "role": "navigation" },
            "id": "landmark-is-unique",
            "impact": "moderate",
            "message": "The landmark must have a unique aria-label, aria-labelledby, or title to make landmarks distinguishable",
            "relatedNodes": [
              {
                "html": "<div data-v-2ac94927=\"\" data-v-4afaba5a=\"\" data-v-2fa9d48e=\"\" class=\"ft-flex-box sideNav closed\" role=\"navigation\">",
                "target": [".sideNav"]
              }
            ]
          }
        ],
        "failureSummary": "Fix any of the following:The landmark must have a unique aria-label, aria-labelledby, or title to make landmarks distinguishable",
        "html": "<nav data-v-50bc178e=\"\" data-v-2fa9d48e=\"\" class=\"topNav\">",
        "impact": "moderate",
        "none": [],
        "target": ["nav"]
      }
    ],
    "tags": ["cat.semantics", "best-practice"]
  },
  {
    "description": "Ensure that the page, or at least one of its frames contains a level-one heading",
    "help": "Page should contain a level-one heading",
    "helpUrl": "https://dequeuniversity.com/rules/axe/4.11/page-has-heading-one?application=playwright",
    "id": "page-has-heading-one",
    "impact": "moderate",
    "nodes": [
      {
        "all": [
          {
            "data": null,
            "id": "page-has-heading-one",
            "impact": "moderate",
            "message": "Page must have a level-one heading",
            "relatedNodes": []
          }
        ],
        "any": [],
        "failureSummary": "Fix all of the following:Page must have a level-one heading",
        "html": "<html lang=\"en-US\">",
        "impact": "moderate",
        "none": [],
        "target": ["html"]
      }
    ],
    "tags": ["cat.semantics", "best-practice"]
  }
]

For desktop, it's a lot of buttons without text/titles/labels and colour contrast issues as well

absidue

The biggest issue I see with this pull request at the moment is that instead of running inside Electron like FreeTube does, it uses the unsupported, mostly broken web build. Unless it is possible to do it inside Electron, I don't think it is worth merging this, as running tests in an environment that doesn't match how FreeTube is actually used and where most of the functionality is unsupported and broken, is not a proper test.

Additionally could you please rename the pull request and code references to say accessibility testing, so it actually matches what it does (calling accessibility tests, in the wrong environment on only 3 pages "end to end tests" is quite misleading)

P.S. We only have the web build in-place in CI because it serves as a smokescreen for downstream FreeTubeAndroid, to check that Electron only APIs are properly guarded by build flags. FeeeTubeAndroid has reimplemented a lot of the Electron functionality with Android APIs instead so it doesn't use the web build either.

radmorecameron · 2026-02-14T16:32:53Z

Additionally could you please rename the pull request and code references to say accessibility testing, so it actually matches what it does (calling accessibility tests, in the wrong environment on only 3 pages "end to end tests" is quite misleading)

Thanks for looking at this @absidue 😄

It should now be running through electron instead (the only drawback to this is that the tests can't be ran in parallel)
For now it's only accessibility tests but it can definitely expand to other types of tests so I don't think it'd be a good idea to make the code references "accessibility tests" but I think that makes sense for the PR title

package.json

eslint.config.mjs

absidue

In the future it would probably be good to add a test build to FreeTube, that uses in-memory nedb databases rather than a production build that shares the development data stores. That's not a major issue for this pull request as it doesn't modify any significant user data but it would definitely need to be done before any tests that do actual data changes are implemented.

package.json

tests/playwright/helpers.mjs

eslint.config.mjs

package.json

github-actions · 2026-03-07T20:12:31Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

github-actions · 2026-03-08T01:10:33Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2026-03-11T19:29:17Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

github-actions · 2026-03-13T14:52:05Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2026-03-14T00:30:49Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

github-actions · 2026-03-17T00:03:24Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

…te test-results directory

todo: look into using it for playwright tests

github-actions · 2026-03-25T01:25:41Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

FreeTubeBot enabled auto-merge (squash) February 14, 2026 01:26

github-actions bot added PR: dependencies Pull requests that update a dependency file PR: waiting for review For PRs that are complete, tested, and ready for review labels Feb 14, 2026

efb4f5ff-1298-471a-8973-3d47447115dc requested review from PikachuEXE and absidue February 14, 2026 07:17

absidue requested changes Feb 14, 2026

View reviewed changes

efb4f5ff-1298-471a-8973-3d47447115dc added PR: changes requested and removed PR: waiting for review For PRs that are complete, tested, and ready for review labels Feb 14, 2026

radmorecameron marked this pull request as draft February 14, 2026 12:13

auto-merge was automatically disabled February 14, 2026 12:13
Pull request was converted to draft

radmorecameron commented Feb 14, 2026

View reviewed changes

package.json Show resolved Hide resolved

radmorecameron commented Feb 14, 2026

View reviewed changes

eslint.config.mjs Show resolved Hide resolved

radmorecameron changed the title ~~Setup playwright tests~~ Setup playwright tests for accessibility testing Feb 14, 2026

radmorecameron marked this pull request as ready for review February 14, 2026 16:45

FreeTubeBot enabled auto-merge (squash) February 14, 2026 16:45

github-actions bot added the PR: waiting for review For PRs that are complete, tested, and ready for review label Feb 14, 2026

radmorecameron requested a review from absidue February 14, 2026 20:38

ChunkyProgrammer removed the PR: changes requested label Feb 17, 2026

absidue reviewed Feb 22, 2026

View reviewed changes

package.json Show resolved Hide resolved

tests/playwright/helpers.mjs Outdated Show resolved Hide resolved

eslint.config.mjs Outdated Show resolved Hide resolved

package.json Show resolved Hide resolved

package.json Show resolved Hide resolved

absidue added PR: changes requested and removed PR: waiting for review For PRs that are complete, tested, and ready for review labels Feb 22, 2026

auto-merge was automatically disabled February 22, 2026 16:58
Head branch was pushed to by a user without write access

radmorecameron force-pushed the setup-playwright-tests branch from e24b928 to f831d0c Compare February 22, 2026 16:58

FreeTubeBot enabled auto-merge (squash) February 22, 2026 16:58

auto-merge was automatically disabled February 22, 2026 17:04
Head branch was pushed to by a user without write access

FreeTubeBot enabled auto-merge (squash) February 22, 2026 17:04

auto-merge was automatically disabled February 22, 2026 17:06
Head branch was pushed to by a user without write access

FreeTubeBot enabled auto-merge (squash) February 22, 2026 17:06

github-actions bot removed the PR: merge conflicts / rebase needed label Mar 7, 2026

github-actions bot added the PR: merge conflicts / rebase needed label Mar 8, 2026

auto-merge was automatically disabled March 11, 2026 19:29
Head branch was pushed to by a user without write access

radmorecameron force-pushed the setup-playwright-tests branch from 42f9150 to 9b176a9 Compare March 11, 2026 19:29

FreeTubeBot enabled auto-merge (squash) March 11, 2026 19:29

github-actions bot removed the PR: merge conflicts / rebase needed label Mar 11, 2026

github-actions bot added the PR: merge conflicts / rebase needed label Mar 13, 2026

auto-merge was automatically disabled March 14, 2026 00:30
Head branch was pushed to by a user without write access

radmorecameron force-pushed the setup-playwright-tests branch from 9b176a9 to 8fc1724 Compare March 14, 2026 00:30

FreeTubeBot enabled auto-merge (squash) March 14, 2026 00:30

github-actions bot removed the PR: merge conflicts / rebase needed label Mar 14, 2026

github-actions bot added the PR: merge conflicts / rebase needed label Mar 17, 2026

radmorecameron added 7 commits March 24, 2026 21:21

Setup playwright tests

fa2e500

Run tests in electron, add eslint plugin for playwright

a347cbf

change order of imports in eslint config, update clean script to dele…

7740aaf

…te test-results directory

Use Desktop Chrome instead of Desktop Firefox for viewport config

90c9933

add command line argument to dev-runner to use in-memory database

eae903c

todo: look into using it for playwright tests

Do a separate pack for playwright tests

c2c5452

Undo change to dev-runner

ef3060e

auto-merge was automatically disabled March 25, 2026 01:25
Head branch was pushed to by a user without write access

radmorecameron force-pushed the setup-playwright-tests branch from 8fc1724 to ef3060e Compare March 25, 2026 01:25

FreeTubeBot enabled auto-merge (squash) March 25, 2026 01:25

github-actions bot removed the PR: merge conflicts / rebase needed label Mar 25, 2026

Conversation

radmorecameron commented Feb 14, 2026

Pull Request Type

Related issue

Description

Testing

Future

Desktop

Uh oh!

radmorecameron commented Feb 14, 2026

Uh oh!

absidue left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radmorecameron commented Feb 14, 2026

Uh oh!

Uh oh!

Uh oh!

absidue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 7, 2026

Uh oh!

github-actions bot commented Mar 8, 2026

Uh oh!

github-actions bot commented Mar 11, 2026

Uh oh!

github-actions bot commented Mar 13, 2026

Uh oh!

github-actions bot commented Mar 14, 2026

Uh oh!

github-actions bot commented Mar 17, 2026

Uh oh!

github-actions bot commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

absidue left a comment •

edited

Loading