Accessibility Features Proposal - Expand Web Accessibility module

[calebfoss]

Topic

Written for the p5.js Documentation Organization & Accessibility project, this issue proposes a plan for upcoming improvements to p5's accessibility features.

Background on Existing Features

p5's gridOutput() and textOutput() methods are very cleverly designed tools that aim to automatically describe the content on the canvas.

Current limitations and issues for these methods include:

• descriptions are exclusively in English
• text is not described
• 3D shapes are not described
• 2D primitives rendered in webGL are described incorrectly due to not factoring in the camera's perspective
• 2D primitives with similar features are combined in descriptions, resulting in inaccurate counts of shapes
• 2D primitives positioned outside the canvas are described as though they are visible
• Custom 2D shapes made with beginShape() etc. are not described

So currently, these functions can accurately describe sketches featuring exclusively 2D primitives, each with somewhat distinct features, all on the canvas for an English-reading audience. They describe the color, approximate location, and approximate percentage of space occupied of each shape.

I would argue that this improves accessibility for a very limited selection of p5 sketches. Even those that can be accurately described.

Example

textOutput() produces the following description

Your output is a, 400 by 400 pixels, white canvas containing the following 4 shapes:

    yellow circle, at middle, covering 79% of the canvas.
    black circle, at top left, covering 5% of the canvas.
    black circle, at top right, covering 5% of the canvas.
    black arc, at middle, covering 22% of the canvas.

yellow circle	location = middle	area = 79%
black circle	location = top left	area = 5%
black circle	location = top right	area = 5%
black arc	location = middle	area = 22%

The description is technically accurate, but I would argue that it is significantly less useful at conveying the content than a concise, manually written description such as

A yellow smiley face on a light gray background.

Side Note on AI

An idea I anticipate coming up in relation to the issue I presented above is to use AI-generated descriptions. I would like to take a moment to strongly advise against that because

• A feature like that would most likely rely on a Large Language Model, which could put p5 in a position of depending on exploitative and inhumane labor practices.
• Even the most cutting edge AI models have an ongoing issue with conveying false information as fact (often referred to as "hallucinating").
• AI models have a well-documented habit of generating harmful content, particularly directed toward people of marginalized identities.

Moving forward

I would say that these features have a fork in the road ahead. They can either be further developed to improve accuracy and warn the creator about limitations or they can be removed from p5. With respect for the contributors who developed these very impressive features, unfortunately, I think the best path forward is the latter.

Keeping textOutput() and gridOutput()

That being said, if textOutput() and gridOutput() not removed, I would recommend:

• Throw a friendly error if these functions are used in webGL mode. I think accurately describing 3D content is an extremely ambitious goal. Accounting for camera perspective, occlusion, and lighting would be an enormous challenge. Describing custom models and procedurally geometry would be perhaps an even bigger challenge. Note: I had offered to implement the webGL error message a while back, started to work on it, got stuck on how to work it into the FES, and dropped the ball.
• Implement descriptions for text drawn on the canvas in 2D
• Update documentation in textOutput() and gridOutput() to clearly indicate their limitations
• Update outputs.js to check if a shape is off canvas and remove it from "ingredients" if so. This would involve collision detection, so this could be developed in parallel with a series of methods to help creators handle collision.
• Add translations and add a parameter to these methods to allow creators to select a language.

My opinion is that the above changes are a lot of work for features that would only improve accessibility in very limited situations.

Removing textOutput() and gridOutput()

Instead, I would recommend removing those methods and focusing on encouraging creators to describe their sketches effectively and to provide good resources to do so. While automating this task is an enticing idea, there is something very human about effectively conveying visual content through written language. Learning to write a helpful accessible description is just as valuable as to making strong web content as learning to generate imagery programmatically.

Already, the Writing Accessible Canvas Descriptions tutorial is a fantastic resource. Additionally, sketches featured Examples page on the new p5 site include describe(), describeElement(), textOutput(), and/or gridOutput().

Returning an Object with a Describe Method

Here is an idea that would be a big undertaking but perhaps not as big as the improvements textOutput() and gridOutput() listed above and, I would argue, would have a much bigger impact:

What if instead of returning the p5 instance, visual functions returned an object that contained information about it and a method to describe it?

Example:

let sunY = 50;

function setup() {
  createCanvas(400, 400);
  describe('An outdoor scene.');
}

function draw() {
  background('lightblue');

  fill('yellow');
  const sun = circle(50, sunY, 50);

  fill('green');
  const grass = rect(0, 300, width, 100);
  grass.describe('An empty grassy space is in the foreground.');

  if (sun.y < 200) {
    sun.describe('The sun is high in the sky.');
  } else if (sun.y < grass.y + sun.diameter / 2) {
    sun.describe('The sun is setting at the horizon.');
  }
  sunY++;
}

This is essentially a way to more directly connect what describeElement() does to the functions producing what it describes. Storing the arguments passed into the shape functions helps the creator keep track of what those values represent.

In many cases, an "element" of the composition will be rendered using multiple functions, so describeElement() will still be useful for those cases.

A potential additional feature this could open the door for would be using isPointInPath() and/or isPointInStroke() to store in this object a boolean property indicating whether the shape was clicked this frame.

In my experience, the work required to check if a shape was clicked (either manually through collision detection algorithms or through loading a library) is a significant obstacle to p5 creators.

Performance impact would need testing, as the way I am describing this feature, those methods would need to be called every time any shape is drawn. Additionally, the mouse coordinates would need to be transformed using the current transformation matrix.

I believe that some of the existing helper functions for textOutput() and gridOutput() could be repurposed to keep track of "ingredients" and reuse HTML elements.

Color Contrast

Issue #6971 proposes a way to check whether the contrast between colors in a sketch is meeting WCAG AA or AAA.

I drafted a method to calculate contrast based on WebAIM's tool. When passed two p5.Color's, it returns the contrast ratio between them: https://editor.p5js.org/cfoss/sketches/-okUAfg7Z

Note that this does not currently account for alpha level or blend mode, and it would need to do that for accuracy.

This could be modified to check for specific AA and AAA target ratios. I would argue, though, that this information would be better placed in the reference, where it could be linked, contextualized, and updated without modifying the function itself.

@limzykenneth responded

I wonder if this can be expanded into a wider accessibility checker feature that can form part of the Web Accessibility module of p5.js? Eg. also adding a function that when passed a HTML container element, check everything inside it for adherene to best practices of web accessibilty/WCAG recommendations?

While I think a broader accessibility checker is an exciting idea, I believe that implementing it such that it reliably gives accurate results would be a daunting task.

I am not aware of a way to check what default styles a browser has applied to an element beyond the styles set via CSS, style property, etc. On top of that, accounting for all the possible factors that would influence which element appears on top of which would be very challenging.

For those reasons, when it comes to color contrast at least, I do not think automatically checking elements is a realistic goal.

I could imagine an automatic checker for semantic structure: checking for more than one h1, mismatching h1 and title, disorganized heading levels, etc. Having a button in the p5 editor that opens such a checker would be great.

Flashing light

WCAG 2.3.2 specifies limits on flashing light.

I would propose a method for testing whether what is rendered to the canvas meets these limits written something like

function setup() {
  createCanvas(400, 400);
  checkFlashing();
}

that would throw a friendly error if and when the sketch fails the test. This could share a helper function with the contrast function as both are based on relative luminance.

Calculating relative luminance across the canvas every frame would likely be quite computationally expensive, but for running a test, I think that would be acceptable.

Disabling Animation

WCAG 2.3.3 specifies that "Motion animation triggered by interaction can be disabled, unless the animation is essential to the functionality or the information being conveyed."

I would propose that, by default, p5 create an HTML element that acts as a pause/play button for the sketch, connected to loop()/noLoop(). This behavior could be disabled or reenabled with methods called something like noPause() and allowPause() respectively.

(edited typos)

[Qianqianye]

Thanks @calebfoss! Tagging @limzykenneth and @davepagurek to review.

[Forchapeatl]

Hi @calebfoss I am interested in taking the implementation these 3 features.

text is not described.
3D shapes are not described.
inaccurate covering results [new] .

Should I submit as 3 separate Pull Requests ?

[calebfoss]

@Forchapeatl, I appreciate you volunteering to work on this!

I think before starting on those, there is a decision to be made on whether textOutput and gridOutput should remain part of the library. Above, I argue for deprecating them. If they are not removed, I include suggestions in the "Keeping textOutput() and gridOutput()" section. I would hate for you to put time and effort until something just before it gets removed anyway!

@Qianqianye, is there a plan on whether to keep those functions? And if not, are there other web accessibility features that need support? I took a quick look at the current issues and p5 2.0 proposals list to see if there was anything to suggest, but I didn't immediately find anything.

[ksen0]

Regarding splitting this into more workable chunks, given that neither textOutput() not, here's my 2 cents (please feel free to iterate on this):

1. [ready for work] Documenting limitations of textOutput() and gridOutput() in 1.x (and then duplicating that in 2.0) - this might be a "Good First Issue" for a contributor; maybe textOutput() and gridOuput() are incomplete and inaccurate in WEBGL #6126 this issue should be converted to that?
2. [open for discussion] Flashing light - functionality I agree on, the syntax I'm not sure about. I think rather than something like checkFlashing() it would be better to have a class that may later be extended, e.g., A11yTools.checkFlashing() or something. It would also be great to get some feedback from others on this API design.
3. [open for discussion] Disabling Animation - I'm not totally clear on the scope of work here but if the outcome is that there's a html component over the canvas that allows turning off animation (and respects prefers-reduced-motion? Or is that scope creep?) that seems great to work toward
4. [open for discussion] In 2.0 - Improving textOutput() and gridOutput() for text in both 2D and 3D (same; 2.0 has improvements that I hope could make 3D implementation more tractable)
5. [open for discussion] In 2.0 - Improving textOutput() and gridOutput() in WEBGL 3D (I suspect there's prior discussion on this I don't know about, can be linked - not sure how major of an undertaking but I imagine considerable)
6. [open for discussion] In 2.0 - Improving textOutput() and gridOutput() for shapes (same)
7. [open for discussion] Uses of AI for a11y (see also comments in p5.js 2.0 Update: Beta, Timeline, and Compatibility Addons #7488)

I think 4-7 and the following are blocked by additional input, as I've mentioned before, same as:

Here is an idea that would be a big undertaking but perhaps not as big as the improvements textOutput() and gridOutput() listed above and, I would argue, would have a much bigger impact:
What if instead of returning the p5 instance, visual functions returned an object that contained information about it and a method to describe it?

This is where I really don't feel like I have the ability to make a judgment. The more I go through into the prior research (almost a decade) on these features in p5.js, the more I feel that deciding whether to invest development in something like .describeElement (or another alternative) or building out textOutput and gridOutput support (and if/how AI is involved, and which kind of AI) has to be done in community of screen reader users, and as I mentioned I'm working on a pathway toward this, so I really hope that this can be unblocked in the next couple of months. (But I will update more concrete timelines as I can!)

However, I think 1-3 above can be underway. Apologies if I missed something. What do you think @calebfoss ?

[calebfoss]

@ksen0 - Great breakdown and feedback!

Totally agree that the more open question of how to write descriptions for screen readers would greatly benefit from the testing you have planned. Before evaluating the scope for the future of textOutput() and gridOutput(), it would be important to determine how often a technical description of the shapes present on a canvas is preferable to a custom written one (via describe() / describeElement()) for people who use screen readers. My expectation would be almost never, but I do not use a screen reader. I imagine that there is data/documentation archived somewhere from the research Claire Kearney-Volpe did with screen reader users years ago that informed these features in the first place.

I also found this presentation from over a decade ago with some really interesting ideas on accessible HTML5 canvas descriptions.

A couple specific notes:
2 - I appreciate the thinking ahead for packaging together multiple accessibility tools. Historically p5's design has avoided dot notation for the most beginner friendly end of the API. It would be great to start p5 creators early on accessibility tools, so that would be an argument for a global function. Then again, my recent suggestion in #6971 goes against that. Were you to go the global route for both, you could go with the contrast() function I originally proposed. Organizing accessibility tools in the same reference section would be another way to group them together. That would also help with the fact that textOutput() and gridOutput() to me don't sound like they would be accessibility features by their names. So I agree - a few different approach to be discussed that would benefit from more perspectives.

3 - Good point about prefers-reduced-motion, but my take is that starting paused as a default, regardless of prefers-reduced-motion, would keep the implementation simpler and not rely on the page visitor having marked that setting on their device. Bonus benefits of this beyond accessibility would be that the play button could also switch focus to the canvas element and/or initialize audio contexts, allowing keyboard input and audio playback as soon as the button is pressed.

Regarding my pitch to return objects from shape functions: I should note that this is sort of a breaking change. While I don't think this was every documented on the website, I believe those functions were marked as chainable in the JSDoc. I have never seen anyone actually use this in p5, but you can technically do this:

const sketch = (p) => {
  p.setup = () => {
    p.createCanvas(400, 400);
    p.background(0)
    .fill('pink')
    .circle(200, 200, 100);
  }
}

new p5(sketch);

(this works in global mode too, but is even less useful there)