HTML-in-Canvas

[khushalsagar]

What problem are you trying to solve?

High-performance web applications rely on <canvas> for control over rendering that goes beyond standard HTML/CSS. However, by using <canvas>, they lose all the rich, built-in features of the web platform, including:

• Advanced formatting, such as styled multi-line text.
• Interactivity, such as forms, selection, and native scrolling.
• Accessibility.

This forces a bad trade-off: developers must either re-implement these fundamental browser features in JS, or simply not provide them, resulting in a sub-par, inaccessible user experience.

What solutions exist today?

• Canvas 2D Text APIs: fillText() and strokeText() are limited in that they only draw a single, un-wrapped line of text with a single style. They have no concept of layout, internationalization, or interactivity. WebGL provides no text rendering support at all.
• DOM Overlays: developers can overlay elements on top of the canvas. This is fragile and makes it impossible to compose HTML content with 2D or 3D canvas graphics (e.g., applying a WebGL shader to an HTML element).
• "Invisible DOM" for Accessibility: Apps build a duplicate, invisible DOM tree just for screen readers.

How would you solve it?

Add the ability to render elements from the canvas subtree directly into canvas, and add the ability to make this rendering interactive (forwarding input and handling updates). Adding these capabilities enables canvas surfaces to benefit from all of the styling, layout and behaviors of HTML/SVG.

Anything else?

This feature is being incubated in WICG following the WHATWG standards process.

Explainer: https://github.com/WICG/html-in-canvas
Ongoing prototyping in Chromium
Contributor: Google, Igalia
Workstream: HTML

[khushalsagar]

We'd like to move to Stage 1, links to relevant documents are in the comment above.

[smaug----]

I think I might want to see the explainer go through the known issues in some more detail before moving this to stage 1. Especially hit testing can be rather problematic (some issues mentioned in WICG/html-in-canvas#9) and it is unclear how various APIs dealing with coordinates would work.
And also things like IME popups - since the element's layout depends on the canvas itself, it is unclear how IME related popups can position themself correctly, if the visual representation on the canvas is in totally different place.

The problem space itself is interesting. Reminds me of mozSetImageElement or drawWindow.

[annevk]

cc @whatwg/canvas

[khushalsagar]

Thanks for the feedback @smaug----. I’ve replied to the hit testing and IME popup points in WICG/html-in-canvas#9. I’m curious to hear if the experience with mozSetImageElement gives us some ideas here.

That said, if I understand the stages correctly, the fact that we’re aligned that the WHATWG should explore this problem space implies we can proceed to stage 1. The questions you’re raising is exactly the kind of iteration/engagement we want to move this feature to. Stage 2 (iteration) is what would need all these issues to be resolved.

[smaug----]

I was referring to "A comprehensive explainer " as requirement for stage 1. A bit nitpicking, I know, but there were quite a few unknowns still.