actions/index.html

<!DOCTYPE html>
<html lang="en-us">
  <head>
    <meta charset="utf-8">
    <title>The Capture-Handle Actions Mechanism</title>
    <script class="remove" src="capture-handle-actions.js" type="text/javascript"></script>
    <script src="https://www.w3.org/Tools/respec/respec-w3c" class="remove"></script>
  </head>
  <body>
    <section id="abstract">
      <h2>Abstract</h2>
      <p>
        This document proposes a mechanism by which an application <var>APP</var> can opt-in to
        exposing certain information with another application <var>CAPTR</var>, if
        <var>CAPTR</var> is screen-capturing the tab in which <var>APP</var> is running.
        It describes a mechanism for <a data-cite="SCREEN-CAPTURE#dfn-browser">tab capture</a> only.
      </p>
    </section>
    <section id="sotd">
      <p>
        This document is not complete.
      </p>
    </section>
    <section id="conformance"></section>
    <section>
      <h2>Problem Description</h2>
      <section id="problem-desc">
        <h3>Problem Description</h3>
        <p>
          Consider a web-application, running in one tab, which we’ll name "<var>main_app</var>."
          Assume <var>main_app</var> calls
          <a data-cite="SCREEN-CAPTURE#dom-mediadevices-getdisplaymedia">getDisplayMedia</a>
          and the user chooses to share another tab, where an application is running which we’ll
          call "<var>captured_app</var>."
        </p>
        <p>Note that:</p>
        <ol>
          <li><var>main_app</var> does not know what it is capturing.</li>
          <li>
            <var>captured_app</var> does not know that it is being captured; let alone by whom.
          </li>
        </ol>
        <p>
          Both of these traits are desirable in the general case, but there exists specific
          use cases where the user of <var>main_app</var> would benefit if <var>main_app</var>
          had available to it a limited set of standard instructions that <var>captured_app</var>
          has opted into receiving.
        </p>
        <p>
          We wish to enable the specific use cases while keeping the general case as it was
          before.
        </p>
      </section>
      <section id="use-case-1">
        <h3>Use-case #1: Driving Presentations from Video Conferencing Apps</h3>
        <p>
          Consider a collaborating presentation software and video-conferencing software. Assume the
          user is in a VC session. The user starts sharing a presentation. Both applications are
          interested in letting the VC app discover that it is capturing a slides session,
          so that the VC application will be able to expose
          controls to the user for flipping through slides. When the user clicks those controls, the
          VC app will be able to send messages to the presentation app, requesting that it do such
          things as flip through slides, etc.
        </p>
      </section>
    </section>
    <section>
      <h2>The Capture-Handle Actions Mechanism</h2>
      <p>
        The capture-handle actions mechanism consists of two parts - one on the captured side, one on
        the capturing side.
      </p>
      <ul>
        <li>
          <a href="#captured-side-actions">Captured applications</a> opt-in by registering support for
          standard actions they handle by calling {{MediaDevices/setSupportedCaptureActions}}.
        </li>
        <li>
          <a href="#capturing-side-actions">Capturing applications</a> may trigger these actions using
          {{MediaStreamTrack/sendCaptureAction}}.
        </li>
      </ul>
      <div class="note">
        There is disagreement on whether actions should be specified here or in a separate document.
      </div>
      <section id="captured-side-actions">
        <h3>Captured Side for Actions</h3>
        <p>
          Applications in top-level documents can declare the [=capture actions=]
          they support, if any. They would typically do so before even knowing if
          they are being captured. The intended use is for an application to expect to receive
          these actions from capturer applications wishing to control the progression of
          the captured session, in response to interaction with the user.
          Supported actions are declared by calling {{MediaDevices/setSupportedCaptureActions}}
          with an array of the names of actions the application is prepared to respond to.
        </p>
        <section id="set-supported-capture-actions">
          <h4>Registering and responding to capture actions</h4>
          <p>
            {{MediaDevices}} is extended with a method - {{MediaDevices/setSupportedCaptureActions}} -
            which accepts an array of {{DOMString}}s. By calling this method, an application
            registers with the user agent a set of zero or more [=capture actions=] it wishes to
            respond to.
          </p>
          <p>
            <dfn>Capture actions</dfn> are values defined in {{CaptureAction}}.
            They are meant to be interpreted as instructions from the capturing application to
            control the advancement of the presentation of the captured session, however the
            captured application wishes to define this. The intent is to support capturer
            applications implementing interactive controls for these actions, whose sending
            requires [=transient activation=] and [=consume user activation=].
          </p>
          <pre class="idl">
            partial interface MediaDevices {
              undefined setSupportedCaptureActions(sequence&lt;DOMString&gt; actions);
              attribute EventHandler oncaptureaction;
            };

            enum CaptureAction {
              "next",
              "previous",
              "first",
              "last"
            };
          </pre>
          <dl data-link-for="MediaDevices" data-dfn-for="MediaDevices" class="methods">
            <dt>
              <dfn>setSupportedCaptureActions</dfn>
            </dt>
            <dd>
              <p>When this method is invoked, the user agent MUST run the following steps:</p>
              <ol>
                <li>
                  If the [=relevant global object=]'s [=associated `Document`=] is
                  either not [=Document/fully active=] or its [=browsing context=] is not a
                  [=top-level browsing context=], then throw {{InvalidAccessError}}.
                </li>
                <li>
                  Let |actions| be the method's first argument.
                </li>
                <li>
                  If |actions| is non-empty, and this method was previously
                  called with a non-empty array on [=this=] {{MediaDevices}} object,
                  then throw {{InvalidStateError}}.
                </p>
                <li>
                  Remove from |actions| any value not found in {{CaptureAction}}.
                </li>
                <li>
                  Remove from |actions| any duplicates.
                </li>
                <li>
                  Set [=this=]'s {{MediaDevices/[[RegisteredCaptureActions]]}} to |actions|.
                </li>
                <li>
                  return `undefined` and run the remaining step [=in parallel=].
                </li>
                <li>
                  If this document is currently being captured as part of a
                  <a data-cite="SCREEN-CAPTURE#dfn-browser">browser</a>
                  <a data-cite="SCREEN-CAPTURE#dfn-display-surface">display surface</a>,
                  then for each capturer of that surface, queue a task on that capturer's
                  task-list to set all associated video {{MediaStreamTrack}}s'
                  {{MediaDevices/[[AvailableCaptureActions]]}} to |actions|.
                </li>
              </ul>
            </dd>
          </dl>
          <dl data-link-for="MediaDevices" data-dfn-for="MediaDevices" class="attributes">
            <dt><dfn>oncaptureaction</dfn> of type {{EventHandler}}</dt>
            <dd>
              <p>The event type of this event handler is `"captureaction"`.</p>
            </dd>
          </dl>
          <p>
            When {{MediaDevices}} is created, give it a
            <dfn data-dfn-for="MediaDevices">[[\RegisteredCaptureActions]]</dfn> internal slot,
            initialized to an empty list.
          </p>
        </section>
        <section id="action-event">
          <h3>Capture Action Event</h3>
          <section id="capture-handle-action-event">
            <h4><dfn>CaptureActionEvent</dfn></h4>
            <p>
              This event is fired on the captured application's {{MediaDevices}}
              object whenever an action it registered with
              {{MediaDevices/setSupportedCaptureActions}} has been triggered. This
              lets the application respond by executing its implementation of this
              action.
            </p>
            <pre class="idl">
              [Exposed=Window]
              interface CaptureActionEvent : Event {
                constructor(optional CaptureActionEventInit init = {});
                readonly attribute CaptureAction action;
              };
            </pre>
            <dl data-link-for="CaptureActionEvent" data-dfn-for="CaptureActionEvent">
              <dt>
                <dfn>action</dfn>
              </dt>
              <dd>
                The {{CaptureAction}} that was triggered.
              </dd>
            </dl>
          </section>
          <section>
            <h4><dfn>CaptureActionEventInit</dfn></h4>
            <pre class="idl">
              dictionary CaptureActionEventInit : EventInit {
                DOMString action;
              };
            </pre>
            <dl
              data-link-for="CaptureActionEventInit"
              data-dfn-for="CaptureActionEventInit"
            >
              <dt>
                <dfn>action</dfn>
              </dt>
              <dd>
                The {{CaptureAction}} to initialize the event with.
              </dd>
            </dl>
          </section>
        </section>
      </section>
      <section id="capturing-side-actions">
        <h3>Capturing Side for Actions</h3>
        <p>
          Capturing applications can enumerate available [=capture actions=] that
          are supported on the video track they have obtained, by using
          {{MediaStreamTrack/getSupportedCaptureActions}}, and can trigger those
          actions by using {{MediaStreamTrack/sendCaptureAction}}.
        </p>
        <section id="capture-handle-get-supported-actions">
          <h4>Enumerating supported actions and triggering them</h4>
          <p>
            When a {{MediaStreamTrack}} is a video track derived from screen-capture
            of a <a data-cite="SCREEN-CAPTURE#dfn-browser">browser</a>
            <a data-cite="SCREEN-CAPTURE#dfn-display-surface">display surface</a>,
            {{MediaStreamTrack/getSupportedCaptureActions}} returns the set of
            available [=capture actions=], if any, supported by the captured
            application associated with this video track.
          </p>
          <pre class="idl">
            partial interface MediaStreamTrack {
              sequence&lt;DOMString&gt; getSupportedCaptureActions();
              Promise&lt;undefined&gt; sendCaptureAction(CaptureAction action);
            };
          </pre>
          <dl data-link-for="MediaStreamTrack" data-dfn-for="MediaStreamTrack">
            <dt>
              <dfn>getSupportedCaptureActions</dfn>
            </dt>
            <dd>
              <p>When this method is invoked, the user agent MUST return [=this=]'
              {{MediaDevices/[[AvailableCaptureActions]]}} if defined, or `[]` if not defined.</p>
            </dd>
            <dt>
              <dfn>sendCaptureAction</dfn>
            </dt>
            <dd>
              <p>When this method is invoked, the user agent MUST run the following steps:</p>
              <ol>
                <li>
                  If the [=relevant global object=] of [=this=] does not have
                  [=transient activation=], return a promise [=rejected=] with
                  {{InvalidStateError}}.
                </li>
                <li>
                  [=Consume user activation=].
                </li>
                <li>
                  Let |action| be the method's first argument.
                </li>
                <li>
                  If |action| is not in [=this=]' {{MediaDevices/[[AvailableCaptureActions]]}},
                  return a promise [=rejected=] with {{NotFoundError}}.
                </li>
                <li>
                  Let |p| be a new promise.
                </li>
                <li>
                  Run the following steps [=in parallel=]:
                  <ol>
                    <li>
                      <p>
                        Queue a task on the task-list of the captured
                        <a data-cite="SCREEN-CAPTURE#dfn-browser">browser</a>
                        <a data-cite="SCREEN-CAPTURE#dfn-display-surface">display surface</a>'s
                        [=top-level browsing context=]'s [=active document=] to run the
                        following steps:
                      </p>
                      <ol>
                        <li>
                          Let |target| be the the [=relevant global object=]'s
                          [=associated `Document`=]'s
                          associated navigator's {{MediaDevices}} object.
                        <li>
                          If |action| is not in |target|'s
                          {{MediaDevices/[[RegisteredCaptureActions]]}}, abort these steps.
                        <li>
                          [=Fire an event=] named `"captureaction"`, using a
                          {{CaptureActionEvent}} with {{CaptureActionEventInit/action}}
                          set to |action|, at |target|.
                        </li>
                      </ol>
                      <li>
                        Wait for the event to have been fired.
                      </li>
                      <li>
                        Resolve |p|.
                      </li>
                    </li>
                  </ol>
                </li>
                <li>
                  Return |p|.
                </li>
              </ol>
            </dd>
          </dl>
          <p>
            When a video {{MediaStreamTrack}} is created as part of the
            <a data-cite="SCREEN-CAPTURE#dom-mediadevices-getdisplaymedia">getDisplayMedia</a>
            algorithm, whose source is a
            <a data-cite="SCREEN-CAPTURE#dfn-browser">browser</a>
            <a data-cite="SCREEN-CAPTURE#dfn-display-surface">display surface</a>,
            give it an
            <dfn data-dfn-for="MediaDevices">[[\AvailableCaptureActions]]</dfn> internal
            slot, initialized to the captured
            <a data-cite="SCREEN-CAPTURE#dfn-browser">browser</a>
            <a data-cite="SCREEN-CAPTURE#dfn-display-surface">display surface</a>'s
            [=top-level browsing context=]'s [=Browsing context/active window=]'s
            associated navigator's {{MediaDevices}} object's
            {{MediaDevices/[[RegisteredCaptureActions]]}}.
          </p>
          <p>
            While capture of a
            <a data-cite="SCREEN-CAPTURE#dfn-browser">browser</a>
            <a data-cite="SCREEN-CAPTURE#dfn-display-surface">display surface</a>
            is occurring, whenever that surface's
            [=top-level browsing context=] is navigated, then for each capturer of
            that surface, queue a task on that capturer's task-list to set all
            associated video {{MediaStreamTrack}}s'
            {{MediaDevices/[[AvailableCaptureActions]]}} to `[]`.
          </p>
    </section>
  <body>
</html>