Audio support for non-real-time Graphics

[andreialecu]

Context

#30 proposes an audioEnabled render requirement, which answers "can this Renderer play sound". That is a realtime playout concern and I think it stands on its own.

This issue covers a separate problem: how a non-real-time Graphic contributes audio to an offline render. The two are separable, and I suspect they will get muddled if they share a thread, hence the new issue.

Scope here is non-real-time only, meaning Graphics with supportsNonRealTime: true.

The problem

In non-real-time, the Renderer drives the Graphic with goToTime() and pulls one frame at a time. That loop is discrete, random access, and decoupled from the wall clock. It may run at 3x or at 0.2x.

Audio has no equivalent of "one frame". Anything the Graphic plays through Web Audio or an <audio> element is bound to the audio clock. During a goToTime() sweep the frames get captured, and the audio goes to a device nobody is recording, at the wrong speed.

So a Graphic that brings its own audio currently has no route into a post-production render. The only workaround is to ship the audio separately and have an editor line it up by hand, which defeats the point of packaging it with the Graphic.

Why non-real-time is the tractable case

Worth stating explicitly, because it is slightly counterintuitive.

In realtime, a Graphic does not know its own future. Actions arrive whenever an operator presses a button.

In non-real-time, setActionsSchedule() hands the Graphic its entire timeline up front, with timestamps. The Graphic can therefore compute exactly what audio should exist across the full duration, deterministically, before a single frame is rendered.

That is the property to build on.

A prerequisite gap: Graphics cannot declare their own duration

This is worth fixing with or without audio, and the audio proposal depends on it.

actionDurations is static manifest metadata. But a data-driven non-real-time Graphic does not know its length until load(). The chess Graphic below runs for moveCount * moveInterval + tail, which depends entirely on the payload. The only honest manifest value is -1, and then the Renderer has no idea how many frames to pull.

I propose extending the return of setActionsSchedule():

setActionsSchedule: (params) => Promise<EmptyPayload & {
  result?: {
    /** Total renderable extent, ms, same timebase as goToTime() */
    duration?: number
    /** Range over which renderAudio() may return non-silence */
    audioExtent?: { start: number, end: number }
  }
}>

duration is useful to anyone doing non-real-time work. audioExtent follows naturally once it exists, and is needed because audio tails do not respect the last scheduled action. A sting that rings out 800ms after stopAction() has no equivalent in the video model.

Proposal: renderAudio()

An optional function for non-real-time Graphics. The Renderer requests a time range, the Graphic returns PCM.

renderAudio: (
  params: {
    /** Start of the requested range, ms, same timebase as goToTime() */
    startTime: number
    /** End of the requested range, ms, exclusive */
    endTime: number
    /** Sample rate requested by the Renderer, Hz */
    sampleRate: number
    /** Channel count requested by the Renderer */
    channelCount: number
  } & VendorExtend
) => Promise<ReturnPayload & {
  result: {
    /**
     * One Float32Array per channel, non-interleaved.
     * Length MUST be round((endTime - startTime) / 1000 * sampleRate).
     */
    channels: Float32Array[]
    sampleRate: number
  }
}>

Notes on the shape:

Non-interleaved Float32Array per channel is exactly what AudioBuffer.getChannelData() returns, so a Graphic using OfflineAudioContext can hand back its render output with no conversion.

OfflineAudioContext renders faster than realtime and is deterministic, which matches the non-real-time contract already implied by goToTime().

The buffers are transferable, so this survives a Renderer that isolates the Graphic in an iframe or worker.

The same determinism requirement as goToTime() applies. The same range requested twice MUST produce identical samples. No Date.now(), no unseeded random.

Supporting changes:

1. A manifest field, producesAudio: boolean, so a Renderer knows before load whether to bother. See the schema note below.
2. An audio block in RenderCharacteristics passed to load(), carrying at least { sampleRate, channelCount, supportsRenderAudio }. This matters. renderRequirements is a hard gate, but a well-built Graphic should be able to degrade to silent rather than fail to match a Renderer, and it needs load() to tell it which.
3. The setActionsSchedule() return change described above.

Note on the manifest schema

producesAudio cannot be added by convention. The normative manifest schema sets "additionalProperties": false at the root, with "patternProperties": {"^v_.*": {}}, so any unprefixed field fails validation today.

The sample manifest below therefore uses v_producesAudio so it validates against the current schema as published. If this proposal is accepted, promoting it to producesAudio is a change to graphics/schema.json rather than a documentation change. Same applies to the setActionsSchedule() return fields.

Flagging it so the size of the ask is clear up front.

Sample case

A chess board that receives a full game and replays it. Each move animates, and each move plays a sound at the moment the piece lands.

The sound is not one sound. It varies three ways:

• Destination square sets the stereo position. A knight landing on b3 sits left, a rook landing on g1 sits right.
• Piece sets the pitch of the body. Heavier pieces land lower.
• Kind of move sets the character. A quiet move is a short wooden knock. A capture adds a noise transient. A check adds a chime above the body.

That is 64 squares by 6 pieces by 4 kinds. Pre-rendering the combinations as bundled files is not sensible, and the move list is only known at load().

Manifest

Validates against graphics/schema.json as currently published.

{
  "$schema": "https://ograf.ebu.io/v1/specification/json-schemas/graphics/schema.json",
  "id": "io.ebu.example.chess-replay",
  "version": "1.0.0",
  "name": "Chess board replay",
  "description": "Replays a game move by move. Each move animates and plays a sound as the piece lands.",
  "main": "chess-replay.mjs",
  "supportsRealTime": true,
  "supportsNonRealTime": true,
  "stepCount": 0,
  "actionDurations": [
    { "type": "playAction", "duration": -1 }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "moves": {
        "type": "string",
        "title": "Moves",
        "description": "Space separated move list. Each move is from:to:piece:kind, where piece is one of p n b r q k and kind is one of move capture check mate.",
        "default": "e2:e4:p:move e7:e5:p:move g1:f3:n:move b8:c6:n:move f1:b5:b:check",
        "order": 1
      },
      "moveInterval": {
        "type": "integer",
        "gddType": "duration-ms",
        "title": "Interval between moves",
        "default": 900,
        "minimum": 1,
        "order": 2
      },
      "moveDuration": {
        "type": "integer",
        "gddType": "duration-ms",
        "title": "Move animation length",
        "default": 400,
        "minimum": 1,
        "order": 3
      }
    }
  },
  "v_producesAudio": true
}

Graphic

Abbreviated to the time model and the audio path.

The rule it follows: everything is a pure function of time. No accumulated state, no stepping forward from the last frame. The Renderer is entitled to seek backwards, skip around, or render frames out of order, and in a distributed render it may do all three. A Graphic that tracks "current move" as mutable state produces a correct-looking preview and a wrong render.

// Longest tail any single move sound can produce.
const TAIL_MS = 700

const FILES = "abcdefgh"

class Graphic extends HTMLElement {
  async load({ data, renderType, renderCharacteristics }) {
    this.moves = data.moves
      .trim()
      .split(/\s+/)
      .map((m) => {
        const [from, to, piece, kind] = m.split(":")
        return { from, to, piece, kind }
      })

    this.moveInterval = data.moveInterval ?? 900
    this.moveDuration = data.moveDuration ?? 400
    this.origin = 0

    this.audioEnabled =
      renderType === "non-realtime" &&
      renderCharacteristics.audio?.supportsRenderAudio === true

    this.buildBoard()
    return { statusCode: 200 }
  }

  /** Impact time of move i, relative to the start of the replay. */
  impactTime(i) {
    return i * this.moveInterval + this.moveDuration
  }

  get replayDuration() {
    return this.impactTime(this.moves.length - 1) + TAIL_MS
  }

  async setActionsSchedule({ schedule }) {
    const play = schedule.find((s) => s.action.type === "playAction")
    this.origin = play ? play.timestamp : 0

    return {
      statusCode: 200,
      result: {
        duration: this.origin + this.replayDuration,
        audioExtent: {
          start: this.origin + this.impactTime(0),
          end: this.origin + this.replayDuration,
        },
      },
    }
  }

  /** Board state at t. Pure. Same input always gives the same output. */
  stateAt(t) {
    const settled = []
    let animating = null

    for (let i = 0; i < this.moves.length; i++) {
      const start = i * this.moveInterval
      const end = start + this.moveDuration

      if (t >= end) {
        settled.push(this.moves[i])
        continue
      }
      if (t >= start) {
        animating = {
          move: this.moves[i],
          progress: (t - start) / this.moveDuration,
        }
      }
      break
    }

    return { settled, animating }
  }

  async goToTime({ timestamp }) {
    this.paint(this.stateAt(timestamp - this.origin))
    return { statusCode: 200 }
  }

  async renderAudio({ startTime, endTime, sampleRate, channelCount }) {
    if (!this.audioEnabled) return { statusCode: 501 }

    // Lookback, so a capture landing just before the window rings into it.
    const from = startTime - TAIL_MS
    const length = Math.round(((endTime - from) / 1000) * sampleRate)
    const ctx = new OfflineAudioContext(channelCount, length, sampleRate)

    for (let i = 0; i < this.moves.length; i++) {
      // Identical formula the video path uses. One source of truth.
      const at = this.origin + this.impactTime(i)
      if (at < from || at >= endTime) continue

      this.scheduleMove(ctx, this.moves[i], (at - from) / 1000)
    }

    const buffer = await ctx.startRendering()
    const skip = Math.round((TAIL_MS / 1000) * sampleRate)

    return {
      statusCode: 200,
      result: {
        sampleRate: buffer.sampleRate,
        channels: Array.from({ length: buffer.numberOfChannels }, (_, i) =>
          buffer.getChannelData(i).slice(skip)
        ),
      },
    }
  }

  scheduleMove(ctx, move, when) {
    // Pan follows the destination file, a on the left through h on the right.
    const pan = ctx.createStereoPanner()
    pan.pan.value = FILES.indexOf(move.to[0]) / 3.5 - 1
    pan.connect(ctx.destination)

    const gain = ctx.createGain()
    gain.connect(pan)

    const base = { p: 320, n: 260, b: 260, r: 200, q: 170, k: 190 }[move.piece]
    const quiet = move.kind === "move"
    const decay = quiet ? 0.09 : 0.4

    gain.gain.setValueAtTime(quiet ? 0.22 : 0.4, when)
    gain.gain.exponentialRampToValueAtTime(0.0005, when + decay)

    const body = ctx.createOscillator()
    body.type = "triangle"
    body.frequency.setValueAtTime(base * 1.5, when)
    body.frequency.exponentialRampToValueAtTime(base, when + 0.05)
    body.connect(gain)
    body.start(when)
    body.stop(when + decay)

    if (move.kind === "capture" || move.kind === "mate") {
      const noise = ctx.createBufferSource()
      noise.buffer = this.noiseBuffer(ctx)
      const bp = ctx.createBiquadFilter()
      bp.type = "bandpass"
      bp.frequency.value = 1800
      noise.connect(bp).connect(gain)
      noise.start(when)
      noise.stop(when + 0.12)
    }

    if (move.kind === "check" || move.kind === "mate") {
      const chime = ctx.createOscillator()
      chime.frequency.value = base * 4
      chime.connect(gain)
      chime.start(when)
      chime.stop(when + decay)
    }
  }

  noiseBuffer(ctx) {
    if (this._noise?.sampleRate === ctx.sampleRate) return this._noise

    const buf = ctx.createBuffer(1, ctx.sampleRate * 0.15, ctx.sampleRate)
    const d = buf.getChannelData(0)

    // Seeded. Math.random() here would break re-render determinism silently.
    let seed = 1
    for (let i = 0; i < d.length; i++) {
      seed = (seed * 1103515245 + 12345) % 2147483648
      d[i] = seed / 1073741824 - 1
    }

    this._noise = buf
    return buf
  }
}

export default Graphic

Renderer sequence

load({ data: { moves }, renderType: "non-realtime", renderCharacteristics })
setActionsSchedule({ schedule })     -> duration, audioExtent
loop: goToTime({ timestamp })        -> capture frame
renderAudio({ startTime, endTime, sampleRate, channelCount })
mux

Note the shape of it. impactTime() is called by both paths and nothing else couples them. The frame loop never touches audio, renderAudio() never touches the DOM, and the two agree because they read the same function. That property only holds because the Graphic derives its timeline from data rather than accumulating it.

Alternative considered

A declarative audio timeline. The Graphic returns a list of clips referencing bundled audio resources, with in and out points, gain and fades, and the host mixes them.

That is lighter, it needs no PCM crossing the Renderer boundary, and it gives an NLE something an editor can grab and duck under commentary.

Worth being straight about this: the same chess Graphic shipping three fixed WAVs, one for move, one for capture, one for check, is exactly the declarative case. No synthesis needed. That version is probably what most people would build.

It is the per-square panning and per-piece pitch that push it over the line, because the variant count makes bundling impractical. So the two approaches are not rivals so much as a threshold. Below some amount of data dependence, declarative wins on every axis. Above it, only PCM works.

I would rather see the general mechanism specified first and the declarative form layered on later, since the reverse leaves the harder cases with no route at all. But I do not feel strongly, and if the group prefers the declarative form as the v1 answer I think that is defensible.

Open questions

1. Chunking. Should the spec require the Renderer to request the whole audioExtent in a single call? That removes the lookback problem at the cost of memory on long Graphics. The alternative is to state that Graphics MUST render events beginning before the requested window when the tail falls inside it, and leave the strategy to the implementer.

   The chess case makes this concrete. At a moveInterval of 900ms and a capture tail of 700ms, tails routinely cross chunk boundaries, and whether they do depends entirely on the data. Any Graphic that gets the lookback wrong produces clicks at the seams, and those are exactly the artefacts that survive review and reach air.

   I lean towards specifying the lookback obligation rather than mandating single-call, since long-form Graphics exist and forcing the whole extent into memory is a real cost. But this is the part I am least sure about.

2. Channel count. The sample assumes a stereo destination. On a mono render it downmixes, which is correct but silently loses the spatial information the Graphic is using to convey meaning. Should a Graphic be able to express "this is only meaningful in stereo", and if so does that belong here or in the render requirements discussion in Renderer Audio capabilities #30?

3. Codec floor for bundled audio resources. Uncompressed WAV as the MUST seems right for post, where seek accuracy matters more than file size, with anything else optional.

4. Loudness. Given this is EBU, R 128 will come up. Worth deciding early whether returned samples are expected at a normalised target or as raw signal with the host handling gain staging. I lean towards raw signal plus a documented headroom expectation.

5. skipAnimation interaction. If the animation is skipped, does audio truncate, drop, or still play in full?

6. Realtime. Whether any of this should be mirrored back into the realtime path, or whether realtime stays as Renderer Audio capabilities #30 describes it.

[andreialecu]

Demo of the above at: https://andreialecu.github.io/ograf-nrt-audio-bench/ (source at: https://github.com/andreialecu/ograf-nrt-audio-bench/blob/main/index.html)

[didikunz]

I don't understand the need for "startTime" and "endTime" parameters of "renderAudio" function. Would it not simply render all audio and use the definition given with "setActionsSchedule". The value of the "audioExtent" does then help define the length of the audio clip, if that is still needed. Because the NLE only needs to align the video and audio file at the same in-position to make it work, right? It maybe would not care, how much the audio is longer than the video, but just put it on the track.

I don't see a need to have audio before the in-point of the video, but if you see feet we could add a value for this also.