roslib-media

Low-latency H.264 video for roslibjs apps — WebCodecs decode over a binary media channel or WebTransport, with automatic fallback to standard rosbridge.

@husarion/roslib-media is an additive companion to roslibjs. You keep one ROSLIB.Ros connection for all your topics and services; you reach for a MediaTopic only where real-time video matters. It feeds H.264 access units straight into the browser's hardware WebCodecs decoder and, against a husarion_rosbridge server, moves the bytes over a minimal binary channel or WebTransport (HTTP/3) so a lost frame never head-of-line-blocks the next.

It wraps, never replaces roslibjs — every media tier degrades to plain rosbridge CBOR, so a MediaTopic is always safe to drop in, even against a stock rosbridge_server.

Why

Browsers can hardware-decode H.264 with WebCodecs, but the usual ROS web path either ships frames as a CBOR uint8[] and decodes in JS, or relies on the Foxglove protocol and its unmaintained JS client. roslib-media keeps the standard rosbridge control plane and adds a media plane engineered for live robot video:

• Hardware decode — Annex-B → VideoDecoder → VideoFrame. No JS bitstream parsing, no canvas-2D YUV conversion.
• Keyframe-aligned start — the bridge caches SPS/PPS and holds a fresh subscriber at the next IDR, so you never see the green-smear cold start of a mid-GOP join.
• No head-of-line blocking — over WebTransport, one QUIC stream per access unit; a late frame can't stall the stream behind it. The headline win over any WebSocket/TCP bridge.
• GOP-aware backpressure — under load the bridge drops whole P/B frames to the next IDR rather than growing latency or corrupting the stream.

How it compares

	roslib-media	decode-in-JS over rosbridge	Foxglove SDK (JS)
Decode	WebCodecs (hardware)	software (JS/wasm)	WebCodecs
Transport	binary channel / WebTransport	CBOR uint8[]	Foxglove WS (TCP)
HOL-blocking under loss	no (WebTransport)	yes	yes
Late-join	keyframe-aligned (bridge)	manual	SDK-handled
Client	wraps stock roslibjs	stock roslibjs	unmaintained JS client
Server needed	any rosbridge (CBOR) → husarion_rosbridge (binary/WT)	any rosbridge	foxglove_bridge

Honest scope: H.264 first. MJPEG/CompressedImage and PointCloud2 over the same media plane are planned. The binary and WebTransport tiers require a husarion_rosbridge server; against any other rosbridge server it transparently falls back to CBOR.

Install

npm install @husarion/roslib-media roslib

roslib is a peer dependency — you bring your own version.

Quick start

import { Ros } from "roslib";
import { MediaTopic } from "@husarion/roslib-media";

const ros = new Ros({ url: "wss://my-robot.local/ws" });

const canvas = document.querySelector("canvas")!;
const ctx = canvas.getContext("2d")!;

const cam = new MediaTopic(ros, { name: "/camera/color/h264" });
await cam.subscribe((frame) => {
  // `frame` is a WebCodecs VideoFrame — draw it, then release it.
  ctx.drawImage(frame, 0, 0, canvas.width, canvas.height);
  frame.close();
});

console.log("streaming over", cam.tier); // "webtransport" | "binary" | "cbor"

// later…
cam.unsubscribe();

Everything else in your app keeps using stock roslibjs — ROSLIB.Topic, ROSLIB.Service, the lot — over the same connection.

Transport tiers

new MediaTopic(ros, { name, transport }) accepts:

transport	Behaviour
"auto" (default)	Best the browser + bridge support: WebTransport → binary → CBOR.
"webtransport"	Force WebTransport; falls back if unavailable.
"binary"	Force the WS binary channel; falls back to CBOR.
"cbor"	Plain rosbridge CBOR — works against any rosbridge server.

The chosen tier is reported on cam.tier after subscribe() resolves.

Status

Alpha / scaffold. The API surface and the wire-format parsing are in place; the binary/WebTransport tiers are wired against the husarion_rosbridge media plane as it lands (see the TODO(bridge) markers in src/). Tracking issues live in husarion_rosbridge.

Related

• husarion_rosbridge — the Rust rosbridge v2 server that hosts the media plane this client consumes.
• roslibjs — the standard ROS ↔ JavaScript library this wraps.

License

Apache-2.0