VideoFrameProcessor interface

Per-frame video transform. App implements process(frame); SDK calls it once per captured camera frame; the returned frame is what gets encoded and published to peers.

Attach via VideoSDK.applyVideoProcessor(). Sticky — set once, applies to every camera stream of the local participant.

Blueprint

Single-method contract. No lifecycle hooks. App owns its own resources (models, canvases, WASM, workers) — the SDK never touches them.

interface VideoFrameProcessor { // Called once per captured frame. Return the (possibly modified) frame. // Sync or async — async work that takes longer than a frame interval will drop frames. process(frame: VideoFrame): VideoFrame | Promise<VideoFrame>; }

App owns all resource lifecycle. Load ML models, decode images, allocate canvases before creating the processor — the processor is just the per-frame function. Clean up your resources whenever you want; the SDK doesn't manage them.

Members

process

Called once per captured camera frame at the publish frame rate (typically 30–60Hz). Return a VideoFrame — that's what gets encoded and published. Can be sync or async.

process(frame: VideoFrame): VideoFrame | Promise<VideoFrame>

For source-style processors (canvas, generative, file playback): ignore the input frame and return generated frames. The SDK still drives the call cadence — return a fresh frame each call.

Frame handling. Call frame.close() on the input once you're done reading it. Return a new VideoFrame for output (or return the input frame unchanged for pass-through).

Performance. process runs at 30–60Hz. Async work that takes longer than a frame interval will drop frames. For heavy ML inference, run in a worker thread and return a placeholder frame while pending — or accept the framerate hit.

When the SDK stops calling `process`

Trigger	What happens
`VideoSDK.removeVideoProcessor()`	SDK stops calling `process`. Frames flow camera → encoder unmodified.
`VideoSDK.applyVideoProcessor(other)`	SDK stops calling the old processor and starts calling the new one on the next frame.
`me.unpublishVideo()`	Camera stream ends — no more frames to process.
`room.leave()`	All local streams stop. Processor receives no further frames.
Camera unplugged / permission revoked	Underlying source ends. SDK stops calling `process`.

After any of these, the processor object is just a plain JS object. The SDK no longer references it. Your own resources (models, canvases, WASM) stay alive until you dispose them in your own code.

Examples

Virtual background — segment + composite (app owns the model)

// App-owned setup — done once, in app code
const segmentationModel = await loadSegmentationModel();
const beachBg = await loadImageAsBitmap('/assets/beach.jpg');
const canvas = new OffscreenCanvas(1280, 720);
const ctx = canvas.getContext('2d');

// The processor — just the per-frame transform
const virtualBg = {
  async process(frame) {
    const mask = await segmentationModel.segment(frame);

    ctx.drawImage(beachBg, 0, 0, canvas.width, canvas.height);
    ctx.globalCompositeOperation = 'destination-out';
    ctx.drawImage(mask, 0, 0);
    ctx.globalCompositeOperation = 'source-over';
    ctx.drawImage(frame, 0, 0);

    frame.close();
    return new VideoFrame(canvas, { timestamp: frame.timestamp });
  },
};

await VideoSDK.applyVideoProcessor(virtualBg);

// Later — remove and clean up YOUR own resources
await VideoSDK.removeVideoProcessor();
segmentationModel.dispose();
beachBg.close();

Background blur (using an OffscreenCanvas filter)

const canvas = new OffscreenCanvas(1280, 720);
const ctx = canvas.getContext('2d');

const blur = {
  process(frame) {
    ctx.filter = 'blur(15px)';
    ctx.drawImage(frame, 0, 0);
    frame.close();
    return new VideoFrame(canvas, { timestamp: frame.timestamp });
  },
};

await VideoSDK.applyVideoProcessor(blur);

Watermark overlay

const canvas = new OffscreenCanvas(1280, 720);
const ctx = canvas.getContext('2d');
const logo = await loadImageAsBitmap('/assets/logo.png');

const watermark = {
  process(frame) {
    ctx.drawImage(frame, 0, 0);
    ctx.drawImage(logo, 20, 20, 100, 30);
    frame.close();
    return new VideoFrame(canvas, { timestamp: frame.timestamp });
  },
};

await VideoSDK.applyVideoProcessor(watermark);

Source-style — canvas as video (ignore camera input)

const sourceCanvas = document.createElement('canvas');
sourceCanvas.width = 1280; sourceCanvas.height = 720;
const sourceCtx = sourceCanvas.getContext('2d');
startAnimationLoop(sourceCtx);   // your own render loop

const canvasSource = {
  process(_inputFrame) {
    // Ignore camera input; return frame from our canvas
    _inputFrame.close();
    return new VideoFrame(sourceCanvas, { timestamp: performance.now() * 1000 });
  },
};

await VideoSDK.applyVideoProcessor(canvasSource);

VideoFrameProcessor interface

Blueprint

Members

process

When the SDK stops calling process

Examples

When the SDK stops calling `process`