How We Got 60fps Video Playback in a Browser-Based Editor

How We Got 60fps Video Playback in a Browser-Based Editor

Velocaption is a desktop video editor built with Tauri + React. Two editor surfaces: a silence remover and a captions editor. Both play back multiple video and audio sources simultaneously.

For months, playback stuttered at every clip boundary. Three overlapping video overlays froze the UI for 200ms. The silence remover clicked audibly at every cut point.

We fixed it with 540 lines of imperative TypeScript that manages media elements outside of React.

Browsers destroy decoders on element removal

When you create a <video> and set src, the browser allocates a hardware decoder, opens a file handle, parses mp4 atoms, and buffers data. Remove that element from the DOM and all of it is gone. The next <video> starts from scratch.

A 2-minute recording with 15 silence cuts means 15 clip transitions. Each one was creating a new element, destroying the old one, and racing against the 33ms frame budget to initialize a new decoder.

Hardware decoder init takes 50 to 100ms. That math doesn’t work.

Three things were visibly broken:

Decoder storms. Adjacent silence segments triggered rapid element swaps. The playhead crosses from segment A to B, old element dies, new one spins up. At 30fps you have 33ms. Decoders need 80.

Audio clicks. Audio gaps are perceptible at sub-frame durations. 20ms of silence between segments is an audible click. Every silence cut had one.

Overlay stacking. The captions editor supports picture-in-picture and multi-camera. Three overlapping video overlays meant three simultaneous decoder teardown/init cycles at every boundary.

React’s lifecycle is the wrong model for media

<video> elements have internal state that doesn’t survive a mount/unmount cycle: buffered ranges, decoder bindings, playback position. When React unmounts a video component at a clip boundary and mounts a new one, that state is gone.

You can’t key your way out of this. You can’t memo around it. React’s model is create/destroy. Media elements need create-once-reuse-forever.

The pool

Maintain a bounded set of <video> and <audio> elements that are never destroyed during normal playback. At clip boundaries, swap the src attribute. When a clip is done, recycle the element into a bucket.

The pool is a plain class. No hooks, no context, no state. React components interact with it through leases.

interface MediaLease {
  leaseKey: string; // logical slot (overlay ID)
  sourceKey: string; // actual bytes (URL + trim window)
  channel: "video" | "audio";
  role: "active" | "incoming" | "lookahead";
  startTime: number;
  endTime: number;
  source: {
    url: string;
    trimStartSeconds?: number;
    trimEndSeconds?: number;
    volume?: number;
    muted?: boolean;
  };
}

pool.setClock({ time: 12.345, playbackState: "PLAYING", playbackRate: 1 });

private scheduleFlush() {
  if (this.flushScheduled) return;
  this.flushScheduled = true;
  queueMicrotask(() => {
    this.flushScheduled = false;
    this.flush();
  });
}

private flush() {
  for (const slot of this.slots.values()) {
    if (!slot.container || !slot.lease) continue;

    const { lease } = slot

    const signature = buildLeaseSignature(lease);

    // Grab or create element
    if (!slot.element) {
      slot.element = this.takeRecycledElement(lease.channel)
        ?? createMediaElement(lease.channel);
    }

    // Source changed? Swap it.
    if (slot.leaseSignature !== signature) {
      slot.revision += 1;
      slot.swapStartedAt = performance.now();
      this.configureSource(slot, lease);
      slot.leaseSignature = signature;
    }

    this.applyRuntimeProperties(slot, slot.element, lease);
    this.attachElement(slot.element, slot.container);
    this.syncSlot(slot, lease); // seek, play/pause
  }
}

private recycleElement(element: HTMLMediaElement, kind: MediaKind) {
  element.pause();
  element.muted = true;
  element.removeAttribute("src");
  element.load();
  element.onloadedmetadata = null;
  element.oncanplay = null;
  element.onplaying = null;
  element.onwaiting = null;
  element.onerror = null;
  element.onended = null;

  // Park off-screen
  this.host.appendChild(element);

  const bucket = this.recycled.get(kind);
  bucket.push(element);
  if (bucket.length > this.maxRecycled) {
    bucket.shift()?.remove();
  }
}

Per-frame volume without React

This is the interesting part.

Video overlays have fade envelopes. A 0.5s fade-in at 30fps is 15 frames of volume interpolation. The naive way:

// 60 re-renders/sec per overlay. 3 overlays = 180/sec.
const [frameTime, setFrameTime] = useState(0);
useEffect(() => {
  /* RAF loop updating frameTime */
}, []);
const volume = computeFadeVolume(frameTime, overlay);
return <video volume={volume} />;

The actual work is element.volume = 0.7. One DOM write. But React wraps it in: run component function, diff vDOM, commit. That overhead dominates.

The pool takes a callback instead:

// VideoOverlayRenderer.tsx
// Built once. Stable reference. Not called by React.
const computeVolume = useCallback(
  (clockTime: number) => {
    const totalFrames = Math.round((overlay.end - overlay.start) * fps);
    const { fadeIn, fadeOut } = clampFades(
      Math.round((overlay.fade?.fadeIn ?? 0) * fps),
      Math.round((overlay.fade?.fadeOut ?? 0) * fps),
      totalFrames,
    );
    const localTime = Math.max(0, clockTime - overlay.start) + trimStart;
    const frame = Math.round(localTime * fps);
    const fadeMult = computeFadeMult(frame, fadeIn, fadeOut, totalFrames);
    return Math.min(1, Math.max(0, overlay.volume * fadeMult));
  },
  [overlay.start, overlay.end, overlay.fade, overlay.volume, fps, trimStart],
);

<PoolVideo computeVolume={computeVolume} /* ... */ />;

The pool calls it during flush:

private applyRuntimeProperties(slot, element, lease) {
  if (slot.config?.computeVolume) {
    const v = slot.config.computeVolume(this.clock.time);
    element.volume = Math.min(1, Math.max(0, v));
    element.muted = v === 0;
    return;
  }
  // fallback chain: runtimeVolume → lease.source.volume
}

The volume priority chain:

Three overlapping fades cost three function calls per flush. Microseconds. The component itself renders once, when the overlay config changes.

Stale event rejection

Media events are async. oncanplay might fire 200ms after you set src. If you swapped the source again in the meantime, that event belongs to a dead source.

Every event handler captures the slot’s revision at bind time:

private bindMediaEvents(slot: SlotRecord) {
  const revision = slot.revision;

  const isStale = () => {
    if (slot.revision !== revision) {
      this.staleOperationCount += 1;
      return true;
    }
    return false;
  };

  element.oncanplay = () => {
    if (isStale()) return;
    if (slot.swapStartedAt !== null) {
      this.lastSwapLatencyMs = performance.now() - slot.swapStartedAt;
      slot.swapStartedAt = null;
    }
    slot.config?.onCanPlay?.(element);
  };
}

Source swap bumps revision. Old callbacks see a mismatch and bail. The pool counts how many stale events it caught.

Three metrics

To ensure this architecture actually worked in production, we needed observability. We track three core metrics:

interface MediaPoolSnapshot {
  slots: Array<{ slotKey; readyState; currentTime; isPlaying; isMuted }>;
  lastSwapLatencyMs: number | null; // src change → canplay. Healthy: <50ms
  handoffMissCount: number; // playhead hit startTime, readyState < 3
  staleOperationCount: number; // async events from a previous revision
}

handoffMissCount is the one that matters. Every increment is an audible click or visible frame drop. If it climbs, either the preload window is too small or the source file is too large for the decoder to warm up in time.

staleOperationCount surfaces race conditions you’d never catch in manual testing. A count of zero means every async event fired for the correct source revision.

The slot lifecycle, end to end

What this pattern generalizes to

The pool is specific to media elements, but the underlying technique applies to anything where:

1. The browser object is expensive to create (WebGL contexts, Web Audio nodes, WebSocket connections)
2. The object has internal state that doesn’t survive React’s mount/unmount
3. You need per-frame updates that would be wasteful as React re-renders

The shape is always the same: imperative manager class, lease-based contracts from React, queueMicrotask flush batching, revision-guarded async callbacks.

540 lines. No dependencies.