Frontend System Design18 min readPremium

Design a Browser Video Editor – Frontend System Design

64. Design a Browser Video Editor

We’ll walk through the frontend system design for a browser video editor like CapCut or Descript: import clips, arrange them on a multi-track timeline, preview at the playhead, and export an MP4 in the tab. You’ll see why the document is an edit decision list rather than pixels, why the playhead must not live in React state, how WebCodecs seeks from a keyframe, and which export path you propose.

This is not the Twitch or Netflix question. Those play one stream. It is also not Photoshop on the web. Photoshop undoes pixels with copy-on-write tiles. A video editor undoes clip ops: trim, split, move. The pixels stay on disk, or in OPFS, and you only decode the ones under the playhead.

Problem Statement

Design the frontend for a timeline video editor that runs in the browser, like CapCut or Descript.

Requirements:

  • Import video, audio, and stills. Arrange them on stacked video and audio tracks
  • Trim, split, ripple, and slip clips. Link or unlink a video clip from its audio
  • A frame-accurate preview with a playhead, j/k/l transport, and scrubbing
  • Waveforms on audio clips, filmstrip thumbs on video clips
  • Undo and redo of edits, not of decoded frames
  • Export an H.264 MP4 (or WebM) from the tab, with a server encode as a fallback for long 4K jobs

Describe the client architecture, the timeline model, the preview clock, and the export pipeline.

Interviewers are checking three things. Can you keep 60 fps preview without putting time in React? Can you virtualize a timeline that would otherwise be tens of thousands of pixels wide? And do you know that MediaRecorder of a canvas is a realtime capture, not an offline encode?


Clarifying Questions

# Question Interviewer’s Answer
1 NLE (CapCut) or transcript editor (Descript)? NLE. Captions are a track. No speech-to-text in scope.
2 How long is a large project? 90 minutes, 12 tracks, a few thousand clips. Sources are 1080p30.
3 Effects and color? Opacity, transform, volume, fades, and a dissolve. No 3D or LUT graph.
4 Multiplayer on one timeline? No. Cloud save and versions. One editor at a time.
5 Which browsers? Desktop Chromium first. Safari 17+ and Firefox with fallbacks.
6 Must export stay in the tab? Yes for 1080p under ~20 minutes. Longer or 4K may go to a job queue.
7 Offline? Keep working through a refresh. Media lives in OPFS after import.

Question 2 decides virtualization and the decode cache. Question 3 keeps the compositor simple: a painter’s-algorithm stack, not a node graph. Question 6 is the export fork: WebCodecs in a worker for the common case, a server job when the tab would melt. Question 5 tells you VideoDecoder is the happy path and <video> is the degrade.


Requirements

Functional

  • Import MP4, WebM, MOV (H.264 / VP9 / AV1 where decodable), WAV, MP3, AAC, PNG, JPEG, WebP.
  • Video, audio, and caption tracks. Reorder, mute, lock, hide.
  • Ripple and overwrite. Split at the playhead. Slip in/out without moving the clip.
  • Linked A/V from one import. Unlink to slip independently.
  • Play, pause, j/k/l, scrub. Preview matches the exported frame at the playhead.
  • Waveform on audio. Filmstrip or poster on video, by zoom.
  • Undo / redo grouped by gesture. Autosave the EDL.
  • Export 1080p H.264 MP4 with progress and cancel.

Non-Functional

  • 60 fps preview while playing and while dragging a clip, on a mid-range laptop, for a 90-minute EDL.
  • Scrub to any frame in under ~100 ms if the GOP is already warm; a cold keyframe walk should stay under a few hundred ms.
  • Memory bounded: never hold a second of decoded 1080p RGBA in RAM as a policy. Decoded frames are rented and closed.
  • Timeline scroll and zoom stay at 60 fps with thousands of clips. Only the viewport plus overscan mounts.
  • Keyboard-first transport. The canvas is not the only way to select a clip.

16.7 ms is the budget. A React render of the timeline on every tick spends it before a single VideoFrame is drawn.

Napkin Math

Metric Estimate Calculation So what?
One 1080p RGBA frame 8.3 MB 1920 × 1080 × 4 A 30-frame cache is 249 MB. Close frames.
1 s of decoded 1080p30 ~249 MB 30 × 8.3 MB You cannot cache the movie. Cache GOPs encoded, frames decoded.
10 min of H.264 1080p30 ~150–300 MB ~8 Mbps × 600 s OPFS holds sources. The EDL is kilobytes.
90 min × 50 px/s scroll 270,000 px 5400 s × 50 A full-width DOM will jank. Window the strip.
8,000 clips, 40 s in view ~40 mounted interval query + 1 viewport overscan Virtualize on time and track, not content-visibility alone.
Waveform peaks, 10 min ~192 KB 1 bucket / 10 ms × min+max × Int8 Peaks are cheap. PCM is not. Store LOD peaks.
MediaRecorder 4 min export 4 min wall capture at 1× Wrong default. Offline encode can beat realtime.

Architecture

Browser video editor architecture: React UI shell, project store, preview engine with WebCodecs, virtualized timeline, audio graph, export worker, and servers

The client has six pieces, and the important decision is which one owns time.

Layer Owns Runs on Talks to
UI Shell Library, inspector, transport chrome, export dialog React, main thread Sends intents, reads derived state
Project Core EDL, selection, undo, asset index Main thread (typed store) Timeline view, preview, autosave
Preview Engine Clock, decoder pool, compositor Worker + OffscreenCanvas Reads EDL, presents frames
Audio Graph Master clock, mix of active clips AudioContext / AudioWorklet Clock, export mix
Timeline View Virtualizer, waveforms, playhead overlay Main thread + clip canvases Queries EDL by time window
Export Worker Offline walk, encode, mux Worker Same decoders, then a muxer

React never holds the playhead during playback. Transport calls clock.play() and clock.seek(). The compositor samples the clock on requestAnimationFrame. Chrome subscribes at about 10 Hz for the timecode and a CSS translateX on the playhead. The canvas does not read React, and React does not re-render on rAF.

“The document is the EDL. The preview is a function of (EDL, clock.now()). React renders the chrome around that function. If the playhead is in useState, I have already lost the frame budget.”

Put media bytes in the Origin Private File System after import. IndexedDB is fine for EDL JSON and peak indexes. It is a poor home for a 300 MB MP4.


Data Model

Browser video editor data model: project, track, clip, asset, history op, and ephemeral preview clock

A video editor’s document is an edit decision list: tracks of clips that point at immutable assets. A clip says “from this source, take [inUs, outUs), put it on the timeline at startUs.” You never store decoded frames as the document. Two clips can share one asset. Trimming a clip does not rewrite the MP4.


type Us = number; // media timestamps match WebCodecs: microseconds

type TrackKind = "video" | "audio" | "caption";

interface Project {
  id: string;
  name: string;
  fps: 24 | 30 | 60;
  width: number;
  height: number;
  durationUs: Us;
  trackIds: string[];
}

interface Track {
  id: string;
  kind: TrackKind;
  index: number;
  muted: boolean;
  locked: boolean;
  hidden: boolean;
  clipIds: string[];
}

interface Clip {
  id: string;
  trackId: string;
  assetId: string;
  startUs: Us;              // timeline position
  inUs: Us;                 // source in
  outUs: Us;                // source out (exclusive)
  volume: number;           // 0..1
  fadeInUs: Us;
  fadeOutUs: Us;
  transform?: { x: number; y: number; scale: number };
  linkedClipId?: string;    // A/V pair from one import
}

interface Asset {
  id: string;
  kind: "video" | "audio" | "image";
  mime: string;
  durationUs: Us;
  width?: number;
  height?: number;
  sampleRate?: number;
  blobId: string;           // OPFS key
  peaksId?: string;
  thumbId?: string;         // filmstrip sprite
}

Two derived values come up everywhere. A clip ends on the timeline at startUs + (outUs - inUs). The source time for timeline time t is inUs + (t - startUs). Compute them on read. Storing them invites drift after a trim.

Four choices are worth defending.

Microseconds, not frames, as the stored unit. WebCodecs timestamps are microseconds. Mixing 24 fps and 30 fps sources on a 30 fps sequence makes a frame index ambiguous. Convert to the sequence grid only at the compositor: frame = Math.round(tUs / 1e6 * fps).

Assets are immutable. Import writes a blob and a probe. Peaks and filmstrip thumbs are caches keyed by content hash. Two clips of hello.mp4 are two rows and one file.

Linked clips are a pointer, not a container. Import creates a video clip and an audio clip with linkedClipId both ways. A move updates both until the user unlinks. You do not need a group entity for the common case.

The clock is not a field on Project. Playback is session state. Autosave writes the EDL. Persisting the playhead would restore a playing tab that should open paused.

A dissolve is a durationUs seam between two clips. Keep it off the clip row so a trim does not rewrite a sibling.


Preview Clock versus React

Sequence of one playback frame: transport starts the clock, compositor decodes and presents, React gets a throttled timecode

This is the answer interviewers are waiting for. Playback time updates at the display rate, or at the audio rate. React reconciliation does not. If playheadUs is state, every mounted clip re-renders 60 times a second, the inspector re-renders, and the compositor misses vsync. You will feel it as a sticky playhead and dropping frames.

Treat the clock as an external store, the same way a game loop treats performance.now().

The preview clock is a small class outside React, built on one AudioContext. Its state is whether it is playing, an origin pair (timeline microseconds and audio time at the last play or seek), the playback rate, and two subscriber sets.

  • now() returns the origin when paused. When playing, it returns the origin plus audio time elapsed since the origin, times the rate, converted to microseconds.
  • play() resumes a suspended AudioContext (it must be created or resumed from a user gesture), records the audio origin, and starts a rAF loop.
  • pause() freezes the origin at the current now(), stops the loop, and notifies everyone once.
  • seek(t) resets both origins and notifies everyone once.
  • Two subscription tiers. Frame subscribers (the compositor) are called on every rAF. UI subscribers (timecode, playhead line) are called at most every 100ms.

Audio is the master clock. AudioContext.currentTime is a monotonic media clock. requestAnimationFrame is a display clock. If you advance the playhead from rAF alone, video looks fine and audio drifts, because the audio thread does not care about vsync. Drive now() from audio time. Use rAF only to sample that clock and present the video frame that belongs at that sample. This is the same rule hardware NLEs and <video> engines use.

React reads the clock through useSyncExternalStore(clock.subscribeUi, clock.now) for the timecode. The playhead line is a single absolutely positioned element. On a UI tick you set el.style.transform = translate3d(${tUs * pxPerUs - scrollX}px,0,0). You do not set state. You do not ask the virtualizer to reconcile.

A drag updates the EDL once per frame (coalesce autosave to 50 ms). The clip node moves with a transform. The rest of the track does not re-render if clip props stay stable.


WebCodecs Preview

A <video> element is the wrong primitive for an NLE. You would need one element per source, they do not seek frame-accurately on every browser, and you cannot composite two of them with a dissolve without drawing both to a canvas anyway. WebCodecs gives you VideoDecoder / AudioDecoder and VideoFrame. You own the timeline.

The pipeline is demux → decode → composite → close.

  1. Demux in a worker. mp4box.js (or a similar ISO-BMFF parser) reads the OPFS blob, emits EncodedVideoChunk / EncodedAudioChunk with timestamps, and an index of sync samples (keyframes). Persist that index next to the asset so the second open does not reparse the whole file.
  2. Configure VideoDecoder with the codec string and the description from the avcC / hvcC / av1C box. Hardware decode is used when the UA can. If configure rejects, fall back to a <video> + requestVideoFrameCallback path for that asset, or tell the user the codec is unsupported.
  3. Decode on demand around the playhead. Keep a small pool: one decoder per active source, not per clip. Two clips that share an asset share a decoder if they do not both need different GOPs in the same frame (a dissolve on the same file does, a cut does not).
  4. Close every VideoFrame you do not present. An unclosed frame holds GPU memory. The compositor presents at most one output frame per vsync and closes the inputs.

A frame fetch for a target time is a short procedure in the decode worker. Look up the last sync sample at or before the target in the persisted index. Find that chunk in the demuxed list. Feed chunks to the decoder from there until one passes the target by about a millisecond. Then flush. The decoder’s output callback closes every frame earlier than the target and keeps the last one, which is the frame to present.

There is no VideoDecoder.seek(). A delta chunk is useless without every packet since the previous keyframe. Scrub to 12.04 s, find the sync sample at 10.00 s, decode forward, drop frames until timestamp >= 12.04s, present that one, close the rest. GOP length is why some files scrub like butter (1 s IDR) and some feel sticky (5–10 s).

Prefetch the next GOP while playing forward. Reverse jog still walks from the previous keyframe. Cache encoded chunks, not RGBA. Stills are one ImageBitmap, not a decoder.


Timeline Virtualization

Virtualized timeline showing a viewport over a 120 second project, overscan ring, and unmounted clips outside the window

A 90-minute project at 50 pixels per second is 270,000 pixels wide. Twelve tracks of that, with a DOM node per clip, will drop frames on scroll even if every clip is a empty div. Virtualize on two axes: time and track.


interface View {
  scrollX: number;
  scrollY: number;
  width: number;
  height: number;
  pxPerUs: number;
  trackHeight: number;
}

The timeline virtualizer derives a visible window from the view: a time range (scroll offset and width divided by pxPerUs, padded by the overscan) and a row range (scroll offset and height divided by track height, padded the same way). It walks only the tracks in the row range and keeps clips that overlap the time range. The output is the short list of clips to mount, usually a few dozen.

Keep clipIds sorted by startUs and binary-search the first candidate. An interval tree is for a caption dump, not a dozen clips.

Overscan is one viewport, not the rest of the project. Do not size the scrollport to 270,000 px if you can avoid it. Prefer a windowed strip: a 3-viewport inner layer whose translateX is rewritten as the user scrolls. Hit-testing maps client X through scrollX and pxPerUs. The scrollbar thumb is a custom overlay driven by durationUs * pxPerUs.

Zoom changes pxPerUs. Pinch toward the cursor and keep the time under the pointer fixed. At low zoom, draw the track as bars on one canvas. At medium zoom, mount clips wide enough to grab. At high zoom, show trim handles and the filmstrip.

The clip’s chrome is a pure function of its on-screen width, which is its duration times pxPerUs. Under about 8px it is a "bar" painted on the track canvas. Under about 48px it is a "clip" element you can grab. At 48px and wider it is "handles", with trim handles and the filmstrip.

Selection runs against the EDL, not the DOM. A click maps to time and track, then the topmost clip in range. Locked and hidden tracks are skipped, so an unmounted clip is still reachable from the keyboard.


Waveforms and Filmstrips

Do not draw PCM. A 10-minute 48 kHz mono file is 28.8 million samples. An SVG path of that will freeze the tab. Decode once after import, downsample to peak buckets, persist the peaks, and draw a canvas.

Peak extraction runs once per asset in a worker. Its input is mono PCM, the sample rate, and a bucket size in microseconds. Its output is a PeakLod: the bucket size plus two Int8Arrays holding the minimum and maximum sample of each bucket, scaled to −127 to 127. Eight bits per value is plenty for a waveform a few dozen pixels tall, and it makes a 10-minute file’s peaks a few kilobytes instead of 115 MB of floats.

Build three LODs (10 ms, 50 ms, 200 ms) after import, persist them, drop the PCM. At draw time pick the LOD whose bucket is about one pixel, then fillRect min/max per column. At 10 px/s a 200 ms bucket is enough; a breath between words uses 10 ms. Zoom does not rebuild anything.

Video filmstrips are the same idea: one poster per second (or per GOP) in an OPFS sprite. Low zoom is a tinted bar. Medium zoom samples the sprite. Thumbs must not block import.


Tracks, Mixing, and Compositing

Video tracks stack. Higher index paints later (on top). Hidden tracks skip. A still or a title is just a clip with a transform. The compositor, for output frame n at time t:

  1. Query video clips that contain t, bottom track first.
  2. For each, decode the source frame at the clip’s source time for t.
  3. Draw into the output canvas with the clip’s transform and opacity. If a dissolve is active, draw both sides and fade.
  4. Close input frames.

Audio tracks mix; they do not stack. An AudioWorklet is the mixer (128-sample quantum). Feed it a ring of PCM for each active clip, with volume and fades applied in the worklet. Mute is gain zero, so unmuting does not reopen a decoder mid-word. If the worklet runs dry, output silence and let video continue. Do not pause the clock for a late AAC packet.

Linked A/V move together. A ripple delete on video must ripple the linked audio by the same delta, or you drift. Say that out loud.


Undo of Clips, Not Pixels

Photoshop’s undo is expensive because the document is the pixels. Yours is not. A trim is three integers. History stores inverse EDL ops, grouped by gesture.


type HistoryOp =
  | { type: "add"; clip: Clip }
  | { type: "remove"; clip: Clip }
  | { type: "patch"; clipId: string; before: Partial<Clip>; after: Partial<Clip> };

interface Gesture {
  id: string;
  label: string;          // "Trim", "Ripple delete"
  ops: HistoryOp[];
}

Inverting an op is mechanical. add becomes remove of the same clip, remove becomes add, and a patch swaps its before and after.

On pointer down, open a gesture. Coalesced moves replace the last patch on that clip so a drag is one op. On pointer up, push the gesture. Undo applies inverses in reverse. Never put a VideoFrame or a PNG on the stack. Those are caches. A trim undo changes inUs; the decoder walks it on the next tick.

Ripple and multi-clip moves are one gesture. Split is remove plus two adds. If the gesture moved linked video, include the audio patch so undo does not break the pair. Autosave the EDL on a 2 s debounce. Keep history in memory.


Export

Export pipeline comparing WebCodecs encoder plus JS muxer, MediaRecorder of the preview canvas, and ffmpeg.wasm

There are three ways to get a file out of the tab. Name all three. Pick one as the default.

Path How Wall clock Use when
WebCodecs + JS muxer Walk the EDL, VideoEncoder / AudioEncoder, mp4-muxer Faster than realtime on hardware encode Default. 1080p MP4/WebM
MediaRecorder canvas.captureStream() + AudioContext.createMediaStreamDestination() 1× duration Fallback. Quick prototype, odd UA gaps
ffmpeg.wasm Ported ffmpeg, often threaded Slow, RAM heavy GIF, concat filters, a codec the encoder lacks

Default: offline WebCodecs. The export worker reuses the same EDL walker and decoders as preview. For each output frame index it composites one VideoFrame, hands it to VideoEncoder, and closes it. Audio is mixed to PCM and fed to AudioEncoder. A JS muxer (mp4-muxer or webm-muxer) interleaves the encoded chunks into a file the user downloads. Progress is frame / totalFrames. Cancel aborts the encoder and throws away the partial blob.

The export worker owns a VideoEncoder whose output callback feeds the muxer. It configures H.264 (for example avc1.640028) at the project’s size and frame rate with a target bitrate. Then it loops over every output frame index. For each one it checks the abort signal, composites the frame at index / fps, encodes it (forcing a keyframe every two seconds so the file seeks well), and closes it. At the end it flushes the encoder, closes it, and asks the muxer for the final Blob. Abort closes the encoder and rejects with an AbortError, so the UI can tell cancel apart from failure.

MediaRecorder is a capture of the preview. Play at rate 1, pipe canvas.captureStream(fps) and a MediaStreamAudioDestinationNode into MediaRecorder, stop at duration. A 4-minute sequence takes 4 minutes. Chrome likes WebM/VP9; Safari is narrower. Use it when encode is missing, not as the product export.

ffmpeg.wasm is a format wrench. The fast build wants SharedArrayBuffer, so the document must be cross-origin isolated (COOP: same-origin + COEP: require-corp). That is the same trap as Photoshop on the web. Use it for a GIF palette or a codec you will not write. Do not encode a 90-minute 1080p timeline in it. Long 4K jobs POST the EDL to a server worker. Mention that hatch, then stay on the tab.


API

REST for documents and bytes. No socket. There is one editor, and preview is local.


// REST
GET    /api/projects/:id                 // { project, tracks, clips, revision }
PUT    /api/projects/:id                 // body: EDL snapshot, If-Match: revision
POST   /api/assets                       // resumable upload, returns { id, hash }
GET    /api/assets/:id                   // probe + signed GET for the blob
POST   /api/exports                      // { projectId, preset } -> { jobId }  // long jobs
GET    /api/exports/:jobId               // { status, progress, url? }

interface ExportPreset {
  width: number;
  height: number;
  fps: 24 | 30 | 60;
  videoBitrate: number;
  audioBitrate: number;
  container: "mp4" | "webm";
}

Autosave is a PUT of the EDL with If-Match. The body is kilobytes. A revision conflict (two tabs) makes the loser view-only. You do not need a CRDT. Uploads are resumable and content-addressed. OPFS is the working set; the network copy is backup and the server-encode input. POST /api/exports is only the 4K hatch. The job walks the same EDL schema.


Optimizations

Problem Technique Effect
React rendering at 60 Hz Clock module; UI subscribe at 10 Hz Chrome updates, clips do not
Huge timeline DOM Time + track virtualization, windowed strip Tens of nodes, not thousands
Cold scrub on a long GOP Sync-sample index + prefetch next GOP Seek walks less, play stays warm
RGBA memory Close VideoFrames; cache encoded chunks GPU memory stays flat
Waveform of full PCM LOD peaks in OPFS, canvas blit Kilobytes, 60 fps draw
Export at 1× Offline VideoEncoder, not MediaRecorder Minutes become tens of seconds
Import hitch Probe + placeholder, peaks/thumbs async User can edit immediately
Decoder configure cost Pool by asset id, reuse across clips Cuts on one file stay hot

Move demux, peaks, and export off the main thread. Transfer the preview with transferControlToOffscreen. If that is too far for the interview, start the compositor on the main thread and move it once the clock is isolated.


Handling Edge Cases

  • Variable frame rate. Use each chunk’s timestamp. The output grid is regular. Duplicate or drop the nearest source frame.
  • Unsupported codec. If VideoDecoder.isConfigSupported is false, reject or server-transcode to H.264 before preview. A clip that only works at export is a ticket.
  • A/V length mismatch. Phone MP4s often have longer audio. Each linked clip keeps its own duration. Do not stretch by default.
  • Long GOP. A 10 s GOP means a cold seek decodes 10 s. Hold the last frame or nearest thumb so scrub never flashes black.
  • Background tab. rAF pauses. Pause the clock, or keep audio with a worker export so hiding the tab does not freeze encode. Banner long exports.
  • Refresh / two tabs. OPFS and the last EDL survive. In-flight gestures do not. If-Match makes the losing tab view-only.

Accessibility and Security

Accessibility

The timeline is a custom widget. A screen reader cannot see the canvas filmstrip, and it should not hear the playhead 60 times a second.

  • A list of tracks and clips (name, start, duration) is the accessible document. Arrows select. [ / ] nudge a frame. I / O set in/out.
  • Transport is labeled buttons. Space toggles play. Announce “paused at 00:12:04” in aria-live="polite" on pause and seek end, never on every tick.
  • The waveform is decorative. Export uses role="progressbar". Respect prefers-reduced-motion for zoom, not for the playhead (that is time).

Security

  • Imported files are untrusted bytes. Parse with the demuxer. Captions are escaped text, not HTML.
  • Revoke object URLs. Validate EDL ids and ranges. Do not eval project JSON.
  • ffmpeg.wasm needs COOP/COEP. Third-party scripts need CORP. Keep ffmpeg on an isolated origin if OAuth popups would break.
  • Signed asset URLs expire. Copy into OPFS so a stale cookie does not black-frame preview.

What a Strong Answer Sounds Like

Topic Junior answer Senior answer
Preview A <video> per clip, currentTime = playhead WebCodecs pool, GOP walk, frames closed every tick
Time setPlayhead in React at 60 Hz Audio-master clock, rAF sample, React at 10 Hz
Timeline Map every clip to a div Interval query, overscan, windowed strip, LOD chrome
Undo Snapshot the canvas or the file Inverse EDL ops, one gesture, no pixels
Export MediaRecorder of the preview Offline VideoEncoder + muxer; Recorder fallback; ffmpeg for formats; server for 4K

For the raster-editor cousin of this problem, see Design Photoshop on the Web. For a single-stream player, see Design Twitch.


Conclusion: How to Design a Browser Video Editor

Start with the document. The EDL is a few tables of clips that point at immutable assets. Preview is a pure function of that list and a clock the compositor samples. React draws the chrome and a virtualized window of clip chrome. It does not own time, and it does not own pixels.

Then pick the boring, correct export: walk the same EDL offline through WebCodecs and a muxer. Keep MediaRecorder as the capture fallback, and ffmpeg.wasm as the format wrench that costs you cross-origin isolation. If you can explain a keyframe walk, why audio is the master clock, and why undo is a trim inverse rather than a PNG, you have answered the question the interviewer actually asked.