Design a Browser Video Editor – Frontend System Design

We’ll walk through the frontend system design for a browser video editor like CapCut or Descript: import clips, arrange them on a multi-track timeline, preview at the playhead, and export an MP4 in the tab. You’ll see why the document is an edit decision list rather than pixels, why the playhead must not live in React state, how WebCodecs seeks from a keyframe, and which export path you propose.
This is not the Twitch or Netflix question. Those play one stream. It is also not Photoshop on the web. Photoshop undoes pixels with copy-on-write tiles. A video editor undoes clip ops: trim, split, move. The pixels stay on disk, or in OPFS, and you only decode the ones under the playhead.
Problem Statement
Design the frontend for a timeline video editor that runs in the browser, like CapCut or Descript.
Requirements:
- Import video, audio, and stills. Arrange them on stacked video and audio tracks
- Trim, split, ripple, and slip clips. Link or unlink a video clip from its audio
- A frame-accurate preview with a playhead, j/k/l transport, and scrubbing
- Waveforms on audio clips, filmstrip thumbs on video clips
- Undo and redo of edits, not of decoded frames
- Export an H.264 MP4 (or WebM) from the tab, with a server encode as a fallback for long 4K jobs
Describe the client architecture, the timeline model, the preview clock, and the export pipeline.
Interviewers are checking three things. Can you keep 60 fps preview without putting time in React? Can you virtualize a timeline that would otherwise be tens of thousands of pixels wide? And do you know that MediaRecorder of a canvas is a realtime capture, not an offline encode?
Clarifying Questions
| # | Question | Interviewer’s Answer |
|---|---|---|
| 1 | NLE (CapCut) or transcript editor (Descript)? | NLE. Captions are a track. No speech-to-text in scope. |
| 2 | How long is a large project? | 90 minutes, 12 tracks, a few thousand clips. Sources are 1080p30. |
| 3 | Effects and color? | Opacity, transform, volume, fades, and a dissolve. No 3D or LUT graph. |
| 4 | Multiplayer on one timeline? | No. Cloud save and versions. One editor at a time. |
| 5 | Which browsers? | Desktop Chromium first. Safari 17+ and Firefox with fallbacks. |
| 6 | Must export stay in the tab? | Yes for 1080p under ~20 minutes. Longer or 4K may go to a job queue. |
| 7 | Offline? | Keep working through a refresh. Media lives in OPFS after import. |
Question 2 decides virtualization and the decode cache. Question 3 keeps the compositor simple: a painter’s-algorithm stack, not a node graph. Question 6 is the export fork: WebCodecs in a worker for the common case, a server job when the tab would melt. Question 5 tells you VideoDecoder is the happy path and <video> is the degrade.
Requirements
Functional
- Import MP4, WebM, MOV (H.264 / VP9 / AV1 where decodable), WAV, MP3, AAC, PNG, JPEG, WebP.
- Video, audio, and caption tracks. Reorder, mute, lock, hide.
- Ripple and overwrite. Split at the playhead. Slip in/out without moving the clip.
- Linked A/V from one import. Unlink to slip independently.
- Play, pause, j/k/l, scrub. Preview matches the exported frame at the playhead.
- Waveform on audio. Filmstrip or poster on video, by zoom.
- Undo / redo grouped by gesture. Autosave the EDL.
- Export 1080p H.264 MP4 with progress and cancel.
Non-Functional
- 60 fps preview while playing and while dragging a clip, on a mid-range laptop, for a 90-minute EDL.
- Scrub to any frame in under ~100 ms if the GOP is already warm; a cold keyframe walk should stay under a few hundred ms.
- Memory bounded: never hold a second of decoded 1080p RGBA in RAM as a policy. Decoded frames are rented and closed.
- Timeline scroll and zoom stay at 60 fps with thousands of clips. Only the viewport plus overscan mounts.
- Keyboard-first transport. The canvas is not the only way to select a clip.
16.7 ms is the budget. A React render of the timeline on every tick spends it before a single VideoFrame is drawn.
Napkin Math
| Metric | Estimate | Calculation | So what? |
|---|---|---|---|
| One 1080p RGBA frame | 8.3 MB | 1920 × 1080 × 4 | A 30-frame cache is 249 MB. Close frames. |
| 1 s of decoded 1080p30 | ~249 MB | 30 × 8.3 MB | You cannot cache the movie. Cache GOPs encoded, frames decoded. |
| 10 min of H.264 1080p30 | ~150–300 MB | ~8 Mbps × 600 s | OPFS holds sources. The EDL is kilobytes. |
| 90 min × 50 px/s scroll | 270,000 px | 5400 s × 50 | A full-width DOM will jank. Window the strip. |
| 8,000 clips, 40 s in view | ~40 mounted | interval query + 1 viewport overscan | Virtualize on time and track, not content-visibility alone. |
| Waveform peaks, 10 min | ~192 KB | 1 bucket / 10 ms × min+max × Int8 | Peaks are cheap. PCM is not. Store LOD peaks. |
| MediaRecorder 4 min export | 4 min wall | capture at 1× | Wrong default. Offline encode can beat realtime. |
Architecture
The client has six pieces, and the important decision is which one owns time.
| Layer | Owns | Runs on | Talks to |
|---|---|---|---|
| UI Shell | Library, inspector, transport chrome, export dialog | React, main thread | Sends intents, reads derived state |
| Project Core | EDL, selection, undo, asset index | Main thread (typed store) | Timeline view, preview, autosave |
| Preview Engine | Clock, decoder pool, compositor | Worker + OffscreenCanvas | Reads EDL, presents frames |
| Audio Graph | Master clock, mix of active clips | AudioContext / AudioWorklet | Clock, export mix |
| Timeline View | Virtualizer, waveforms, playhead overlay | Main thread + clip canvases | Queries EDL by time window |
| Export Worker | Offline walk, encode, mux | Worker | Same decoders, then a muxer |
React never holds the playhead during playback. Transport calls clock.play() and clock.seek(). The compositor samples the clock on requestAnimationFrame. Chrome subscribes at about 10 Hz for the timecode and a CSS translateX on the playhead. The canvas does not read React, and React does not re-render on rAF.
“The document is the EDL. The preview is a function of (EDL, clock.now()). React renders the chrome around that function. If the playhead is in useState, I have already lost the frame budget.”
Put media bytes in the Origin Private File System after import. IndexedDB is fine for EDL JSON and peak indexes. It is a poor home for a 300 MB MP4.
Data Model
A video editor’s document is an edit decision list: tracks of clips that point at immutable assets. A clip says “from this source, take [inUs, outUs), put it on the timeline at startUs.” You never store decoded frames as the document. Two clips can share one asset. Trimming a clip does not rewrite the MP4.
type Us = number; // media timestamps match WebCodecs: microseconds
type TrackKind = "video" | "audio" | "caption";
interface Project {
id: string;
name: string;
fps: 24 | 30 | 60;
width: number;
height: number;
durationUs: Us;
trackIds: string[];
}
interface Track {
id: string;
kind: TrackKind;
index: number;
muted: boolean;
locked: boolean;
hidden: boolean;
clipIds: string[];
}
interface Clip {
id: string;
trackId: string;
assetId: string;
startUs: Us; // timeline position
inUs: Us; // source in
outUs: Us; // source out (exclusive)
volume: number; // 0..1
fadeInUs: Us;
fadeOutUs: Us;
transform?: { x: number; y: number; scale: number };
linkedClipId?: string; // A/V pair from one import
}
interface Asset {
id: string;
kind: "video" | "audio" | "image";
mime: string;
durationUs: Us;
width?: number;
height?: number;
sampleRate?: number;
blobId: string; // OPFS key
peaksId?: string;
thumbId?: string; // filmstrip sprite
}
Two derived values come up everywhere. A clip ends on the timeline at startUs + (outUs - inUs). The source time for timeline time t is inUs + (t - startUs). Compute them on read. Storing them invites drift after a trim.
Four choices are worth defending.
Microseconds, not frames, as the stored unit. WebCodecs timestamps are microseconds. Mixing 24 fps and 30 fps sources on a 30 fps sequence makes a frame index ambiguous. Convert to the sequence grid only at the compositor: frame = Math.round(tUs / 1e6 * fps).
Assets are immutable. Import writes a blob and a probe. Peaks and filmstrip thumbs are caches keyed by content hash. Two clips of hello.mp4 are two rows and one file.
Linked clips are a pointer, not a container. Import creates a video clip and an audio clip with linkedClipId both ways. A move updates both until the user unlinks. You do not need a group entity for the common case.
The clock is not a field on Project. Playback is session state. Autosave writes the EDL. Persisting the playhead would restore a playing tab that should open paused.
A dissolve is a durationUs seam between two clips. Keep it off the clip row so a trim does not rewrite a sibling.
Preview Clock versus React
This is the answer interviewers are waiting for. Playback time updates at the display rate, or at the audio rate. React reconciliation does not. If playheadUs is state, every mounted clip re-renders 60 times a second, the inspector re-renders, and the compositor misses vsync. You will feel it as a sticky playhead and dropping frames.
Treat the clock as an external store, the same way a game loop treats performance.now().
The preview clock is a small class outside React, built on one AudioContext. Its state is whether it is playing, an origin pair (timeline microseconds and audio time at the last play or seek), the playback rate, and two subscriber sets.
now()returns the origin when paused. When playing, it returns the origin plus audio time elapsed since the origin, times the rate, converted to microseconds.play()resumes a suspendedAudioContext(it must be created or resumed from a user gesture), records the audio origin, and starts a rAF loop.pause()freezes the origin at the currentnow(), stops the loop, and notifies everyone once.seek(t)resets both origins and notifies everyone once.- Two subscription tiers. Frame subscribers (the compositor) are called on every rAF. UI subscribers (timecode, playhead line) are called at most every 100ms.
Audio is the master clock. AudioContext.currentTime is a monotonic media clock. requestAnimationFrame is a display clock. If you advance the playhead from rAF alone, video looks fine and audio drifts, because the audio thread does not care about vsync. Drive now() from audio time. Use rAF only to sample that clock and present the video frame that belongs at that sample. This is the same rule hardware NLEs and <video> engines use.
React reads the clock through useSyncExternalStore(clock.subscribeUi, clock.now) for the timecode. The playhead line is a single absolutely positioned element. On a UI tick you set el.style.transform = translate3d(${tUs * pxPerUs - scrollX}px,0,0). You do not set state. You do not ask the virtualizer to reconcile.
A drag updates the EDL once per frame (coalesce autosave to 50 ms). The clip node moves with a transform. The rest of the track does not re-render if clip props stay stable.
WebCodecs Preview
A <video> element is the wrong primitive for an NLE. You would need one element per source, they do not seek frame-accurately on every browser, and you cannot composite two of them with a dissolve without drawing both to a canvas anyway. WebCodecs gives you VideoDecoder / AudioDecoder and VideoFrame. You own the timeline.
The pipeline is demux → decode → composite → close.
- Demux in a worker.
mp4box.js(or a similar ISO-BMFF parser) reads the OPFS blob, emitsEncodedVideoChunk/EncodedAudioChunkwith timestamps, and an index of sync samples (keyframes). Persist that index next to the asset so the second open does not reparse the whole file. - Configure
VideoDecoderwith the codec string and thedescriptionfrom theavcC/hvcC/av1Cbox. Hardware decode is used when the UA can. Ifconfigurerejects, fall back to a<video>+requestVideoFrameCallbackpath for that asset, or tell the user the codec is unsupported. - Decode on demand around the playhead. Keep a small pool: one decoder per active source, not per clip. Two clips that share an asset share a decoder if they do not both need different GOPs in the same frame (a dissolve on the same file does, a cut does not).
- Close every
VideoFrameyou do not present. An unclosed frame holds GPU memory. The compositor presents at most one output frame per vsync and closes the inputs.
A frame fetch for a target time is a short procedure in the decode worker. Look up the last sync sample at or before the target in the persisted index. Find that chunk in the demuxed list. Feed chunks to the decoder from there until one passes the target by about a millisecond. Then flush. The decoder’s output callback closes every frame earlier than the target and keeps the last one, which is the frame to present.
There is no VideoDecoder.seek(). A delta chunk is useless without every packet since the previous keyframe. Scrub to 12.04 s, find the sync sample at 10.00 s, decode forward, drop frames until timestamp >= 12.04s, present that one, close the rest. GOP length is why some files scrub like butter (1 s IDR) and some feel sticky (5–10 s).
Prefetch the next GOP while playing forward. Reverse jog still walks from the previous keyframe. Cache encoded chunks, not RGBA. Stills are one ImageBitmap, not a decoder.
Timeline Virtualization
A 90-minute project at 50 pixels per second is 270,000 pixels wide. Twelve tracks of that, with a DOM node per clip, will drop frames on scroll even if every clip is a empty div. Virtualize on two axes: time and track.
interface View {
scrollX: number;
scrollY: number;
width: number;
height: number;
pxPerUs: number;
trackHeight: number;
}
The timeline virtualizer derives a visible window from the view: a time range (scroll offset and width divided by pxPerUs, padded by the overscan) and a row range (scroll offset and height divided by track height, padded the same way). It walks only the tracks in the row range and keeps clips that overlap the time range. The output is the short list of clips to mount, usually a few dozen.
Keep clipIds sorted by startUs and binary-search the first candidate. An interval tree is for a caption dump, not a dozen clips.
Overscan is one viewport, not the rest of the project. Do not size the scrollport to 270,000 px if you can avoid it. Prefer a windowed strip: a 3-viewport inner layer whose translateX is rewritten as the user scrolls. Hit-testing maps client X through scrollX and pxPerUs. The scrollbar thumb is a custom overlay driven by durationUs * pxPerUs.
Zoom changes pxPerUs. Pinch toward the cursor and keep the time under the pointer fixed. At low zoom, draw the track as bars on one canvas. At medium zoom, mount clips wide enough to grab. At high zoom, show trim handles and the filmstrip.
The clip’s chrome is a pure function of its on-screen width, which is its duration times pxPerUs. Under about 8px it is a "bar" painted on the track canvas. Under about 48px it is a "clip" element you can grab. At 48px and wider it is "handles", with trim handles and the filmstrip.
Selection runs against the EDL, not the DOM. A click maps to time and track, then the topmost clip in range. Locked and hidden tracks are skipped, so an unmounted clip is still reachable from the keyboard.
Waveforms and Filmstrips
Do not draw PCM. A 10-minute 48 kHz mono file is 28.8 million samples. An SVG path of that will freeze the tab. Decode once after import, downsample to peak buckets, persist the peaks, and draw a canvas.
Peak extraction runs once per asset in a worker. Its input is mono PCM, the sample rate, and a bucket size in microseconds. Its output is a PeakLod: the bucket size plus two Int8Arrays holding the minimum and maximum sample of each bucket, scaled to −127 to 127. Eight bits per value is plenty for a waveform a few dozen pixels tall, and it makes a 10-minute file’s peaks a few kilobytes instead of 115 MB of floats.
Build three LODs (10 ms, 50 ms, 200 ms) after import, persist them, drop the PCM. At draw time pick the LOD whose bucket is about one pixel, then fillRect min/max per column. At 10 px/s a 200 ms bucket is enough; a breath between words uses 10 ms. Zoom does not rebuild anything.
Video filmstrips are the same idea: one poster per second (or per GOP) in an OPFS sprite. Low zoom is a tinted bar. Medium zoom samples the sprite. Thumbs must not block import.
Tracks, Mixing, and Compositing
Video tracks stack. Higher index paints later (on top). Hidden tracks skip. A still or a title is just a clip with a transform. The compositor, for output frame n at time t:
- Query video clips that contain
t, bottom track first. - For each, decode the source frame at the clip’s source time for
t. - Draw into the output canvas with the clip’s transform and opacity. If a dissolve is active, draw both sides and fade.
- Close input frames.
Audio tracks mix; they do not stack. An AudioWorklet is the mixer (128-sample quantum). Feed it a ring of PCM for each active clip, with volume and fades applied in the worklet. Mute is gain zero, so unmuting does not reopen a decoder mid-word. If the worklet runs dry, output silence and let video continue. Do not pause the clock for a late AAC packet.
Linked A/V move together. A ripple delete on video must ripple the linked audio by the same delta, or you drift. Say that out loud.
Undo of Clips, Not Pixels
Photoshop’s undo is expensive because the document is the pixels. Yours is not. A trim is three integers. History stores inverse EDL ops, grouped by gesture.
type HistoryOp =
| { type: "add"; clip: Clip }
| { type: "remove"; clip: Clip }
| { type: "patch"; clipId: string; before: Partial<Clip>; after: Partial<Clip> };
interface Gesture {
id: string;
label: string; // "Trim", "Ripple delete"
ops: HistoryOp[];
}
Inverting an op is mechanical. add becomes remove of the same clip, remove becomes add, and a patch swaps its before and after.
On pointer down, open a gesture. Coalesced moves replace the last patch on that clip so a drag is one op. On pointer up, push the gesture. Undo applies inverses in reverse. Never put a VideoFrame or a PNG on the stack. Those are caches. A trim undo changes inUs; the decoder walks it on the next tick.
Ripple and multi-clip moves are one gesture. Split is remove plus two adds. If the gesture moved linked video, include the audio patch so undo does not break the pair. Autosave the EDL on a 2 s debounce. Keep history in memory.
Export
There are three ways to get a file out of the tab. Name all three. Pick one as the default.
| Path | How | Wall clock | Use when |
|---|---|---|---|
| WebCodecs + JS muxer | Walk the EDL, VideoEncoder / AudioEncoder, mp4-muxer |
Faster than realtime on hardware encode | Default. 1080p MP4/WebM |
| MediaRecorder | canvas.captureStream() + AudioContext.createMediaStreamDestination() |
1× duration | Fallback. Quick prototype, odd UA gaps |
| ffmpeg.wasm | Ported ffmpeg, often threaded | Slow, RAM heavy | GIF, concat filters, a codec the encoder lacks |
Default: offline WebCodecs. The export worker reuses the same EDL walker and decoders as preview. For each output frame index it composites one VideoFrame, hands it to VideoEncoder, and closes it. Audio is mixed to PCM and fed to AudioEncoder. A JS muxer (mp4-muxer or webm-muxer) interleaves the encoded chunks into a file the user downloads. Progress is frame / totalFrames. Cancel aborts the encoder and throws away the partial blob.
The export worker owns a VideoEncoder whose output callback feeds the muxer. It configures H.264 (for example avc1.640028) at the project’s size and frame rate with a target bitrate. Then it loops over every output frame index. For each one it checks the abort signal, composites the frame at index / fps, encodes it (forcing a keyframe every two seconds so the file seeks well), and closes it. At the end it flushes the encoder, closes it, and asks the muxer for the final Blob. Abort closes the encoder and rejects with an AbortError, so the UI can tell cancel apart from failure.
MediaRecorder is a capture of the preview. Play at rate 1, pipe canvas.captureStream(fps) and a MediaStreamAudioDestinationNode into MediaRecorder, stop at duration. A 4-minute sequence takes 4 minutes. Chrome likes WebM/VP9; Safari is narrower. Use it when encode is missing, not as the product export.
ffmpeg.wasm is a format wrench. The fast build wants SharedArrayBuffer, so the document must be cross-origin isolated (COOP: same-origin + COEP: require-corp). That is the same trap as Photoshop on the web. Use it for a GIF palette or a codec you will not write. Do not encode a 90-minute 1080p timeline in it. Long 4K jobs POST the EDL to a server worker. Mention that hatch, then stay on the tab.
API
REST for documents and bytes. No socket. There is one editor, and preview is local.
// REST
GET /api/projects/:id // { project, tracks, clips, revision }
PUT /api/projects/:id // body: EDL snapshot, If-Match: revision
POST /api/assets // resumable upload, returns { id, hash }
GET /api/assets/:id // probe + signed GET for the blob
POST /api/exports // { projectId, preset } -> { jobId } // long jobs
GET /api/exports/:jobId // { status, progress, url? }
interface ExportPreset {
width: number;
height: number;
fps: 24 | 30 | 60;
videoBitrate: number;
audioBitrate: number;
container: "mp4" | "webm";
}
Autosave is a PUT of the EDL with If-Match. The body is kilobytes. A revision conflict (two tabs) makes the loser view-only. You do not need a CRDT. Uploads are resumable and content-addressed. OPFS is the working set; the network copy is backup and the server-encode input. POST /api/exports is only the 4K hatch. The job walks the same EDL schema.
Optimizations
| Problem | Technique | Effect |
|---|---|---|
| React rendering at 60 Hz | Clock module; UI subscribe at 10 Hz | Chrome updates, clips do not |
| Huge timeline DOM | Time + track virtualization, windowed strip | Tens of nodes, not thousands |
| Cold scrub on a long GOP | Sync-sample index + prefetch next GOP | Seek walks less, play stays warm |
| RGBA memory | Close VideoFrames; cache encoded chunks | GPU memory stays flat |
| Waveform of full PCM | LOD peaks in OPFS, canvas blit | Kilobytes, 60 fps draw |
| Export at 1× | Offline VideoEncoder, not MediaRecorder | Minutes become tens of seconds |
| Import hitch | Probe + placeholder, peaks/thumbs async | User can edit immediately |
| Decoder configure cost | Pool by asset id, reuse across clips | Cuts on one file stay hot |
Move demux, peaks, and export off the main thread. Transfer the preview with transferControlToOffscreen. If that is too far for the interview, start the compositor on the main thread and move it once the clock is isolated.
Handling Edge Cases
- Variable frame rate. Use each chunk’s timestamp. The output grid is regular. Duplicate or drop the nearest source frame.
- Unsupported codec. If
VideoDecoder.isConfigSupportedis false, reject or server-transcode to H.264 before preview. A clip that only works at export is a ticket. - A/V length mismatch. Phone MP4s often have longer audio. Each linked clip keeps its own duration. Do not stretch by default.
- Long GOP. A 10 s GOP means a cold seek decodes 10 s. Hold the last frame or nearest thumb so scrub never flashes black.
- Background tab. rAF pauses. Pause the clock, or keep audio with a worker export so hiding the tab does not freeze encode. Banner long exports.
- Refresh / two tabs. OPFS and the last EDL survive. In-flight gestures do not.
If-Matchmakes the losing tab view-only.
Accessibility and Security
Accessibility
The timeline is a custom widget. A screen reader cannot see the canvas filmstrip, and it should not hear the playhead 60 times a second.
- A list of tracks and clips (name, start, duration) is the accessible document. Arrows select.
[/]nudge a frame.I/Oset in/out. - Transport is labeled buttons. Space toggles play. Announce “paused at 00:12:04” in
aria-live="polite"on pause and seek end, never on every tick. - The waveform is decorative. Export uses
role="progressbar". Respectprefers-reduced-motionfor zoom, not for the playhead (that is time).
Security
- Imported files are untrusted bytes. Parse with the demuxer. Captions are escaped text, not HTML.
- Revoke object URLs. Validate EDL ids and ranges. Do not
evalproject JSON. - ffmpeg.wasm needs COOP/COEP. Third-party scripts need CORP. Keep ffmpeg on an isolated origin if OAuth popups would break.
- Signed asset URLs expire. Copy into OPFS so a stale cookie does not black-frame preview.
What a Strong Answer Sounds Like
| Topic | Junior answer | Senior answer |
|---|---|---|
| Preview | A <video> per clip, currentTime = playhead |
WebCodecs pool, GOP walk, frames closed every tick |
| Time | setPlayhead in React at 60 Hz |
Audio-master clock, rAF sample, React at 10 Hz |
| Timeline | Map every clip to a div | Interval query, overscan, windowed strip, LOD chrome |
| Undo | Snapshot the canvas or the file | Inverse EDL ops, one gesture, no pixels |
| Export | MediaRecorder of the preview | Offline VideoEncoder + muxer; Recorder fallback; ffmpeg for formats; server for 4K |
For the raster-editor cousin of this problem, see Design Photoshop on the Web. For a single-stream player, see Design Twitch.
Conclusion: How to Design a Browser Video Editor
Start with the document. The EDL is a few tables of clips that point at immutable assets. Preview is a pure function of that list and a clock the compositor samples. React draws the chrome and a virtualized window of clip chrome. It does not own time, and it does not own pixels.
Then pick the boring, correct export: walk the same EDL offline through WebCodecs and a muxer. Keep MediaRecorder as the capture fallback, and ffmpeg.wasm as the format wrench that costs you cross-origin isolation. If you can explain a keyframe walk, why audio is the master clock, and why undo is a trim inverse rather than a PNG, you have answered the question the interviewer actually asked.
