Playback would run for a fraction of a second — about 18 frames — and then freeze. No error, no exception, no rejected promise. The decoder simply went silent and our flush() waited forever for an output that never came. This is the story of a bug that hides in the lifetime semantics of a single object: the WebCodecs VideoFrame.
A VideoFrame is a borrowed library book, not a photo
The hardware video decoder is a tiny library with a fixed shelf of roughly 16 books — its internal frame pool. Each time it decodes a picture it hands you one book: a VideoFrame. Crucially, a VideoFrame is not a copy of the pixels. It is a borrowed handle to one shelf slot inside the decoder.
The library's rule is strict: you may read the book (upload it to the GPU), but you must return it by calling frame.close() so the slot frees up. If you keep books, the shelf empties. And with an empty shelf the librarian — the decoder — cannot make new books. It does not crash. It just stops.
Decoder's shelf (~16 slots):
[B][B][B][B][B][B][B][B][B][B][B][B][B][B][ ][ ]
every B = one VideoFrame you are still holding open
keep ~16 open -> shelf full -> decoder can't decode -> FREEZEThe bug: a cache that never gave the books back
Our FrameCache stored the raw VideoFrame and only closed it much later, on eviction. That sounds fine until you look at the numbers: the cache was sized at 30 entries, but a short clip only ever decodes around 14 frames. Eviction never fired. No frame was ever closed. The shelf filled to ~16, the decoder went silent, and the next flush() hung waiting for a slot that would never come back.
The breaking line was cache.put(frame) — the cache held the book itself. Everything downstream was correct; the ownership model was the defect.
The fix: photocopy the book, return the original immediately
createImageBitmap(frame) makes a photocopy on plain paper. The ImageBitmap is yours forever and costs the decoder nothing — it holds no pool slot. So the instant a frame arrives we copy it, hand the book straight back with frame.close(), and then cache the photocopy.
// onFrame fires from VideoDecoder.output for every decoded frame
private async _copyAndCache(frame: VideoFrame, index: number) {
// 1. Photocopy: an ImageBitmap holds no decoder pool slot
const bitmap = await createImageBitmap(frame)
// 2. Return the book NOW — the slot is free again immediately
frame.close()
// 3. Cache the copy; the cache is its sole owner
this.cache.put(index, bitmap)
}The result: the shelf is almost always empty, the decoder never starves, and playback is smooth. Both VideoFrame and ImageBitmap are valid TexImageSource, so the copy uploads to the GPU exactly like the original would have.
The knock-on simplification: one owner, one closer
Before the fix, VideoLayer.draw called frame.clone() before uploading, because two owners both believed they had to close the frame: the cache and the texture upload's finally block. Cloning was a smell that said nobody had agreed who owns this. Once the cache held a photocopy that only it owned, the clone disappeared and upload became a pure borrow. The whole pipeline collapsed to one sentence:
The FrameCache owns every cached frame and is the only thing that closes it. Everyone else borrows and never closes.
Because the cache holds plain-memory copies rather than GPU objects, it survives a WebGL context loss (driver reset, tab backgrounded, alt-tab). When the GPU comes back, the next render re-uploads from the surviving cache — no re-decode, no stutter. A photocopy is just as re-uploadable as the original book.
The lesson generalizes beyond video: any time you cache a borrowed, pool-backed resource, decide who owns it and who closes it before you store it. The instant two code paths both think they own it, you have either a leak or a freeze — you just have not hit it yet.
Building something with browser-native video?
Try the SDK, read the docs, or join the conversation.