A few weeks ago I launched Convert to Shorts — a free browser-based tool that converts horizontal videos to YouTube Shorts format (9:16) without uploading anything to a server. I wrote about the ffmpeg.wasm + Vite setup in a previous article.
The most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious.
The challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server?
The stack - Transformers.js (@xenova/transformers) — Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly Whisper tiny — OpenAI's speech recognition model, 75MB, surprisingly accurate for clear speech Web Audio API — for extracting and resampling audio from the video file ffmpeg.wasm — for burning captions into the video ASS subtitles — the subtitle format libass (inside ffmpeg.wasm) understands.
Whisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly:
Creating the AudioContext at 16kHz means the browser automatically resamples from whatever the source rate is (usually 44.1kHz or 48kHz). No manual resampling needed.
return_timestamps: true gives you back an array of segments with text and timestamp: [start, end] — exactly what you need for timed captions.
env.useBrowserCache = true means the model downloads once and is stored in IndexedDB. Every subsequent use loads from cache — no 75MB download each time.
This is where it got interesting. My first instinct was to use ffmpeg's drawtext filter with an enable='between(t,start,end)' expression for each caption segment — one filter per caption, chained with commas.
This failed in multiple ways: The comma inside between(t,start,end) was interpreted as a filter separator Escaping with \\, didn't work in ffmpeg.wasm's argument parsing Using gte(t,start)lte(t,end) instead of between also failed at the filter chain level Chaining 20+ drawtext filters caused ffmpeg.wasm to abort
The reliable solution was ASS subtitles. ASS (Advanced SubStation Alpha) is a subtitle format that ffmpeg's built-in ass filter handles natively via libass:
libass needs a font to render subtitles. In a normal environment it uses system fonts. In ffmpeg.wasm's WebAssembly sandbox there are no system fonts.
Approach 1: Embed font as Base64 in the ASS file The ASS format supports a [Fonts] section with Base64-encoded font data split into 80-character lines. This failed with a libass assertion error in ass.c about Base64 padding — the ffmpeg.wasm build of libass appears to have a bug in its font decoder.
Write the font file to /fonts/Roboto-Bold.ttf in the virtual filesystem before running ffmpeg. This worked.
The key insight: ffmpeg.wasm has a full virtual filesystem (Emscripten's FS). You can create directories and write files to it just like a real filesystem, and ffmpeg commands can reference those paths.
As a result the full caption pipeline added roughly 10-30 seconds to the export time for a 30-60 second clip. The Whisper tiny model is surprisingly accurate for clear English speech. Multiple languages work out of the box since Whisper was trained on multilingual data.
Transformers.js is genuinely production-ready. The API is clean, browser caching just works, and the ONNX runtime handles the WebAssembly execution reliably.
ASS subtitles are more robust than drawtext filter chains. If you're burning timed text into video with ffmpeg.wasm, reach for the ass filter before trying to chain multiple drawtext filters.