How We Built a Browser-Based Video AI with Zero Servers

The Challenge

Building a video AI tool that runs entirely in the browser seemed impossible at first. Traditional video processing requires server-side GPU instances. But we wanted something different: a tool where users' videos never leave their devices.

The Tech Stack

WebAssembly (WASM)

We compile FFmpeg and Whisper to WebAssembly, enabling near-native performance in the browser.

Web Workers

Heavy computation runs in separate threads, keeping the UI responsive.

IndexedDB

AI models (100MB+) are cached locally after first download.

WebGPU / WebGL

Hardware-accelerated video processing and AI inference.

Architecture

┌─────────────────────────────────────┐
│           Main Thread (React)       │
├─────────────────────────────────────┤
│           Web Worker 1              │
│    FFmpeg.wasm (Audio Extraction)   │
├─────────────────────────────────────┤
│           Web Worker 2              │
│    Whisper.wasm (Transcription)     │
├─────────────────────────────────────┤
│           Web Worker 3              │
│    AI Clip Suggestion Algorithm     │
├─────────────────────────────────────┤
│           IndexedDB                 │
│    Model Cache (Persistent)         │
└─────────────────────────────────────┘

Performance Results

**Audio extraction:** ~2x realtime on modern hardware

**Transcription:** ~3x realtime with Whisper Tiny

**Clip suggestion:** <1 second for 1-hour videos

Challenges We Faced

**SharedArrayBuffer** — Requires COOP/COEP headers

**Memory limits** — Large videos can cause OOM errors

**Browser compatibility** — Safari has limited WASM support

What's Next

We're exploring WebGPU for GPU-accelerated inference, which could bring transcription speeds closer to realtime on supported devices.