Running the model yourself
sounds free. Then you meet the walls.

WebLLM runs a ~3B-parameter model entirely in the page using WebGPU— the browser's low-level GPU API. When it works, nothing you query ever leaves the device, which is the strongest privacy story there is. The catch is that word when: WebGPU is a hard dependency, and Firefox is where that dependency breaks.

• No WebGPU, no WebLLM. Firefox only shipped WebGPU on Windows first; on macOS and Linux it's still gated behind about:config flags and isn't on by default. On those platforms WebLLM simply won't initialize — there's no fallback path.
• A multi-gigabyte first run. Even where it does run, the weights download the first time you use it and get cached — a multi-GB hit before the first answer, and again whenever the model is updated.
• It leans on your GPU. You need a reasonably capable GPU and enough VRAM to hold the model, plus a per-session warm-up while it loads onto the device. On weaker integrated graphics it's slow or won't load at all.

Chrome ships an on-device model — Gemini Nano — reachable through the built-in Prompt API. It's genuinely convenient: no server, no egress, and Chrome manages the download for you. But Nano is a a-few-billion-parameter model tuned to fit on a phone or laptop, and that size shows up as performance limits.

Point QDev at a local ollamaserver and you can run the strongest local models available — no egress, your pick of model. It's the power-user path, and the price is real hardware. As a rough rule, a quantized model needs about as much free RAM (or VRAM, for GPU speed) as its size on disk:

And that's just memory. You also install and keep a local server running, pull each model by hand, and pull again to update. On CPU-only machines the bigger models crawl; the models small enough to feel instant are the ones whose jq accuracy trails.