Skip to content
AI & Tools

Running LLMs in the Browser with WebGPU: Support, WebLLM, Transformers.js and Model Sizes

AIDToolStack EditorialNovember 7, 20267 min read
WebGPUWebLLMTransformers.jsLocal LLMsBrowser
Share:Twitter

Say you want a "summarise this note" button in an app where the notes are private enough that you'd rather not ship them to an API. Running a small model on the user's own GPU, inside the browser tab, is a real option now. It's also an option with sharp edges: not every browser has WebGPU, the first visit downloads the whole model before it can answer anything, and the models that fit are small. This post covers what's supported as of November 2026, the two libraries most people use (WebLLM and transformers.js), and what the model sizes actually look like.

If the model can run on a developer machine instead of the user's, our post on running local LLMs with Ollama, LM Studio and llama.cpp is the simpler path.

Who can actually run it

MDN marks the WebGPU API as "Limited availability": it isn't Baseline because it "does not work in some of the most widely-used browsers." It is also secure-context only, so pages must be served over HTTPS. The details come from MDN's browser-compat-data for GPU, and they're more fragmented than "supported in Chrome":

BrowserSinceCaveats in MDN's data
Chrome desktop113ChromeOS, macOS and Windows; Linux added in 144, Intel Gen12+ GPUs only
Edgemirrors Chrome
Chrome for Android121
Firefox desktop141partial: Windows from 141, macOS Tahoe on Apple silicon from 145, older macOS on Apple silicon from 147; no Intel Macs, no Linux, not in service workers
Firefox for Androidnot supported
Safari (macOS and iOS)26

So Firefox on Linux and Firefox for Android get no WebGPU at all, and Chrome on Linux only covers Intel Gen12+ GPUs. The transformers.js docs quote caniuse at "around 85%" global support as of March 2026, which is a fair ballpark but hides exactly the platform gaps above. Plan a fallback from day one: a server endpoint, a smaller WASM model, or simply hiding the feature.

Detection is cheap, so do it before you download anything:

async function getGpu() {
  if (!("gpu" in navigator)) return null;          // no WebGPU at all
  const adapter = await navigator.gpu.requestAdapter();
  if (!adapter) return null;                         // API exists, no usable GPU
  return {
    adapter,
    f16: adapter.features.has("shader-f16"),         // needed for q4f16 builds
  };
}

requestAdapter() can return null even when navigator.gpu exists, so check both, as MDN's own example does. MDN also documents a compatibility mode (requestAdapter({ featureLevel: "compatibility" })) that targets older graphics APIs such as OpenGL ES 3.1 and Direct3D 11 with a restricted subset of WebGPU. Whether a given ML library runs under it is a separate question; test before relying on it.

Two libraries, two different jobs

Architecture: the page talks to a Web Worker running WebLLM or transformers.js, which uses WebGPU on the GPU; weights are fetched once from a model host and kept in origin storage
Architecture: the page talks to a Web Worker running WebLLM or transformers.js, which uses WebGPU on the GPU; weights are fetched once from a model host and kept in origin storage
WebLLMtransformers.js
Package@mlc-ai/web-llm@huggingface/transformers
Latest on npm0.2.87, 30 Oct 20264.3.2, 27 Oct 2026
LicenceApache 2.0Apache 2.0
What it's forchat-style LLMs with an OpenAI-shaped APImany tasks: embeddings, speech, vision, text generation
Model formatMLC-compiled weights plus a WebGPU .wasm libraryONNX models from the Hugging Face Hub
Precision choicebaked into the model ID (q4f16_1, q4f32_1)dtype option: fp32, fp16, q8, q4, q4f16 and others

WebLLM gives you engine.chat.completions.create() with streaming, so code written against OpenAI's client mostly ports over. It supports JSON mode (our structured outputs post covers how hosted APIs handle the same job); function calling is marked work-in-progress with "preliminary support" for tools and tool_choice. It can run the engine in a Web Worker or a Service Worker, which matters because inference on the main thread will freeze your UI.

Transformers.js v4 shipped on 30 March 2026 with a new WebGPU runtime rewritten in C++, built with the ONNX Runtime team. On WebGPU it defaults to fp32, and on WASM to q8, per its README. It pulls ONNX models from the Hugging Face Hub, and it's the better pick when you need embeddings for client-side search, Whisper for transcription, or an image classifier, and a chat model is only part of the picture.

Licences for the libraries are permissive. The models are not all the same; check each model card before shipping its weights to users.

How big are the models, really

WebLLM publishes the VRAM it expects for every prebuilt model in its config.ts, along with a low_resource_required flag meaning the model "can run on limited devices (e.g. Android phone)." Here are the q4f16 builds of some common models:

Bar chart of WebLLM VRAM requirements for q4f16 builds, from 376 MB for SmolLM2-360M to 5,696 MB for Qwen3-8B
Bar chart of WebLLM VRAM requirements for q4f16 builds, from 376 MB for SmolLM2-360M to 5,696 MB for Qwen3-8B

What the chart tells you:

  • Sub-1 GB is possible but tiny. SmolLM2-360M at 376 MB and Llama-3.2-1B at 879 MB are both flagged as suitable for limited devices. I would use them for short rewriting and classification, not open-ended reasoning.
  • f16 saves real memory. Llama-3.2-1B needs 1,129 MB as q4f32_1 versus 879 MB as q4f16_1. The catch: 30 entries in the config list shader-f16 as a required feature, so check adapter.features.has("shader-f16") and fall back to the q4f32 build when it's missing.
  • Context length costs memory too. Most builds set context_window_size to 4096. The -1k variants cut it to 1024: Llama-3.1-8B q4f16 drops from 5,001 MB to 4,598 MB.
  • 8B is a desktop-GPU feature. The full-context 8B builds, around 5 to 5.7 GB, are flagged low_resource_required: false. Offer it as an opt-in "large model" for people who know their hardware.

VRAM isn't the only cost. The weights also land in browser storage. WebLLM uses the Cache API by default and can also use IndexedDB, OPFS or a Chrome cross-origin storage extension (the config notes the Cache API is "the most well-tested"). MDN's storage quota page says Chromium allows up to 60% of total disk per origin and Firefox's best-effort limit is the smaller of 10% of disk or 10 GiB. Best-effort data can be evicted under storage pressure, and Safari may delete script-created data for a site after seven days of browser use without interaction when tracking prevention is on. Call navigator.storage.persist() if you want the download to stick, and use navigator.storage.estimate() to check room before you start.

A minimal WebLLM setup that doesn't freeze the page

Put the engine in a worker. This is straight from the WebLLM README, trimmed:

// worker.ts
import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg: MessageEvent) => {
  handler.onmessage(msg);
};
// main.ts
import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateWebWorkerMLCEngine(
  new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }),
  "Llama-3.2-1B-Instruct-q4f16_1-MLC",
  { initProgressCallback: (p) => console.log(p) },
);

const chunks = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "Summarise the user's note in two sentences." },
    { role: "user", content: noteText },
  ],
  stream: true,
});

let reply = "";
for await (const chunk of chunks) {
  reply += chunk.choices[0]?.delta.content || "";
  render(reply);
}

Wire initProgressCallback to a real progress bar. On a first visit the user is downloading the whole model, and a spinner with no number reads as "broken."

For embeddings, transformers.js is a few lines. This example is from its WebGPU guide:

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline(
  "feature-extraction",
  "mixedbread-ai/mxbai-embed-xsmall-v1",
  { device: "webgpu" },
);
const embeddings = await extractor(["Hello world!"], { pooling: "mean", normalize: true });

For text generation, its dtype guide shows onnx-community/Qwen2.5-0.5B-Instruct with { dtype: "q4", device: "webgpu" }, and ModelRegistry.get_available_dtypes() tells you which quantisations a repo actually has before you pick one.

Gotchas worth checking before launch

  • WebGPU errors are asynchronous. MDN notes calls are "validated asynchronously in the GPU process," so a failure may show up later than the call that caused it. Wrap model loading in try/catch and listen for device loss.
  • Firefox flags in old docs. The transformers.js guide still tells Firefox users to flip dom.webgpu.enabled. MDN's data says Firefox 141 ships it on Windows. Trust the compat data and test the platforms you care about.
  • Service workers can be killed. WebLLM's README warns the browser can terminate a service worker at any time; the dedicated-worker setup above is the safer default.
  • Token counts still matter. A 4096-token window fills fast with a long note plus a system prompt. Our token counter helps you size prompts before you test on a phone.
  • Compare against the API bill. If the only reason for in-browser inference is cost, run your volume through the LLM pricing calculator first; a cheap hosted model can beat a 2 GB download on user experience.

When to reach for it

Use in-browser inference when the data shouldn't leave the device, when the task is small (summaries, classification, embeddings, transcription), and when you can tell users up front that the first run downloads a model. Use transformers.js if the job is anything other than chat; use WebLLM if you want an OpenAI-style chat API with streaming. Keep a server fallback for Firefox on Linux and Android, and keep the default model under about 1 GB unless the user opts in to something larger.

AE

Written by AIDToolStack Editorial

Researched from official documentation, specifications and vendor pages, listed under Sources below. Last checked November 2026.

Sources

  1. developer.mozilla.org/en-US/docs/Web/API/WebGPU_API
  2. github.com/mdn/browser-compat-data/blob/main/api/GPU.json
  3. caniuse.com/webgpu
  4. www.npmjs.com/package/@mlc-ai/web-llm
  5. www.npmjs.com/package/@huggingface/transformers
  6. github.com/mlc-ai/web-llm
  7. github.com/huggingface/transformers.js
  8. github.com/huggingface/transformers.js/releases/tag/4.0.0
  9. github.com/mlc-ai/web-llm/blob/main/src/config.ts
  10. developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria
  11. huggingface.co/docs/transformers.js/guides/webgpu
  12. huggingface.co/docs/transformers.js/guides/dtypes
  13. raw.githubusercontent.com/mdn/browser-compat-data/main/api/GPU.json

Related Posts

Explore on AIDToolStack

Discover tools and servers featured in our directory.

MCP Servers

Trigger.dev Jobs MCP ServerManage Trigger.dev background jobs from your AI coding assistant. Monitor job runs, trigger tasks manually, inspect execution logs, manage schedules, and debug failed runs. The server provides operational visibility into your background job infrastructure — see which tasks are running, queued, or failed, and take action. Your AI can help diagnose job failures from execution logs, retry failed tasks, and optimize concurrency configurations for better throughput.Tailscale Network MCP ServerManage your Tailscale mesh VPN network from your AI coding assistant. View connected devices, manage ACLs, configure exit nodes, check device status, and manage DNS settings. The server provides visibility into your Tailscale network — see which machines are online, troubleshoot connectivity issues, manage network access policies, and configure routes. Useful for developers managing distributed infrastructure where Tailscale provides secure networking between services.Portainer Docker MCP ServerManage Docker environments through Portainer's API from your AI coding assistant. Control containers, images, networks, and volumes across multiple Docker hosts and Kubernetes clusters. The server provides tools for container lifecycle management, image pulling and building, stack deployment, and resource monitoring. Portainer's multi-environment management makes this server particularly valuable for teams managing containers across development, staging, and production environments.

Reviews

Sign in to leave a review.

No reviews yet. Be the first!

Comments

Sign in to leave a comment.

No comments yet. Be the first!

Sponsor Our Newsletter

Reach thousands of developers who are actively evaluating AI tools, MCP servers, and dev infrastructure. Our weekly newsletter goes to engaged technical decision-makers.

Growing subscriber baseHigh open rates

All sponsored content is clearly labeled per our editorial policy.