Advanced Use Cases

Audio Input with a Model Manifest (Experimental)

A custom MLC model directory can include mlc-model-manifest.json. The manifest describes the task, the audio format the model expects, the compiled function that embeds audio, the prompt tokens around it, and two hashes that must match the compiled library. WebLLM reads the manifest only when the model record sets model_manifest, so other models load as before. A manifest that is missing, invalid or does not match the library fails the load.

Weights are read from tensor-cache.json. WebLLM never asks WebGPU for a buffer over 1 GiB, so conversion has to split any weight larger than that.

To send audio, pass a base64 WAV or a WAV data URL as an input_audio content part:

const response = await engine.chat.completions.create({
  messages: [{
    role: "user",
    content: [
      {
        type: "input_audio",
        input_audio: { format: "wav", data: wavBase64 },
      },
      { type: "text", text: "What do you hear?" },
    ],
  }],
});

Callers that already have samples can pass mono PCM as a Float32Array and skip WAV encoding:

const inputAudio = {
  type: "input_audio" as const,
  input_audio: {
    format: "pcm_f32" as const,
    data: samples, // Float32Array
    sample_rate: 48000,
  },
};

WebLLM downmixes to mono and resamples to the rate the manifest asks for. Feature extraction happens in the compiled model.

This works with custom google/gemma-4-E2B-it q4f16_1 builds. There is no prebuilt model record yet. Audio URLs and compressed formats are not supported.

A manifest may declare an image input instead of or next to the audio one. It names the size the model expects, whether the image is stretched or scaled and center cropped to that size, the compiled function that embeds it, and the prompt tokens around it. Send an image as an image_url content part with an http URL or a data:image URL, the same part the prebuilt vision models take. WebLLM decodes and resizes the image, and the compiled model normalizes it. This works with custom llava-hf/llava-1.5-7b-hf builds. Models that pick a resolution per image, such as Phi-3.5-vision, keep loading without a manifest.

A model that keeps a recurrent state next to the KV cache, such as Qwen3.5, declares create_rnn_state in its program, and WebLLM passes both states to prefill and decode. The record needs the same max_history_size override as the prebuilt Qwen3.5 records.

model_manifest is a URL relative to the model URL:

const modelRecord = {
  model_id: "gemma-4-E2B-it-q4f16_1-MLC",
  model: modelUrl,
  model_lib: modelLibUrl,
  model_manifest: "mlc-model-manifest.json",
};

Using Workers

You can put the heavy computation in a worker script to optimize your application performance. To do so, you need to:

Create a handler in the worker thread that communicates with the frontend while handling the requests. Create a worker engine in your main application that sends messages to the handler in the worker thread under the hood. For detailed implementations of different kinds of workers, look at the following sections.

Using Web Workers

WebLLM comes with API support for Web Workers so you can offload the computation-heavy generation work into a separate worker thread. WebLLM has implemented cross-thread communication through messages under the hood, so manual implementation is not required.

In the worker script, import and instantiate a WebWorkerMLCEngineHandler, which handles communication with other scripts and processes incoming requests.

// worker.ts
import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg: MessageEvent) => {
    handler.onmessage(msg);
};

In the main script, import and instantiate a WebWorkerMLCEngine that implements the same MLCEngineInterface and exposes the same APIs. Then, simply use it as you would a normal MLCEngine.

import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm";

async function runWorker() {
    const engine = await CreateWebWorkerMLCEngine(
        new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }),
        "Llama-3.1-8B-Instruct"
    );

    const messages = [{ role: "user", content: "How does WebLLM use workers?" }];
    const reply = await engine.chat.completions.create({ messages });
    console.log(reply.choices[0].message.content);
}

runWorker();

Under the hood, WebWorkerMLCEngine does not perform any computation. It translates all calls into messages and sends them to the WebWorkerMLCEngineHandler for processing. The worker thread receives these messages and processes the actual computation using a hidden engine, and returns the result to the main thread using messages.

Service Workers

WebLLM also supports offloading computation using Service Workers. This allows you to avoid reloading the model between page refreshes and optimize your application’s offline experience.

(Note, the lifecycle of a Service Worker is managed by the browser and can be killed any time without notifying the web application. WebLLM’s ServiceWorkerMLCEngine attempts to keep the service worker thread alive by periodically sending heartbeat events. However, the script could still be killed at any time by Chrome, and your application should include proper error handling. Check keepAliveMs and missedHeartbeat in ServiceWorkerMLCEngine for more details.)

In the worker script, import and instantiate ServiceWorkerMLCEngineHandler, which handles communication with page scripts and processes incoming requests. Instantiate it at the top level so its message listener is registered during initial script evaluation. Do not instantiate it from an activate or message listener: the browser can restart an already-active worker without dispatching another activate event.

// sw.ts
import { ServiceWorkerMLCEngineHandler } from "@mlc-ai/web-llm";

new ServiceWorkerMLCEngineHandler();
console.log("Service Worker is ready!");

Then, in the main page script, register the service worker and instantiate the engine using the CreateServiceWorkerMLCEngine factory function that implements the same MLCEngineInterface and exposes the same APIs. Then, simply use it as you would a normal MLCEngine.

// main.ts
import { MLCEngineInterface, CreateServiceWorkerMLCEngine } from "@mlc-ai/web-llm";

if ("serviceWorker" in navigator) {
navigator.serviceWorker.register(
    new URL("sw.ts", import.meta.url),  // worker script
    { type: "module" },
);
}

const engine: MLCEngineInterface =
await CreateServiceWorkerMLCEngine(
    selectedModel,
    { initProgressCallback }, // engineConfig
);

Similar to the WebWorkerMLCEngine above, the ServiceWorkerMLCEngine is also a proxy and does not perform any actual computation. Instead, it forwards all calls to the service worker thread and receives the result through messages.

Chrome Extension

WebLLM can be used in Chrome extensions to empower local LLM inference. You can find examples of building Chrome extension using WebLLM in examples/chrome-extension and examples/chrome-extension-webgpu-service-worker. The latter leverages Service Worker, so the extension is persistent in the background.

Additionally, we have a full Chrome extension project, WebLLM Assistant, which leverages WebLLM to provide a personal web browsing copilot assistant experience. Feel free to check it out and contribute if you are interested.

Additional Customization

Using IndexedDB Cache

By default, WebLLM caches model artifacts using the Cache API for faster subsequent model loads. You can alternatively use IndexedDB caching by setting appConfig.cacheBackend = "indexeddb". When changing only the cache backend, preserve the prebuilt model list by spreading prebuiltAppConfig.

import { AppConfig, CreateMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm";

const appConfig: AppConfig = {
    ...prebuiltAppConfig,
    cacheBackend: "indexeddb",
};

const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct-q4f32_1-MLC", {
    appConfig,
});

Using Cross-Origin Storage Cache

WebLLM also supports caching model artifacts across different origins using the experimental Cross-Origin Storage API. You can enable this cache backend by setting appConfig.cacheBackend = "cross-origin". For users with the Cross-Origin Storage browser extension installed, resources will then be cached and shared across origins. This means two independent apps opted into this cache backend using the same AI model will download and cache the required resources only once. See the cache usage example for more details. If Cross-Origin Storage isn’t available, WebLLM will automatically fall back to using the default cache.

import { AppConfig, CreateMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm";

const appConfig: AppConfig = {
    ...prebuiltAppConfig,
    cacheBackend: "cross-origin",
};

const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct-q4f32_1-MLC", {
    appConfig,
});

Customizing Token Behavior

You can modify logit_bias in GenerationConfig to control token likelihood. Setting a token’s bias to a positive value increases its likelihood of being generated, while a negative value decreases it. A large negative value (e.g., -100) can effectively prevent the token from being generated.

const messages = [
    { role: "user", content: "Describe WebLLM in detail." },
];

const response = await engine.chatCompletion({
    messages,
    logit_bias: { "50256": -100 }, // Example: Prevent specific token generation
});