WebGPU bridge for browser inference

On the web, llamadart runs GGUF models through an external JavaScript bridge that wraps llama.cpp. The bridge uses WebGPU when the browser and device support it and runs on the WebAssembly CPU path otherwise. .litertlm models use @litert-lm/core instead; see Support matrix.

Invalid WebGPU request arguments throw LlamaArgumentException; scoreNextToken vocabulary and int32 bounds retain RangeError. Unsupported capabilities throw LlamaUnsupportedException. Runtime generation, embedding and scoring failures throw LlamaInferenceException; LoRA fetch failures remain LlamaModelException.

device: ComputeDevice.gpu requires a WebGPU adapter and active GPU layers after load. The bridge ignores negative mainGpu values, including with splitMode: ModelSplitMode.none, and keeps the default GPU index; use device: ComputeDevice.cpu to request the WebAssembly CPU runtime.

Requirements#

• Secure context: serve from https://, http://localhost or http://127.0.0.1. WebGPU is unavailable on other insecure origins.

• Cross-origin isolation: send these headers from the app origin so the bridge can run worker threads:

  Copy

  Cross-Origin-Opener-Policy: same-origin
  Cross-Origin-Embedder-Policy: require-corp

  require-corp is the portable choice. Cross-Origin-Embedder-Policy: credentialless isolates the page in Chromium-based browsers only: Safari 27 reports crossOriginIsolated === false with it and true with require-corp (#1025). Without isolation (window.crossOriginIsolated === false) the bridge caps inference at one thread and records threads_capped_no_coi in its runtime notes.

A supported browser version does not guarantee that a GGUF loads with WebGPU offload; the GPU, driver, OS, flags and memory pressure all matter.

Add the bridge to your app#

llamadart does not inject the bridge script. The app must load it in web/index.html before the first model load. Serve the bridge assets from the app origin: the bridge core starts its worker threads from its own URL, and a cross-origin isolated page cannot start workers from a CDN URL.

From the app root, run:

dart run llamadart:web_setup

It downloads the bridge assets your llamadart version pins (Pinned bridge assets) into web/webgpu_bridge/, checks every file against the release manifest whose SHA-256 the package pins, and adds a marked bootstrap block to web/index.html before flutter_bootstrap.js. Files that already match are kept and an up-to-date index.html is left alone, so rerun it after each llamadart upgrade. Nothing is replaced unless it verifies.

Switching from a hand-written loader such as the manual setup snippet: remove that <script> block first. The tool refuses a page with a <script> whose src or inline import names llama_webgpu_bridge.js, or that assigns LlamaWebGpuBridge (also as window.LlamaWebGpuBridge or window["LlamaWebGpuBridge"]) outside its own block, and quotes the line, because a second loader loads the bridge twice and a failing one makes model loads fail. A <link rel="modulepreload"> or plain text mention is not a loader. It cannot see a loader inside an external JavaScript file, so remove such a loader yourself. To keep your own loader, pass --no-index-html.

The tool itself makes no network requests for --check, but dart run may first run llamadart's native build hooks, which download the native runtimes for the host into the pub cache when it is cold.

It exits 64 for a usage error, 66 when the web directory or index.html is missing, and 1 for a failed download, a checksum mismatch, a page that already loads the bridge, or a page with neither a flutter_bootstrap.js script outside a comment nor </head>. Nothing is downloaded when index.html cannot be patched. If the bridge module then fails to load in the browser, the generated bootstrap sets __llamadartBridgeLoadError, so the first model load throws LlamaUnsupportedException at once instead of waiting.

Manual setup#

Without the tool, download the assets into web/webgpu_bridge/, with TAG set to the tag in Pinned bridge assets:

TAG=vX.Y.Z
mkdir -p web/webgpu_bridge
for f in llama_webgpu_bridge.js llama_webgpu_bridge_worker.js \
         llama_webgpu_core.js llama_webgpu_core.wasm \
         llama_webgpu_core_mem64.js llama_webgpu_core_mem64.wasm; do
  curl -fL -o "web/webgpu_bridge/$f" \
    "https://cdn.jsdelivr.net/gh/leehack/llama-web-bridge-assets@$TAG/$f"
done

This loop does not verify checksums. In a llamadart checkout, scripts/fetch_webgpu_bridge_assets.sh with WEBGPU_BRIDGE_OUT_DIR=<app>/web/webgpu_bridge does the same and verifies them.

Then load the bridge in web/index.html, before flutter_bootstrap.js:

<script type="module">
  try {
    const base = new URL('webgpu_bridge/', document.baseURI);
    window.__llamadartBridgeCoreModuleUrlMem64 =
      new URL('llama_webgpu_core_mem64.js', base).href;
    window.__llamadartBridgeSpeechToTextSupported = true;
    const mod = await import(new URL('llama_webgpu_bridge.js', base).href);
    window.__llamadartBridgeAdaptiveSafariGpu =
      mod.LlamaWebGpuBridge.supportsSafariAdaptiveGpu === true;
    window.LlamaWebGpuBridge = mod.LlamaWebGpuBridge;
  } catch (error) {
    window.__llamadartBridgeLoadError = String(error);
  }
</script>
<script src="flutter_bootstrap.js" async></script>

• __llamadartBridgeCoreModuleUrlMem64 enables the memory64 core; without it only the 32-bit core loads.
• __llamadartBridgeSpeechToTextSupported opts into Qwen3-ASR; set it only for assets that include Qwen3-ASR, such as the pinned ones.
• __llamadartBridgeAdaptiveSafariGpu lets Safari keep GPU layers when the assets support the adaptive probe.

At model load, llamadart waits up to 12 seconds for window.LlamaWebGpuBridge, and stops early once __llamadartBridgeLoadError is set. If the bridge never appears, the load throws LlamaUnsupportedException whose message contains Web bridge is unavailable. Ensure LlamaWebGpuBridge assets are loaded and reachable. or Web bridge is unavailable: <load error>.

example/chat_app/web/index.html is a fuller bootstrap: CDN-first loading with local fallback and a readiness promise. A cross-origin isolated page cannot start bridge workers from a CDN, so serve the assets from your app origin; see doc/webgpu_bridge.md.

Check readiness#

Paste this into the browser console of the running app:

const adapter = await navigator.gpu?.requestAdapter();
console.table({
  secureContext: window.isSecureContext,
  crossOriginIsolated: window.crossOriginIsolated,
  hasAdapter: !!adapter,
  adapterFeatures: adapter ? [...adapter.features].join(', ') : '',
  bridgeLoaded: typeof window.LlamaWebGpuBridge === 'function',
  bridgeLoadError: window.__llamadartBridgeLoadError || '',
  mem64CoreUrl: window.__llamadartBridgeCoreModuleUrlMem64 || '',
  workerFallbackReason: window.__llamadartBridgeWorkerFallbackReason || '',
});

The page is ready when it is a secure context, bridgeLoaded is true, bridgeLoadError is empty, and an adapter exists. Without an adapter, load with gpuLayers: 0 or switch browsers. Large single-file models also need crossOriginIsolated. For a first load, use a small quantized GGUF, contextSize of 2048 or less, and gpuLayers: 0 to prove CPU loading before raising GPU offload.

Model size and memory64#

The 32-bit core has a 4 GiB address space, which must also hold the KV cache and intermediate buffers. llamadart selects the 64-bit (memory64) core when ModelParams.preferMemory64 is true, or when it is null and ModelParams.modelBytesHint is at least 2 GiB. false selects the 32-bit core, though a load that aborts or runs out of memory on it is retried on memory64. Otherwise, the pinned bridge starts on memory64 whenever __llamadartBridgeCoreModuleUrlMem64 is set, as in the snippet above; set window.__llamadartBridgePreferMemory64 = false to start on the 32-bit core instead. Pass the model size when you know it:

await engine.setModel(
  LlamaModel(ModelSource.parse(modelUrl)),
  params: const ModelParams(
    modelBytesHint: 3043927168,
    contextSize: 2048,
  ),
);

Pass the size up front: the retry from wasm32 to wasm64 after an out-of-memory failure is slower and best-effort. Both fields apply only on the web and need the memory64 core URL from Add the bridge to your app. Qwen3-TTS needs memory64.

What differs from native#

The feature-by-runtime table is in the support matrix. On WebGPU:

• Bridge assets with supportsCompletionUsage report length when generation reaches a token, context or media cap. create preserves that finish reason, and sendWithTools returns truncated and rolls back the turn. Cancellation and normal stops retain their existing outcomes. Older assets or a missing termination signal cannot establish truncation. The bridge does not distinguish which cap was reached.
• Lazy grammars and custom grammarRoot require the loaded core's lazyGrammar probe (v0.1.75+ assets). Supported runtimes forward grammarLazy, word/pattern/pattern-full/token triggers and the root. Otherwise explicit lazy/custom-root requests throw LlamaUnsupportedException, and ordinary ToolChoice.auto retains best-effort parsing without its lazy grammar (Tool calling).
• Speculative decoding needs bridge assets whose getCompletionCapabilities() reports speculativeDecoding strategies: llama-web-bridge#153. A strategy the loaded assets do not report throws LlamaUnsupportedException. With such assets:
  • Each strategy runs llama.cpp's --spec-type of the same name, validated as on native. backendDefault and speculativeDecoding: true run ngram-mod, as on native.
  • draftModel is a ModelSource whose URL the runtime fetches; a ModelSource.path is a URL relative to the document, or a blob: URL. The bridge holds one draft model: the first generation that needs it loads it, through Cache Storage unless the URL carries credentials, and it stays loaded until another draft replaces it or a model loads. Cancelling the generation cancels the load. A draft that cannot run the requested strategy, or an EAGLE3 or DFlash draft built for another hidden size, is not kept and throws LlamaUnsupportedException; one the bridge cannot fetch or load throws LlamaModelException. A draft context that does not fit, such as an EAGLE3 draft at nBatch/nUbatch 2048 on wasm32, throws LlamaUnsupportedException naming smaller batches or memory64, and the target model stays usable (llama-web-bridge#198).
  • mtp runs the model's own MTP layers, so load the model with ModelParams(loadMtp: true); an MTP draftModel throws LlamaUnsupportedException.
  • The n-gram cache paths are URLs, sent only with ngram-cache, as native uses them only there. The bridge never writes the dynamic cache back.
  • A model with recurrent state, such as Qwen3.5, needs ModelParams.speculativeRollbackTokenMax of at least the draft length for mtp and the draft-model strategies. N-gram strategies need none: the bridge replays accepted tokens after a rejection.
  • N-gram sizes above 65535 throw LlamaArgumentException.
  • The draft and acceptance counts are not reported: LlamaEngine.getPerformanceContext() returns null on WebGPU.
• presencePenalty, minP and thinkingBudget need bridge assets whose getCompletionCapabilities() reports them; runtime LoRA (setLoraSource, removeLoraSource, clearLoras) and ModelParams.loras need assets whose getLoraAdapterCapabilities() reports support. Where the probes do not report them, a non-zero presencePenalty or minP, any thinkingBudget, every LoRA call and a load with ModelParams.loras throw LlamaUnsupportedException. LlamaEngine.capabilities reports the completion capabilities of the loaded assets. The capabilities come from llama-web-bridge#140, #144 and #142.
• Repeat and presence penalties count a text prompt without a grammar (the probe reports penaltyPromptTokens), as native llama.cpp does. ToolChoice.auto requests that skip the lazy tool-call grammar keep the prompt out, as native does under that grammar (llama-web-bridge#187). A thinking budget is text-only. A LoRA adapter is a ModelSource whose URL the bridge downloads once per model load (a ModelSource.path is a URL relative to the document, or a blob: URL), through Cache Storage unless the URL carries credentials; reload adapters after loading another model. An aLoRA adapter throws LlamaUnsupportedException, and an adapter the bridge cannot load, such as one made for another base model, throws LlamaModelException.
• A stop sequence equal to a preservedTokens entry is ignored, as on native llama.cpp.
• State files live in the bridge's WASMFS virtual filesystem and do not survive a page reload. Loading a missing state file throws LlamaStateException and the runtime stays usable; a truncated state snapshot rejects too and the current context state is kept.
• A character split across tokens, including at the start of a response, streams once complete instead of as a wrong character, and detokenize returns U+FFFD for invalid bytes or an incomplete tail instead of wrong characters. A tail still incomplete when generation ends becomes U+FFFD, detokenize keeps a leading U+FEFF, and output after a NUL byte is delivered (llama-web-bridge#177).
• In worker mode, cancel() right after a call starts stops it instead of letting it run to the end, and a bare cancel() streams the pieces generated before the worker yields, so they add up to the returned text (llama-web-bridge#179, #181).
• Text containing U+0000 is not truncated at the NUL in tokenize, embed, score and prompts, and speech text containing U+0000 is rejected (llama-web-bridge#184, #190).
• embed and embedBatch on a rank-pooled reranker throw LlamaUnsupportedException, and input that exceeds the micro-batch where it must fit one pass throws LlamaEmbeddingInputTooLongException, as on native (llama-web-bridge#194).
• The runtime fetches the model and projector itself. ModelSource.parse takes a remote https:// URL; a ModelSource.path is a URL relative to the document, or a blob: URL. download must stay at its defaults, and onProgress reports a fraction from 0 to 1: with a projector, the model fetch fills the first half and progress reaches 1 once the projector has loaded. An uncached model's progress follows its download, and a cancelled or failed load leaves nothing in Cache Storage (llama-web-bridge#195).
• LlamaEngine.scoreNextToken(...) needs assets that support it; others report supportsNextTokenScoring == false.

Fallback behavior#

Before failing a load, the web backend retries with safer settings:

• If GPU layers were requested, it retries on CPU (nGpuLayers = 0), first at the same context size, then at smaller ones.
• It steps the context size down through bounded candidates when the browser cannot fit the requested context.
• Qwen3.5-0.8B WebGPU loads are capped to a small GPU-layer count unless CPU is requested.
• With older Safari bridge assets, it forces CPU unless the assets support the adaptive Safari GPU probe or window.__llamadartAllowSafariWebGpu = true.
• It retries on the other core (wasm32 or wasm64) when bridge metadata points to an interop or memory-pressure failure. A large wasm32 staging abort is treated as memory pressure and retried on wasm64 when available.
• Fetch-backed loading is off by default and never used for retries unless the page opts in (see Advanced overrides).
• Qwen3-TTS retries a failed worker WebGPU synthesis once on the main thread with the cached model and projector bytes. Eligible WebGPU errors and generic worker timeouts retry with CPU-only settings; the exact worker request timeout and worker init timeout errors keep the original GPU offload. Models already on CPU are not retried, cancellation wins over recovery, and other errors propagate unchanged. The retry is slower, does not loop, and does not make up for too little browser memory; see the bridge recovery contract.

Worker requests have a two-minute budget by default. An embed or embedBatch with image or audio parts gets 10 minutes per input instead, and embedBatch restarts the budget after each finished input. workerRequestTimeoutMs overrides both.

When retries run out, the load throws an error with runtime hints such as core, source, nThreads, nGpuLayers, cache and bridge notes.

Troubleshooting map#

More cases: Troubleshooting.

Advanced overrides#

llamadart reads these globals when it creates the bridge. Set them before the first model load, for diagnosis or controlled deployments:

Pinned bridge assets#

This llamadart version pins bridge assets v0.1.82, which include the API-version-1 media embedding contract.

• The pinned v0.1.82 bridge assets embed llama.cpp v0.8.0, matching the native runtime.

Provenance and the pinned manifest digest are in doc/webgpu_bridge.md.

In a llamadart checkout, vendor the pinned assets into the chat app with:

WEBGPU_BRIDGE_ASSETS_TAG=v0.1.82 ./scripts/fetch_webgpu_bridge_assets.sh

The chat app bootstrap takes its CDN source from these globals:

<script>
  window.__llamadartBridgeAssetsRepo = 'leehack/llama-web-bridge-assets';
  window.__llamadartBridgeAssetsTag = 'v0.1.82';
</script>

Pin a known bridge asset tag in production and check the loaded module URL before reporting runtime behavior. The bridge JavaScript contract, the chat app's bootstrap knobs and hosting headers for Hugging Face Spaces are in doc/webgpu_bridge.md. Which repository owns bridge changes: Runtime ownership.

Older bridge assets#

The pinned assets include everything below. This table is for diagnosing a page that loads older assets, and the only place this documentation lists older asset tags. Where a feature has a capability probe, llamadart checks the probe, not the tag.