Backend benchmarks

This page records app-level benchmark results comparing llama.cpp / GGUF and LiteRT-LM / .litertlm in llamadart. They are historical measurements, not refreshed per release. The Gemma 4 E2B tables did not record runtime pins, so re-run the scripts under Reproducing on the current pins before relying on a number. The llama.cpp parity check records its date and pin below. User guidance is in Performance tuning.

These are deployment benchmarks, not pure kernel benchmarks. The artifacts are different runtime formats:

• gemma-4-E2B-it-Q4_K_S.gguf for llama.cpp.
• gemma-4-E2B-it.litertlm for native LiteRT-LM.
• gemma-4-E2B-it-web.litertlm for LiteRT-LM web.

Method#

All runs used the same long-form prompt, contextSize / max context of 4096, a target output cap of 256 tokens, one warmup run, and three measured runs. The tables report the median of the measured runs unless noted.

The prompt asked for a practical guide covering privacy, latency, offline behavior, personalization, battery tradeoffs, model formats, benchmarking, rollout strategy, and failure modes.

Web runs use the chat app and the benchmark static server, which sets the COOP/COEP headers required for threaded WebAssembly and supports byte-range requests for large local model artifacts.

Results#

The Pixel 9 Pro Vulkan row is a speed measurement, not a recommendation. Android llama.cpp Vulkan is experimental and device-dependent, and on this device (Mali-G715) a prompt longer than about 32 tokens evaluated in one batch returns wrong text, confirmed so far with one model (Qwen3.5-0.8B Q4_0) (#948, known limitations). Current native Qwen3.5 on Android Vulkan with unsafe or unproven small-matmul subgroup facts accepts only ordinary text with microBatchSize unset or 1..8. Larger explicit sizes, media, speculation and embeddings throw typed unsupported before evaluation. This conservative restriction is not proof of a native repair or final-artifact GPU qualification; CPU remains available.

The Pixel 9 Pro was explicitly woken and kept awake with svc power stayon true. Thermal status was 0 before the benchmark and 1 after the run, so the Android numbers should be treated as practical app-level numbers rather than a cooled lab baseline.

Speculative decoding check#

After GenerationParams.speculativeDecoding was exposed for native LiteRT-LM, the Gemma 4 E2B .litertlm path was rerun with the flag off and on. The flag remains off by default because the measured result was slower for this model on both devices.

The Pixel NPU path was also attempted for gemma-4-E2B-it.litertlm, but native LiteRT-LM failed engine creation for backend npu on this device/model bundle and reported that the Android NPU delegate may not support the device, OS, model, or bundle. Use GPU or CPU for this artifact unless a newer LiteRT-LM bundle/runtime combination validates NPU support.

llama.cpp upstream speculative parity check#

The llama.cpp speculative investigation on 2026-07-05 found no package-level model-cache issue and no evidence that upstream speculative decoding is universally faster. It is workload- and knob-sensitive. The clear local speedup came from an explicit repeated-context workload with smaller n-gram lookup settings.

At the time of this investigation, the package runtime pin was leehack/llamadart-native@b9873. That release did not publish standalone llama-cli / llama-server tool binaries, so upstream CLI comparison used the closest local llama.cpp tool build available from llamadart-native (build b1-e3471b3e7, from the b9571 line). Treat the CLI rows as historical behavior references, not exact artifact parity for the current runtime pin.

Conclusion: draftless n-gram speculation can be faster in llamadart when the prompt has reusable repeated context and the n-gram knobs match the workload. For natural short prompts or code prompts without a matching recent history, the draftless n-gram strategies can produce no drafts, so the extra speculative loop work is slower than baseline. Knob semantics per strategy are in Speculative decoding.

Interpretation#

On Pixel 9 Pro, LiteRT-LM GPU was about 9x faster than llama.cpp Vulkan for this Gemma 4 E2B deployment comparison. The GGUF/Vulkan path also consumed much more memory and pushed the device into light thermal pressure by the end of the run.

On the M4 Max, llama.cpp Metal and LiteRT-LM GPU were close for wall-clock throughput. Use model format, feature needs, and distribution constraints as the deciding factors on macOS rather than assuming LiteRT-LM is faster.

On web, both Gemma 4 artifacts completed through the chat app. LiteRT-LM WebGPU was about 2x faster than the GGUF WebGPU bridge on the measured decode counter, and its cold model load was much shorter. GGUF web still worked, but it was more sensitive to serving behavior because the large artifact needs a range-capable server or browser cache path.

Reproducing#

Run these from a repository checkout. Keep the model, prompt, contextSize, maxTokens and warmup/run counts identical across compared runs.

Backend comparison (Gemma 4)#

macOS:

DECODE_TOKENS=256 tool/macos_fair_litert_vs_llamadart.sh

# Native LiteRT-LM speculative decoding off/on check
SPECULATIVE=false DECODE_TOKENS=256 tool/macos_fair_litert_vs_llamadart.sh
SPECULATIVE=true  DECODE_TOKENS=256 tool/macos_fair_litert_vs_llamadart.sh

Web:

DOWNLOAD_LITERT_WEB_MODEL=1 \
DECODE_TOKENS=256 \
WARMUPS=1 \
RUNS=3 \
TARGETS=llamadart,litert_lm \
tool/web_fair_litert_vs_llamadart.sh

Use TARGETS=litert_lm or TARGETS=llamadart to run one runtime. For local large GGUF files, use the included benchmark server or another range-capable server; simple single-threaded file servers can make large browser model loads fail before the runtime sees real GGUF bytes. python -m http.server does not provide the same browser isolation and large-file behavior.

Pixel / Android:

ADB=/path/to/adb
DEVICE=<adb-serial>
"$ADB" -s "$DEVICE" shell svc power stayon true
"$ADB" -s "$DEVICE" shell input keyevent KEYCODE_WAKEUP

DEVICE="$DEVICE" \
ADB="$ADB" \
OUTPUT_TOKENS=256 \
WARMUPS=1 \
RUNS=3 \
TARGETS=litert_lm,llamadart \
tool/litert_lm_pixel_benchmark.sh

# Native LiteRT-LM GPU speculative decoding off/on check
DEVICE="$DEVICE" ADB="$ADB" TARGETS=litert_lm BACKEND=gpu \
  SPECULATIVE=false OUTPUT_TOKENS=256 WARMUPS=1 RUNS=3 \
  tool/litert_lm_pixel_benchmark.sh
DEVICE="$DEVICE" ADB="$ADB" TARGETS=litert_lm BACKEND=gpu \
  SPECULATIVE=true OUTPUT_TOKENS=256 WARMUPS=1 RUNS=3 \
  tool/litert_lm_pixel_benchmark.sh

Native llama.cpp generation and prompt reuse#

# Check prompt-prefix reuse parity before relying on it in production
dart run tool/testing/native_prompt_reuse_parity.dart \
  --model path/to/model.gguf \
  --prompt-file tool/testing/prompts/native_prompt_reuse_parity_prompts.txt \
  --max-prompts 8 \
  --runs 3 \
  --fail-on-mismatch

# Benchmark native generate/create TTFT and throughput
dart run tool/testing/native_inference_benchmark.dart \
  --model path/to/model.gguf \
  --gpu-layers 0 \
  --mode all \
  --runs 3 \
  --max-tokens 128

llama.cpp speculative decoding#

# Draftless n-gram strategies
dart run tool/testing/llama_cpp_speculative_benchmark.dart \
  --model path/to/model.gguf \
  --cases baseline,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,mixed-ngram \
  --backend cpu \
  --gpu-layers 0 \
  --max-tokens 128 \
  --runs 3 \
  --draft-token-max 1,2 \
  --ngram-size-m 8,16 \
  --warmups 1

# Experimental DSpark against the same target baseline
dart run tool/testing/llama_cpp_speculative_benchmark.dart \
  --model path/to/target.gguf \
  --draft-model path/to/dspark-draft.gguf \
  --cases baseline,draft-dspark \
  --backend metal \
  --gpu-layers 99 \
  --max-tokens 256 \
  --runs 3 \
  --warmups 1 \
  --include-output

The speculative benchmark reserves the maximum effective draft length among the selected cases. Draft-model cases do not inherit the default n-gram window of 48, and an n-gram window does not inherit an unused draft or ngramTokenMax setting. Selecting only baseline reserves no rollback snapshots. This removes unrelated reservations; it does not make recurrent/hybrid reservations safe.

External draft-model cases need --draft-model. Bundled MTP omits it and loads the target's MTP tensors automatically. The ngram-cache case needs cache paths.

Speculative n-gram upstream parity#

Set MODEL_PATH to the cached GGUF location on your machine. Set LLAMA_CLI to the upstream llama-cli build you are comparing; the parity table above used a local b9571-line tool because the b9873 native release did not publish standalone CLI artifacts.

Use a non-recurrent target such as Qwen2.5 for this workload. The current native runtime rejects the nonzero speculative rollback reserve for recurrent or hybrid targets, including Qwen3.5; the historical measurements above retain their original runtime and model scope.

MODEL_PATH="${MODEL_PATH:-$HOME/Library/Caches/llamadart/models/qwen2.5-0.5b-instruct-q4_k_m.gguf}"
LLAMA_CLI="${LLAMA_CLI:-/path/to/llama-cli}"

PROMPT_REPEAT='Repeat and continue this sequence exactly:
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta'

dart run tool/testing/llama_cpp_speculative_benchmark.dart \
  --model "$MODEL_PATH" \
  --cases baseline,ngram-map-k,ngram-map-k4v \
  --backend cpu \
  --gpu-layers 0 \
  --context-size 4096 \
  --max-tokens 128 \
  --runs 2 \
  --warmups 1 \
  --draft-token-max 8 \
  --ngram-size-n 4 \
  --ngram-size-m 8,16 \
  --ngram-min-hits 1 \
  --temp 0 \
  --repeat-penalty 1.0 \
  --raw-prompt \
  --prompt "$PROMPT_REPEAT"

"$LLAMA_CLI" \
  -m "$MODEL_PATH" \
  --device none \
  -ngl 0 \
  -c 4096 \
  -n 128 \
  --temp 0 \
  --repeat-penalty 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --min-p 0.05 \
  --seed 7 \
  --single-turn \
  --no-display-prompt \
  --simple-io \
  -p "$PROMPT_REPEAT"

"$LLAMA_CLI" \
  -m "$MODEL_PATH" \
  --device none \
  -ngl 0 \
  -c 4096 \
  -n 128 \
  --temp 0 \
  --repeat-penalty 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --min-p 0.05 \
  --seed 7 \
  --single-turn \
  --no-display-prompt \
  --simple-io \
  --spec-type ngram-map-k \
  --spec-draft-n-max 8 \
  --spec-ngram-map-k-size-n 4 \
  --spec-ngram-map-k-size-m 8 \
  --spec-ngram-map-k-min-hits 1 \
  -p "$PROMPT_REPEAT"

LiteRT-LM runtime controls smoke test#

The smoke tool reads LiteRT-LM runtime controls from environment variables and reports the selected values in its JSON result:

LITERT_LM_ACTIVATION_DATA_TYPE=float16 \
LITERT_LM_PREFILL_CHUNK_SIZE=128 \
LITERT_LM_PARALLEL_FILE_SECTION_LOADING=false \
LITERT_LM_DISPATCH_LIB_DIR=/path/to/dispatch \
dart run tool/litert_lm_engine_smoke.dart /models/model.litertlm cpu

Embedding throughput#

Compare sequential vs batch embedding throughput and sweep max-seq (ModelParams.maxParallelSequences) values:

# Single benchmark report
dart run tool/testing/native_embedding_benchmark.dart \
  --model path/to/model.gguf \
  --cpu \
  --mode both \
  --input-count 8 \
  --max-seq 8

# max-seq sweep with CSV output
dart run tool/testing/native_embedding_sweep.dart \
  --model path/to/model.gguf \
  --cpu \
  --max-seq-values 1,2,4,8 \
  --csv-out embedding_speedup.csv