Backend benchmarks

This page records app-level benchmark results for choosing between llama.cpp / GGUF and LiteRT-LM / .litertlm in llamadart. Scripts that reproduce them are under Reproducing.

These are deployment benchmarks, not pure kernel benchmarks. The artifacts are different runtime formats:

• gemma-4-E2B-it-Q4_K_S.gguf for llama.cpp.
• gemma-4-E2B-it.litertlm for native LiteRT-LM.
• gemma-4-E2B-it-web.litertlm for LiteRT-LM web.

Method#

All runs used the same long-form prompt, contextSize / max context of 4096, a target output cap of 256 tokens, one warmup run, and three measured runs. The tables report the median of the measured runs unless noted.

The prompt asked for a practical guide covering privacy, latency, offline behavior, personalization, battery tradeoffs, model formats, benchmarking, rollout strategy, and failure modes.

Web runs use the chat app and the benchmark static server, which sets the COOP/COEP headers required for threaded WebAssembly and supports byte-range requests for large local model artifacts.

Results#

Earlier Gemma 4 GGUF web failures were benchmark-harness artifacts, not a chat app support failure. The current web benchmark uses the same mem64 bootstrap path as the chat app, selects GpuBackend.auto, serves local GGUF files with byte-range support, and falls back from the fetch-backed loader to streamed loading when the bridge reports a generic core_abort.

The Pixel 9 Pro was explicitly woken and kept awake with svc power stayon true. Thermal status was 0 before the benchmark and 1 after the run, so the Android numbers should be treated as practical app-level numbers rather than a cooled lab baseline.

Speculative decoding check#

After GenerationParams.speculativeDecoding was exposed for native LiteRT-LM, the Gemma 4 E2B .litertlm path was rerun with the flag off and on. The flag remains off by default because the measured result was slower for this model on both devices.

The Pixel NPU path was also attempted for gemma-4-E2B-it.litertlm, but native LiteRT-LM failed engine creation for backend npu on this device/model bundle and reported that the Android NPU delegate may not support the device, OS, model, or bundle. Use GPU or CPU for this artifact unless a newer LiteRT-LM bundle/runtime combination validates NPU support.

llama.cpp upstream speculative parity check#

The llama.cpp speculative investigation on 2026-07-05 found no package-level model-cache issue and no evidence that upstream speculative decoding is universally faster. It is workload- and knob-sensitive. The clear local speedup came from an explicit repeated-context workload with smaller n-gram lookup settings.

At the time of this investigation, the package runtime pin was leehack/llamadart-native@b9873. That release did not publish standalone llama-cli / llama-server tool binaries, so upstream CLI comparison used the closest local llama.cpp tool build available from llamadart-native (build b1-e3471b3e7, from the b9571 line). Treat the CLI rows as historical behavior references, not exact artifact parity for the current runtime pin.

Conclusion: draftless n-gram speculation can be faster in llamadart when the prompt has reusable repeated context and the n-gram knobs match the workload. For natural short prompts or code prompts without a matching recent history, the draftless n-gram strategies can produce no drafts, so the extra speculative loop work is slower than baseline. Knob semantics per strategy are in Speculative decoding.

Remaining actionable work was split out instead of being folded into this benchmark documentation. llamadart-native#26 tracks exact native tool artifacts for future same-tag upstream comparisons; the earlier n-gram rollback and native sampler fixes have landed.

Interpretation#

On Pixel 9 Pro, LiteRT-LM GPU was about 9x faster than llama.cpp Vulkan for this Gemma 4 E2B deployment comparison. The GGUF/Vulkan path also consumed much more memory and pushed the device into light thermal pressure by the end of the run.

On the M4 Max, llama.cpp Metal and LiteRT-LM Metal were close for wall-clock throughput. Use model format, feature needs, and distribution constraints as the deciding factors on macOS rather than assuming LiteRT-LM is faster.

On web, both Gemma 4 artifacts completed through the chat app. LiteRT-LM WebGPU was about 2x faster than the GGUF WebGPU bridge on the measured decode counter, and its cold model load was much shorter. GGUF web still worked, but it was more sensitive to serving behavior because the large artifact needs a range-capable server or browser cache path.

Reproducing#

Run these from a repository checkout. Keep the model, prompt, contextSize, maxTokens and warmup/run counts identical across compared runs.

Backend comparison (Gemma 4)#

macOS:

DECODE_TOKENS=256 tool/macos_fair_litert_vs_llamadart.sh

# Native LiteRT-LM speculative decoding off/on check
SPECULATIVE=false DECODE_TOKENS=256 tool/macos_fair_litert_vs_llamadart.sh
SPECULATIVE=true  DECODE_TOKENS=256 tool/macos_fair_litert_vs_llamadart.sh

Web:

DOWNLOAD_LITERT_WEB_MODEL=1 \
DECODE_TOKENS=256 \
WARMUPS=1 \
RUNS=3 \
TARGETS=llamadart,litert_lm \
tool/web_fair_litert_vs_llamadart.sh

Use TARGETS=litert_lm or TARGETS=llamadart to run one runtime. For local large GGUF files, use the included benchmark server or another range-capable server; simple single-threaded file servers can make large browser model loads fail before the runtime sees real GGUF bytes. python -m http.server does not provide the same browser isolation and large-file behavior.

Pixel / Android:

ADB=/path/to/adb
DEVICE=<adb-serial>
"$ADB" -s "$DEVICE" shell svc power stayon true
"$ADB" -s "$DEVICE" shell input keyevent KEYCODE_WAKEUP

DEVICE="$DEVICE" \
ADB="$ADB" \
OUTPUT_TOKENS=256 \
WARMUPS=1 \
RUNS=3 \
TARGETS=litert_lm,llamadart \
tool/litert_lm_pixel_benchmark.sh

# Native LiteRT-LM GPU speculative decoding off/on check
DEVICE="$DEVICE" ADB="$ADB" TARGETS=litert_lm BACKEND=gpu \
  SPECULATIVE=false OUTPUT_TOKENS=256 WARMUPS=1 RUNS=3 \
  tool/litert_lm_pixel_benchmark.sh
DEVICE="$DEVICE" ADB="$ADB" TARGETS=litert_lm BACKEND=gpu \
  SPECULATIVE=true OUTPUT_TOKENS=256 WARMUPS=1 RUNS=3 \
  tool/litert_lm_pixel_benchmark.sh

Native llama.cpp generation and prompt reuse#

# Check prompt-prefix reuse parity before relying on it in production
dart run tool/testing/native_prompt_reuse_parity.dart \
  --model path/to/model.gguf \
  --prompt-file tool/testing/prompts/native_prompt_reuse_parity_prompts.txt \
  --max-prompts 8 \
  --runs 3 \
  --fail-on-mismatch

# Benchmark native generate/create TTFT and throughput
dart run tool/testing/native_inference_benchmark.dart \
  --model path/to/model.gguf \
  --gpu-layers 0 \
  --mode all \
  --runs 3 \
  --max-tokens 128

llama.cpp speculative decoding#

# Draftless n-gram strategies
dart run tool/testing/llama_cpp_speculative_benchmark.dart \
  --model path/to/model.gguf \
  --cases baseline,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,mixed-ngram \
  --backend cpu \
  --gpu-layers 0 \
  --max-tokens 128 \
  --runs 3 \
  --draft-token-max 1,2 \
  --ngram-size-m 8,16 \
  --warmups 1

# Experimental DSpark against the same target baseline
dart run tool/testing/llama_cpp_speculative_benchmark.dart \
  --model path/to/target.gguf \
  --draft-model path/to/dspark-draft.gguf \
  --cases baseline,draft-dspark \
  --backend metal \
  --gpu-layers 99 \
  --max-tokens 256 \
  --runs 3 \
  --warmups 1 \
  --include-output

External draft-model cases need --draft-model. Bundled MTP omits it and loads the target's MTP tensors automatically. The ngram-cache case needs cache paths.

Speculative n-gram upstream parity#

Set MODEL_PATH to the cached GGUF location on your machine. Set LLAMA_CLI to the upstream llama-cli build you are comparing; the parity table above used a local b9571-line tool because the b9873 native release did not publish standalone CLI artifacts.

MODEL_PATH="${MODEL_PATH:-$HOME/Library/Caches/llamadart/models/Qwen3.5-0.8B-Q4_K_M.gguf}"
LLAMA_CLI="${LLAMA_CLI:-/path/to/llama-cli}"

PROMPT_REPEAT='Repeat and continue this sequence exactly:
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta
alpha beta gamma delta epsilon zeta eta theta'

dart run tool/testing/llama_cpp_speculative_benchmark.dart \
  --model "$MODEL_PATH" \
  --cases baseline,ngram-map-k,ngram-map-k4v \
  --backend cpu \
  --gpu-layers 0 \
  --context-size 4096 \
  --max-tokens 128 \
  --runs 2 \
  --warmups 1 \
  --draft-token-max 8 \
  --ngram-size-n 4 \
  --ngram-size-m 8,16 \
  --ngram-min-hits 1 \
  --temp 0 \
  --repeat-penalty 1.0 \
  --raw-prompt \
  --prompt "$PROMPT_REPEAT"

"$LLAMA_CLI" \
  -m "$MODEL_PATH" \
  --device none \
  -ngl 0 \
  -c 4096 \
  -n 128 \
  --temp 0 \
  --repeat-penalty 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --min-p 0.05 \
  --seed 7 \
  --single-turn \
  --no-display-prompt \
  --simple-io \
  -p "$PROMPT_REPEAT"

"$LLAMA_CLI" \
  -m "$MODEL_PATH" \
  --device none \
  -ngl 0 \
  -c 4096 \
  -n 128 \
  --temp 0 \
  --repeat-penalty 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --min-p 0.05 \
  --seed 7 \
  --single-turn \
  --no-display-prompt \
  --simple-io \
  --spec-type ngram-map-k \
  --spec-draft-n-max 8 \
  --spec-ngram-map-k-size-n 4 \
  --spec-ngram-map-k-size-m 8 \
  --spec-ngram-map-k-min-hits 1 \
  -p "$PROMPT_REPEAT"

LiteRT-LM runtime controls smoke test#

The smoke tool reads LiteRT-LM runtime controls from environment variables and reports the selected values in its JSON result:

LITERT_LM_ACTIVATION_DATA_TYPE=float16 \
LITERT_LM_PREFILL_CHUNK_SIZE=128 \
LITERT_LM_PARALLEL_FILE_SECTION_LOADING=false \
LITERT_LM_DISPATCH_LIB_DIR=/path/to/dispatch \
dart run tool/litert_lm_engine_smoke.dart /models/model.litertlm cpu

Embedding throughput#

Compare sequential vs batch embedding throughput and sweep max-seq (ModelParams.maxParallelSequences) values:

# Single benchmark report
dart run tool/testing/native_embedding_benchmark.dart \
  --model path/to/model.gguf \
  --cpu \
  --mode both \
  --input-count 8 \
  --max-seq 8

# max-seq sweep with CSV output
dart run tool/testing/native_embedding_sweep.dart \
  --model path/to/model.gguf \
  --cpu \
  --max-seq-values 1,2,4,8 \
  --csv-out embedding_speedup.csv