Find and choose GGUF and LiteRT-LM models

llamadart supports two model artifact families:

• GGUF (GGML Unified Format), the standard format used by llama.cpp.
• .litertlm bundles, used by LiteRT-LM.

You cannot use raw PyTorch (.bin or .safetensors) models directly; they must be converted into a runtime-ready artifact first. For most open model workflows that means a quantized GGUF. For LiteRT-LM deployments, use a published .litertlm bundle that matches the LiteRT-LM runtime.

Fortunately, thousands of pre-converted GGUF models are readily available. LiteRT-LM bundles are more specialized; use them when your target model is distributed for LiteRT-LM or when you are intentionally benchmarking that runtime path.

Where to find models#

The best place to find GGUF models is Hugging Face.

You can search for any model name followed by gguf (e.g., Llama-3-8B-Instruct-GGUF). For a deeper dive into the GGUF ecosystem and how quantization works, check out these insightful resources:

• The GGUF format specification
• Hugging Face's official GGUF documentation
• Quantization in llama.cpp

For LiteRT-LM, look for model repositories that publish .litertlm files and document LiteRT-LM compatibility. If both GGUF and .litertlm variants exist, see Choosing llama.cpp or LiteRT-LM before treating them as equivalent artifacts.

Understanding Quantization#

GGUF models usually come in different "quantization" levels, denoted by tags like Q4_K_M or Q8_0. Quantization reduces the precision of the model's weights to save memory and increase inference speed, at a slight cost to "smartness" (perplexity).

Here is a quick guide to choosing a quantization level:

• Q4_K_M: The recommended baseline. It offers an excellent balance between small file size, fast generation, and retaining the model's original quality.
• Q5_K_M: Slightly larger and slower than Q4, but retains more quality. Good if you have the RAM to spare.
• Q8_0: Almost indistinguishable from the unquantized raw model, but requires double the RAM of Q4.
• Q2_K / Q3_K: Highly compressed. Useful only if you are severely constrained by RAM (e.g., running on older mobile phones), but expect noticeable degradation in reasoning logic.

Recommended constraints per platform#

When downloading a model, check its file size. Your target device needs enough free RAM (or VRAM for GPU offloading) to load the model, plus a bit extra for the context window.

Starting points#

These files are the ones the examples in this repository use. Sizes are for the file as downloaded. Paste a path into ModelSource.parse(...).

Downloading a model#

Once you find a model on Hugging Face:

1. Go to the Files and versions tab of the model repository.
2. Look for a file ending in .gguf (e.g., model-q4_k_m.gguf) or .litertlm.
3. Pass the exact repository path to ModelSource.parse('hf://owner/repo/path/to/model.gguf') and load it with engine.loadModelSource(...). Keep the real file extension in the path so LlamaBackend() can route to the correct runtime.

On native, engine.loadModel() takes a filesystem path; on web, a URL. llamadart has no asset loader: to ship a model as a Flutter asset on native, copy the asset to a file first (for example into the app support directory) and pass that path.

Revisions, private repositories, mmproj projector files and sharded GGUFs: see Download and cache models.