On-device image generation (Preview)

ImageGenerationEngine turns a text prompt into images on the device. It runs stable-diffusion.cpp through the opt-in stable_diffusion native runtime, separately from LlamaEngine.

The API is experimental. Two small, distilled SD 1.x/2.x-family models are validated on phones and desktops: SDXS-512 and SD-Turbo. Desktop models cover SDXL-Lightning, FLUX.1-schnell, SD 3.5 Large Turbo and Z-Image-Turbo on desktop GPUs and Macs (validated on macOS Metal only), and other families load the same way; the recipes give each one's files and settings. Bounded image-to-image and binary-mask editing are available through ImageEditingRequest on the CPU recipe below. LoRA and ControlNet have no public image-generation API yet.

Support#

Measured with the prototype on the same runtime, 512x512, one step, warm:

await ImageGenerationEngine.checkRuntime() reports whether this build and device can generate images, and why not, without loading a model or blocking the calling isolate. Its first call in a process can take seconds; see First-image latency and warm-up. ImageGenerationCapabilities implements EngineCapabilities, like every engine's capabilities.

Bundle the runtime#

The runtime adds about 40 to 70 MB per target, so it is never bundled by default. Name it in the app's pubspec.yaml, then run flutter clean once:

hooks:
  user_defines:
    llamadart:
      llamadart_native_runtimes: all

Unset, llamadart_native_runtimes bundles llama_cpp and litert_lm. A list replaces those defaults, so it names every runtime the app loads. Beside Flutter Edge AI an implied litert_lm is left out; see Alongside Flutter Edge AI.

Linux and Windows get the Vulkan build when llamadart_native_backends selects Vulkan, which it does by default; set llamadart_stable_diffusion_backends: [cpu] for the CPU build. See Native runtime configuration.

Flutter iOS and macOS apps should also add the companion package, which links the runtime's XCFramework through Swift Package Manager and selects the runtime on those platforms by itself, adding about 37 MB per Apple target. Pair 0.16.0 with core 0.16.x, 0.15.2 with core >=0.15.2 <0.16.0, 0.14.0 with core 0.14.x, 0.13.0 with core 0.13.x, 0.0.3 with core 0.12.2, 0.12.1 and 0.12.0, and 0.0.2 with core 0.11.1 and 0.11.0. Cores older than 0.10.0 ignore it:

dependencies:
  llamadart_stable_diffusion_flutter: ^0.16.0

Without it, all leaves the image runtime out of a Flutter iOS or macOS app and ImageGenerationEngine.checkRuntime() reports it as not bundled. The build warns about it: flutter build macos, Xcode and xcodebuild show the warning, and flutter build ios output does not. A list naming stable_diffusion bundles the runtime through the hook instead, into a framework that Flutter marks MinimumOSVersion 13.0 while the library needs iOS 16.4, so App Store Connect rejects the iOS upload. The iOS build reports this as an Xcode build warning, which Xcode and xcodebuild show but plain flutter build and flutter run output does not. See Flutter Apple apps. The hook-bundled runtime also cannot carry an Apple privacy manifest. The companion's XCFramework carries one from stable-diffusion-native v0.2.0-1, which llamadart_stable_diffusion_flutter releases after 0.0.1 link; see Apple privacy manifest for the File Timestamp reasons (C617.1, 3B52.1) an app declares itself when its stable_diffusion.framework has no manifest.

Get a model#

Weight formats and layouts#

ImageGenerationEngine loads GGUF and Safetensors directly. Safetensors does not need conversion to GGUF, and a model can mix containers across its components. These image-loading rules are separate from LlamaEngine's text model formats; leave ModelSource.format unset for image files.

GGUF magic bytes and Safetensors headers drive role detection, not their filename extensions. The header check assigns roles; it does not validate all weight data or prove the architecture, dtype or quantization runs on the active backend. F16/F32/BF16 Safetensors and GGUF quantization still need compatible tensors and kernels. Do not assume arbitrary FP8/INT8 variants work; newer upstream and FP8 CPU qualification are tracked in stable-diffusion-native#20. checkRuntime() reports runtime/device availability, not support for every model. Use Support and the recipe's platform evidence separately. The public model roles do not include LoRA, ControlNet, upscaler or external tokenizer inputs.

FP8 image weights (F8_E4M3/F8_E5M2 in Safetensors, types 43/44 in GGUF) are refused with LlamaUnsupportedException on Metal, including ComputeDevice.auto when it selects Metal, before native context creation. Use ComputeDevice.cpu or compatible non-FP8 weights. Uninspectable GGUF/Safetensors headers and ordinary JSON shard indexes are also refused on Metal. This prevents the native abort tracked in #1446; selecting CPU keeps native dtype admission and does not qualify arbitrary FP8 models.

For a trusted PyTorch checkpoint whose native layout is compatible:

final model = ImageGenerationModel(
  ModelSource.path('model.ckpt'),
  role: ImageModelRole.checkpoint,
);

An explicit role selects a native input; it does not convert the file or make an unsupported checkpoint compatible. PyTorch component loading can work while larger full-checkpoint loading remains unqualified; do not generalize a small decoder's result to an entire diffusion model. The native reader handles a restricted state-dict representation without executing Python callables; arbitrary serialized modules, TorchScript and legacy TAR checkpoints are not supported. Use trusted model files. There is no public conversion API or load-time quantization/mmap option. Weights load eagerly; see the memory check before substituting larger unquantized files.

Pinned-runtime format checks

With the pinned stable-diffusion-native v0.3.0 (0518b34 plus the owner pickle-reader repair), complete byte-equivalent SD-Turbo F16 checkpoints pass through ImageGenerationEngine on macOS arm64 CPU:

These checks use the pinned SD-Turbo F16 source and its embedded full VAE. Local Safetensors and PyTorch 2.11/protocol-2 variants preserve every tensor byte. All seven public loads generate the same 128-square, one-step Euler/discrete image at seed 42, CFG 1 and four CPU threads, matching separate guarded direct-C controls. This is container and transport evidence, not an image-quality or speed benchmark.

Earlier v0.2.0-4 checks rejected the 2.6-GB legacy file and timed out on full ZIP inputs. The current legacy result verifies the published reader repair; ZIP's earlier timeout cause remains unresolved (owner #24). Passing these selected ZIP inputs is a later non-reproduction, not proof of a causal ZIP fix. Earlier small TAESD ZIP/nested/legacy and explicit local single-shard-index controls remain narrower component/layout evidence.

Other architectures, dtypes, producer layouts, arbitrary serialized objects, accelerators, Windows/Linux and mobile full-checkpoint paths are unqualified by this check. No model weights ship with the tests. The basic example's container checks and maintained image-container-formats scenario repeat the selected controls from local files and independent native references.

Sources#

A model is its main file and the other files it needs, each a ModelSource: a local file (ModelSource.path), an HTTP(S) URL or a Hugging Face file. ImageGenerationEngine.load downloads what is not local and gives each file its runtime role from its header, so the files can be listed in any order:

final cancel = ModelDownloadCancelToken();
final engine = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse(
      'hf://second-state/FLUX.1-schnell-GGUF@'
      '8c45a2ba25e2d02bd34230989fb54983f39e44ec/flux1-schnell-Q4_0.gguf',
    ),
    components: [
      ImageModelComponent.auto(ModelSource.path('ae.safetensors')),
      ImageModelComponent.auto(ModelSource.path('clip_l-Q8_0.gguf')),
      ImageModelComponent.auto(ModelSource.path('t5xxl-Q8_0.gguf')),
    ],
  ),
  params: const ImageModelParams(device: ComputeDevice.auto),
  download: ModelLoadOptions(cancelToken: cancel),
  onProgress: (progress) => print(progress.fraction),
);

Roles#

The engine reads only each file's header: the GGUF metadata and tensor names, or the safetensors header. A GGUF whose general.architecture names a llama.cpp language model, such as qwen3, is the llm text encoder; reading stops there. Each file gets one ImageModelRole:

The set needs exactly one checkpoint or diffusionModel and at most one file per role. load throws LlamaModelException, naming files by position (the main file, component 1, ...) and never by path, for:

• a LoRA adapter, ControlNet, upscaler or vision projector, which text to image does not load;
• taesd_decoder.safetensors, a decoder-only export the runtime rejects (use diffusion_pytorch_model.safetensors);
• two files in one role, a checkpoint plus separate diffusion weights, or no diffusion weights;
• a VAE or tiny autoencoder for other latent channels than the diffusion model's, such as TAESD (4 channels) with FLUX (16);
• a file with no known image-model layout, such as a pickled .ckpt;
• a ModelLoadOptions.sha256 mismatch, which names the main file the same way.

Give such a file its role yourself with ImageModelComponent(source, role: ImageModelRole.vae), or ImageGenerationModel(source, role: ...) for the main file. Explicit roles also settle layouts the header cannot tell apart: TAESD and TAESDXL share one layout, as do TAESD3 and TAEF1, and only their latent channels are checked. Classification runs on the downloaded file, so a wrong file costs its download; it stays in the cache.

Loading options#

• onProgress reports all files together as a ModelDownloadProgress. Cached and local files count as received. totalBytes is null until every file's size is known: local files from the start, others when their download starts.
• download: (ModelLoadOptions, as for LlamaEngine.load) applies to every remote file: cache policy and directory, bearerToken and headers (for example for a gated repository), resume, retries and the cancel token. Local files take only the cancel token. bearerToken and headers never cross hosts: with them set, remote files on more than one origin (scheme, host and port) throw LlamaArgumentException before anything downloads from a second host. ModelLoadOptions.sha256 verifies a single-file model; with components it throws LlamaUnsupportedException, since it cannot name one of several files. Verify each component independently before loading it.
• A cancelled load throws LlamaStateException. Cancelling stops a download at once; the native load cannot be interrupted, so a cancel during it takes effect when it returns, and the model it loaded is freed. A failed download throws what the download manager throws, usually LlamaModelException, with URL secrets redacted.
• The runtime check, params: checks and the check that every local file exists run first, so an unsupported platform, including the web, downloads nothing.
• A load is atomic: when it throws, nothing stays loaded.
• store: ModelFileStore(resolver: ..., downloadManager: ...) replaces the defaults, as for LlamaEngine. The default download manager caches in the same place as LlamaEngine's: on Android and iOS, llamadart/models in the app's cache directory, or DefaultModelDownloadManager.globalCacheDirectory when set. Pass ModelFileStore(downloadManager: DefaultModelDownloadManager.appPrivate(cacheDirectory: ...)) to keep the weights somewhere else.

engine.roles reports the file in each role.

Recipes#

Files pinned to the Hugging Face commits the basic app uses, and the request settings each model was validated with. Pin a commit (@<sha>) so the file cannot change under the app. The request carries the generation settings; unset, size falls back to 512x512, steps to 20 and guidance to 7.

SDXS-512 (683 MB, phones and desktops). Distilled for exactly one step at guidance 1:

final sdxs = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse(
      'hf://concedo/sdxs-512-tinySDdistilled-GGUF@'
      '3144d898d61492f8382ffcabec055733fc5b2a0e/'
      'sdxs-512-tinySDdistilled_Q8_0.gguf',
    ),
  ),
);
const request = ImageGenerationRequest(
  prompt: 'a red fox in autumn leaves',
  steps: 1,
  guidanceScale: 1,
);

SD-Turbo with TAESD (2.0 GB plus 10 MB). One step at guidance 1, or up to four for more detail at about four times the sampling time. TAESD replaces the full VAE decoder: several times faster and far less memory, at a small quality cost; use it on phones:

final sdTurbo = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse(
      'hf://Green-Sky/SD-Turbo-GGUF@19a31586d02d64a73b4419bc193b3ecfaf38e1f0/'
      'sd_turbo-f16-q8_0.gguf',
    ),
    components: [
      ImageModelComponent.auto(
        ModelSource.parse(
          'hf://madebyollin/taesd@614f76814bbe30edbe2e627ace1c2234c81a2c0e/'
          'diffusion_pytorch_model.safetensors',
        ),
      ),
    ],
  ),
);
const request = ImageGenerationRequest(
  prompt: 'a lighthouse at dusk',
  steps: 1,
  guidanceScale: 1,
);

SDXL-Lightning 4-step (6.9 GB plus 10 MB). 1024x1024, 4 Euler steps on the sgmUniform schedule at guidance 1, as the model card recommends. TAESDXL halved the time per image on an M4 Max; leave it out to decode with the checkpoint's VAE:

final sdxlLightning = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse(
      'hf://ByteDance/SDXL-Lightning@c9a24f48e1c025556787b0c58dd67a091ece2e44/'
      'sdxl_lightning_4step.safetensors',
    ),
    components: [
      ImageModelComponent.auto(
        ModelSource.parse(
          'hf://madebyollin/taesdxl@b20258aaef75ef61e659c1e0f14f251cf0ad153e/'
          'diffusion_pytorch_model.safetensors',
        ),
      ),
    ],
  ),
);
const request = ImageGenerationRequest(
  prompt: 'a lighthouse at dusk',
  width: 1024,
  height: 1024,
  steps: 4,
  guidanceScale: 1,
  sampler: ImageGenerationSampler.euler,
  scheduler: ImageGenerationScheduler.sgmUniform,
);

FLUX.1-schnell (12.3 GB). 1024x1024, 4 steps at guidance 1, from the Q4_0 transformer, the FLUX autoencoder and the CLIP-L and T5-XXL encoders. TAEF1 (hf://madebyollin/taef1@b1b2d00e9e440cfbf3dedb34266864da86016ceb/diffusion_pytorch_model.safetensors) can replace ae:

const fluxRepo =
    'hf://second-state/FLUX.1-schnell-GGUF@'
    '8c45a2ba25e2d02bd34230989fb54983f39e44ec';
const encoders =
    'hf://second-state/stable-diffusion-3.5-medium-GGUF@'
    '58b78c305a43ddfcffe1ab54d7022995f61667ac';
final flux = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse('$fluxRepo/flux1-schnell-Q4_0.gguf'),
    components: [
      for (final file in [
        '$fluxRepo/ae.safetensors',
        '$encoders/clip_l-Q8_0.gguf',
        '$encoders/t5xxl-Q8_0.gguf',
      ])
        ImageModelComponent.auto(ModelSource.parse(file)),
    ],
  ),
);
const request = ImageGenerationRequest(
  prompt: 'a red fox in autumn leaves',
  width: 1024,
  height: 1024,
  steps: 4,
  guidanceScale: 1,
);

SD 3.5 Large Turbo (10.9 GB). 1024x1024, 4 steps at guidance 1. The SD 3.5 VAE repository is gated, so this decodes with TAESD3:

const encoders =
    'hf://second-state/stable-diffusion-3.5-medium-GGUF@'
    '58b78c305a43ddfcffe1ab54d7022995f61667ac';
final sd35LargeTurbo = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse(
      'hf://city96/stable-diffusion-3.5-large-turbo-gguf@'
      '527c5548afc123f309238ca6bce7dfe3349aa997/sd3.5_large_turbo-Q4_0.gguf',
    ),
    components: [
      for (final file in [
        'hf://madebyollin/taesd3@d58dcaccd2b36fcb7a6b9e93c1cc507acab5a778/'
            'diffusion_pytorch_model.safetensors',
        '$encoders/clip_l-Q8_0.gguf',
        '$encoders/clip_g-Q8_0.gguf',
        '$encoders/t5xxl-Q8_0.gguf',
      ])
        ImageModelComponent.auto(ModelSource.parse(file)),
    ],
  ),
);
const request = ImageGenerationRequest(
  prompt: 'a red fox in autumn leaves',
  width: 1024,
  height: 1024,
  steps: 4,
  guidanceScale: 1,
);

Z-Image-Turbo (6.7 GB). 1024x1024, 8 steps at guidance 1, with the Qwen3-4B text encoder and the FLUX autoencoder:

final zImageTurbo = await ImageGenerationEngine.load(
  ImageGenerationModel(
    ModelSource.parse(
      'hf://leejet/Z-Image-Turbo-GGUF@c61c0e422dc8b541b7548cf33a4ef8302b0f8085/'
      'z_image_turbo-Q4_K.gguf',
    ),
    components: [
      for (final file in [
        'hf://second-state/FLUX.1-schnell-GGUF@'
            '8c45a2ba25e2d02bd34230989fb54983f39e44ec/ae.safetensors',
        'hf://unsloth/Qwen3-4B-Instruct-2507-GGUF@'
            'a06e946bb6b655725eafa393f4a9745d460374c9/'
            'Qwen3-4B-Instruct-2507-Q4_K_M.gguf',
      ])
        ImageModelComponent.auto(ModelSource.parse(file)),
    ],
  ),
);
const request = ImageGenerationRequest(
  prompt: 'a red fox in autumn leaves',
  width: 1024,
  height: 1024,
  steps: 8,
  guidanceScale: 1,
);

Model licenses#

Each model's weights carry their own license, and commercial-use terms differ between them. llamadart does not license any model; check the license on the linked model card before you ship or use a model commercially. Licenses as stated on the repositories in the recipes and their upstream model cards:

Any other model carries its own license.

Desktop models#

SDXL-Lightning, FLUX.1-schnell, SD 3.5 Large Turbo and Z-Image-Turbo generate at 1024x1024 with the recipe settings. They are for desktop GPUs and Macs, not phones. Measured from #802, first image in a new process, automatic attention and VAE settings:

The Vulkan figures come from stable-diffusion.cpp's own CLI on the same runtime commit with vae_conv_direct, which the engine now sets on Vulkan; llamadart itself was not run there. The tiny autoencoders (TAESDXL, TAEF1, TAESD3) cut 6 to 9 s from each 1024x1024 decode on the M4 Max with no visible quality loss in these samples. The SD 3.5 VAE repository is gated, so TAESD3 is the ungated choice.

Other families#

Any family the bundled stable-diffusion.cpp supports loads the same way, including other SDXL, SD 3.5, FLUX, Z-Image and Qwen-Image checkpoints. These need several GB of memory and are meant for desktop GPUs and Macs. Settings run in #802 (SD 3.5 Medium took about 3 minutes per 1024x1024 image on an M4 Max, Qwen-Image about 11):

Left unset, sampler, scheduler and flowShift are the runtime's own for the model: Euler for SD 3.5, FLUX and Z-Image, Euler ancestral with the discrete schedule for SD 1.x, 2.x and SDXL. flowShift applies only to flow-matching models.

Generate#

final engine = await ImageGenerationEngine.load(model);
final capabilities = await engine.capabilities;
print('${capabilities.modelVersion} on ${capabilities.backendName}');

final task = await engine.generate(
  const ImageGenerationRequest(
    prompt: 'a red fox in autumn leaves',
    steps: 1,
    guidanceScale: 1,
    seed: 42,
  ),
);
task.events.listen((event) {
  if (event
      case ImageGenerationProgressEvent(:final phase, :final step, :final steps)) {
    print('$phase $step/$steps');
  }
});
final result = await task.result;
final png = result.images.first.toPng();
print('${png.length} PNG bytes, seed ${result.seed}');
await engine.dispose();

generate returns the running ImageGenerationTask once the request passes its checks. task.events carries progress and never an error. task.result returns the images, or throws the failure, or LlamaStateException when the task is cancelled; task.done reports the same outcome as an ImageGenerationCompletion and never throws. engine.generateImage(request) is the one-call form of (await engine.generate(request)).result.

• Width and height are multiples of 8 from 64 to 2048, and default to 512. Use the model's native size: 512x512 for SDXS and SD-Turbo, 1024x1024 for the desktop models in the recipes. SDXS and SD-Turbo also work at 256. The runtime rounds the size up to a multiple of 64 for these models, so a 200x136 request produces 256x192; GeneratedImage.width and height report the real size.
• steps defaults to 20 and guidanceScale to 7, which suit undistilled SD 1.x and 2.x models; set the model's own, as in the recipes. A guidance of 1 skips the negative prompt and halves the work per step. sampler, scheduler and flowShift default to the runtime's choice for the model.
• A null seed picks one at random. result.seed reports the seed used, and image i of count used seed + i. The same seed, size, steps and model reproduce the same pixels.
• GeneratedImage.pixels holds row-major RGB bytes; toPng() encodes them.
• Invalid requests make generate throw LlamaImageGenerationException before a task starts.

load loads every weight up front. The first image in a process can still be slow while the GPU compiles shaders; see First-image latency and warm-up.

Attention and VAE settings#

Two runtime settings in ImageModelParams change speed and memory. Direct VAE convolutions leave the image identical on the CPU and change it by rounding on Metal; flash attention changes pixels slightly, by rounding. Left null, the engine picks them for the device it loads on:

The phone figures are Firebase Test Lab runs of the chat app's settings comparison. Peak memory is the phys_footprint interval peak on the iPhone and a 20 ms sampled VmRSS peak on Android, where Test Lab refused the VmHWM reset on all three phones. The Galaxy A53's memory check refused SD-Turbo (about 2.9 GiB needed, 2.5 GiB available), so it has no SD-Turbo figures, and its SDXS figures come from a run that does not verify because of that refusal: they are an observation, not verified evidence. The iPhone full-VAE arm has not been measured yet (#1033).

With a tiny autoencoder, direct convolutions cost time for little memory: on an M4 Max CPU they took 1.4 to 1.7 s per 512x512 SDXS image against 1.2 to 1.3 s, for 0.24 GiB less peak process memory. Set vaeDirectConvolution: true where that memory matters more.

Vulkan flash attention has not been measured yet, so it stays off there; pass flashAttention: true to try it. The runtime falls back to regular attention where a device has no kernel for it.

VAE tiling#

TiledImageGenerationRequest(vaeTiling: ImageVaeTiling()) decodes the image in overlapping tiles instead of all at once. The decoder's compute buffer then scales with the tile instead of the image, so peak memory drops most for large images and full VAEs; decoding takes longer, and pixels change slightly where tiles blend. It requires the pinned stable_diffusion v0.3.0 runtime (upstream 0518b34) and accepts header-verified SD 2.x checkpoints with their embedded full VAE or a valid TAESD override on CPU or Metal. SD-Turbo is the measured artifact; other accepted SD 2.x checkpoints and phones remain unmeasured. Unknown headers, external full VAEs, other component overrides, model families and backends throw LlamaUnsupportedException before a task starts. Tiling with either seamless circular axis is also refused until separately qualified. Custom image backends are unavailable for tiling in this slice; no public backend extension contract is added. Sampling and weights are unchanged. Requests without it decode the whole image, as before.

final result = await engine.generateImage(
  const TiledImageGenerationRequest(
    prompt: 'a lighthouse at dusk',
    width: 1024,
    height: 1024,
    steps: 1,
    guidanceScale: 1,
    vaeTiling: ImageVaeTiling(), // 256x256 tiles, overlap 0.5
  ),
);

Tile sizes are output pixels, multiples of 64 from 64 to 2048, and default to 256; a tile larger than the image covers it on that axis. overlap is the target fraction a tile shares with its neighbour, from 0 to 0.5 (default 0.5); the runtime spaces tiles evenly, so neighbours always overlap a little. Other values throw LlamaImageGenerationException before the task starts. The existing warmUp signature stays unchanged; use a single-step tiled generation and discard its image when tiled GPU pipelines need warming. Without tiling, the runtime already retries a decode that fails to allocate with tiles; tiling from the start keeps the decode under limits that end the app before an allocation fails. Cancelling does not interrupt a decode that has started.

SD-Turbo Q8_0 with its full VAE, one step, 256-pixel tiles at overlap 0.5, on an M4 Max:

At 512x512 the tiled image differed from the untiled one by 2.5 of 255 per channel on average and at most 88 near tile seams, with no visible seams. Vulkan GPUs and phones, including the iPhone GPU watchdog of #1047, are unmeasured.

Conditioning cache#

ImageConditioningCacheParams(capacity: ...) controls how many prompt-conditioning entries a loaded model retains. Ordinary ImageModelParams preserve the pinned native runtime's default of four entries; capacity 0 disables reuse. A positive integer sets the least-recently-used entry limit, up to the signed 32-bit C ABI limit (2147483647). Invalid values throw LlamaImageGenerationException before model files are resolved or downloaded.

final engine = await ImageGenerationEngine.load(
  model,
  params: const ImageConditioningCacheParams(capacity: 0),
);

The limit counts entries, not bytes: conditioning size depends on the text encoder and request. Reuse requires matching conditioning inputs, including the prompt and dimensions; another prompt is encoded separately. Entries belong to the loaded native context, remain available after a cancelled generation when valid, and are freed when it is disposed. Load a new engine to change capacity. This cache already operates when the option is unset; this control does not enable approximate diffusion-step caching, change sampling, or promise a speed or memory improvement.

The maintained image-generation-smoke checks compare disabled, default and one-entry caches with SDXS (embedded tiny decoder) and SD-Turbo (full VAE), including repeated/changed prompts, eviction, cancellation/reuse and reload. Qualification for this control covers macOS arm64 CPU and Metal. Android, iOS, Linux CPU/Vulkan, Windows CPU/Vulkan and macOS x64 have not been measured for this option; SDXL, SD 3.5, FLUX, Z-Image and Qwen-Image were not run. Web image generation remains unavailable. Approximation-cache modes remain separate work in #1233.

Seamless textures#

SeamlessImageGenerationRequest opts into circular padding independently on X (left/right) and Y (top/bottom). Both default to false; ordinary ImageGenerationRequest calls keep their existing behavior. The first qualified profile is SDXS (512-DS), its embedded tiny decoder, 512×512, CPU or Metal, loaded with both flashAttention: false and vaeDirectConvolution: false. The engine checks the native family and file headers, so renaming weights or forcing a checkpoint role does not grant support. Other families, component/decoder overrides, sizes, other GPUs and enabled attention/direct-convolution options throw LlamaUnsupportedException before a task starts. Defaults that enable flash attention must be overridden.

final engine = await ImageGenerationEngine.load(
  ImageGenerationModel(ModelSource.path('sdxs-512-tinySDdistilled_Q8_0.gguf')),
  params: const ImageModelParams(
    flashAttention: false,
    vaeDirectConvolution: false,
  ),
);
try {
  final result = await engine.generateImage(
    const SeamlessImageGenerationRequest(
      prompt: 'seamless texture of weathered gray stone, uniform rough stone '
          'surface, flat lighting, top down',
      width: 512,
      height: 512,
      steps: 4,
      guidanceScale: 1,
      seed: 42,
      sampler: ImageGenerationSampler.euler,
      scheduler: ImageGenerationScheduler.discrete,
      circularX: true,
      circularY: true,
    ),
  );
  await File('stone.png').writeAsBytes(result.images.single.toPng());
} finally {
  await engine.dispose();
}

This uses the pinned runtime's ordinary diffusion and VAE decoder circular convolutions. It does not enable VAE tiling or alter sampling defaults. Padding improves boundary continuity; prompt content and seeds can still produce an unsuitable texture. Inspect a 2×2 repeat before using the output. The fixed qualification prompt/seed must pass, on each requested axis: edge mean absolute RGB difference below 8/255, below 1.5 times the average interior adjacent-pixel difference, and below 40% of the same-seed off result. These criteria measure continuity, not artistic quality.

The maintained image-seamless-textures scenario saves individual and repeated PNGs, numerical JSON, checks native-reference pixels when provided, and checks ordinary output after circular generation and cancellation/reuse. Measured qualification is macOS arm64 CPU and actual Metal; other platforms are unmeasured. SD-Turbo's full VAE missed the fixed profile's absolute seam gate and is refused. Other families and external TAESD decoders are not qualified. See the reproducible source and artifact evidence.

First-image latency and warm-up#

ggml compiles GPU shaders the first time a process needs them, in two places:

• The first runtime probe in a process (checkRuntime() or load()) initializes the GPU backend. On Apple GPUs this compiles ggml's Metal library. Both run the probe on a separate isolate.
• The first generation on the GPU compiles the pipelines it runs.

The operating system or GPU driver caches the compiled shaders on disk, so later launches are faster. ggml in the bundled runtime keeps no cache of its own (no Metal binary archive or Vulkan pipeline cache); whether iOS and Android keep the driver cache across launches is not measured yet.

The empty-cache M4 Max figures ran with MTL_SHADER_CACHE_SIZE=0, which turns the Metal shader cache off. The Vulkan figures are from #779. The CPU compiles nothing; its first image is as fast as the next.

warmUp moves the pipeline compile off the first real image. It runs one single-step generation at the size and guidance it is given, with the request fallbacks otherwise, and discards it:

final engine = await ImageGenerationEngine.load(model);
// While the user writes the prompt:
await engine.warmUp(width: 1024, height: 1024, guidanceScale: 1);

• Warm up at the size and guidance the app will generate. ggml picks some pipelines by tensor size: on the M4 Max a 64x64 warm-up left about 0.1 s of the 512x512 compile, while a 512x512 warm-up left none.
• For the desktop models a warm-up is one sampling step and a decode at 1024x1024: about 2 to 4 s for SDXL-Lightning with TAESDXL on the M4 Max, and the same peak memory as an image, which the memory check covers where it runs.
• It moves the cost, it does not remove it. Call it while the user is not waiting, such as right after load while they type; load, warmUp and generate back to back take no less time than load and generate.
• It holds the one-operation slot: await it before the next generate or load, which otherwise throw LlamaStateException. dispose() cancels a running warm-up, which then completes normally.
• On the CPU it returns at once.

warmUp cannot move the Metal library compile, which happens before an engine exists. To keep a Flutter UI responsive on a first launch, check the runtime with checkRuntime() and show a progress indicator until it completes:

final capabilities = await ImageGenerationEngine.checkRuntime();

With an empty shader cache on the M4 Max, probing on the calling isolate stalled a 10 ms timer for the whole 16 s probe. During checkRuntime() the timer kept firing, with gaps of 13 to 31 ms in most runs. One pause remains: a garbage collection on the calling isolate waits while the probe isolate loads the runtime library (0.4 to 0.5 s), so an allocating UI can pause once for up to that long (179 ms in the chat example's macOS E2E). Calls that overlap share one probe. The compiled library belongs to the process, so later probes return at once. load() probes the same way and does not block the caller either.

Progress phases#

The runtime reports all progress as (step, steps) pairs with no phase, so the engine labels each report from the sequence:

1. encodingPrompt when the generation starts.
2. sampling with 0/steps when each image's first step starts, then one event per step. imageIndex says which image.
3. decoding with 0/count after the last image's last step, then the final event. The runtime reports decoding progress only when it decodes in tiles.

loading appears only if the runtime loads weights lazily, which the engine avoids by loading them eagerly.

The runtime records its reports and the engine reads them every 50 ms, so progress events arrive in groups, up to about 50 ms after the runtime made the report. Every report still becomes its events, in order, and the last ones arrive before the final event. A calling isolate that is busy delays them further.

Cancellation, concurrency and disposal#

• task.cancel() stops before the next sampling step or before decoding; task.done then reports cancelled and task.result throws LlamaStateException. A cancel issued before the runtime starts is honored too, up to 50 ms after it starts.
• stable-diffusion.cpp reports progress for the whole process, not per model, so one generation or model load runs at a time. Another generate or load meanwhile throws LlamaStateException, even on a different engine. The guard covers engines in one isolate; do not generate from several isolates at once, or each sees the other's progress.
• dispose() cancels a running generation, waits for it, and frees the model. The task reports cancelled, and its result and generateImage throw LlamaStateException, as does generate after dispose().
• dispose() is idempotent: later calls return the first call's future. isDisposed turns true at the first call, and await engine.capabilities then reports the engine as unsupported. capabilities never throws and changes only on dispose(), so a Flutter app can read it once after load and keep it in its state for build.
• A process that ends with an image model loaded, loading or generating no longer aborts on macOS Metal in the cases llamadart has run: a Dart program that returns from main or dies of an error, a Flutter macOS quit, and a native host's C exit() (see Model lifecycle for what each one waits for). Nothing cancels a running generation when a Dart program ends or dies of an error, or a Flutter app quits or hot restarts, so the process stays, and a hot restart waits, until the generation finishes. The wait is the rest of the generation, with no upper bound: for example 4.8 s for the rest of a 40-step SDXS generation and 11.2 s for SD-Turbo in a Flutter macOS quit on an M4 Max. A native host's C exit() cancels the generation and waits up to 15 seconds for it.
• Still dispose the engine before a Flutter app quits: the quit otherwise waits for a running generation, and the cleanup at exit has been measured on macOS only. Flutter desktop apps do not run State.dispose on quit, so await dispose() in AppLifecycleListener.onExitRequested. If ImageGenerationEngine.load is still running, await it there and dispose the engine it returns. If the engine's owner can be disposed before quit, such as a pushed route, use one app-level exit listener that also awaits a disposal the owner already started; see Model lifecycle.
• Generation runs in a worker isolate; the calling isolate stays responsive.

Memory check#

Before loading, the engine estimates the model's peak memory as its file sizes plus a quarter, plus 512 MiB, and compares that with the memory of the device the model loads on:

A model that does not fit throws LlamaModelException naming the estimate and the figures, instead of the system killing the app. Set ImageModelParams(checkMemory: false) to load anyway, without a check or a warning.

Vulkan GPUs#

A discrete Vulkan GPU does not bound a model by itself. stable-diffusion.cpp plans where each part of a model goes when it loads (its automatic fit, which the engine leaves on): the diffusion model, the text encoders and the VAE each go to the GPU that computes when they fit its free memory with room to compute, otherwise to system memory, otherwise to another GPU, otherwise they stay on disk. A model larger than the GPU therefore loads and generates more slowly.

So the engine works with two figures:

• The memory of the GPU that computes. A GPU's figure is its free memory when the driver reports a budget (VK_EXT_memory_budget): what this process can still allocate, which accounts for other processes and for image models already loaded. Without a budget the runtime reports the total as free, so the engine uses the total. A free figure of 0 against a positive total means the process is over its budget and counts as no free memory: every model gets the warning below, and that GPU adds nothing to the sum that refuses a load on Linux, so with one GPU a model above MemAvailable is refused. A model whose estimate exceeds this figure loads, and the engine logs one warning through LlamaLogging with both numbers and the device, saying that part of the model stays in system memory and generation will be slower. It is a warn record, so the default level none hides it.
• The memory of every discrete GPU plus system memory (MemAvailable, as for the CPU). On Linux a model whose estimate exceeds this sum is refused. Windows does not read system memory, so nothing is refused there and only the warning applies.

With several discrete GPUs, the one that computes is the runtime's choice: for ComputeDevice.auto the discrete GPU with the most free memory, wherever it is listed, and for ComputeDevice.gpu the first one. The engine follows that for the warning. capabilities.backendName still names the first GPU listed.

An integrated GPU uses host memory, and a driver that exposes it as several heaps makes the GPU's own figures count it more than once. The engine never uses them: beside a discrete GPU an integrated one adds nothing to the sum, and with integrated GPUs only it compares with MemAvailable on Linux, as for the CPU.

Nothing is checked or warned about when a GPU does not report its memory, when the SD_VK_DEVICE environment variable is set, or with a stable_diffusion runtime older than stable-diffusion-native v0.2.0-2.

The figures are the runtime's, which are ggml's, for every GPU it lists. The engine asks for them in a short-lived isolate, because the first such call in a process can initialize the GPU.

The Vulkan rules are tested on reported figures, and the call that reads them is checked on macOS Metal, where its total equals the working set above. On an NVIDIA L4 under Vulkan, one GPU each on Linux and Windows, the figures, the device facts and the device selection were checked. The warning, the refusal and the two-GPU paths have not been checked on a physical GPU yet (#1024). The Apple checks do not use these figures: Metal's total is the same working set, and its free figure counts only this process's allocations.

On Android, MemAvailable alone is too strict: it leaves out the memory the low-memory killer frees by stopping cached apps and what it swaps to zram, and it varied by up to a gigabyte between idle readings. On Firebase Test Lab, an app that kept touching all of its memory was killed only after allocating 0.9 to 3.6 GiB more than MemAvailable:

So 8 GB phones load SD-Turbo with TAESD (2.87 GiB estimated) even when MemAvailable reads 2.1 GiB, unless the app already holds more than about 0.6 GiB, such as a loaded chat model, and 6 GB phones refuse it unless MemAvailable alone covers it. With the check off, no SD-Turbo variant was killed on these phones, but the Galaxy A53 swapped most of the app out: a one-step image took 65 s with TAESD and 282 s with the full VAE, against 18 s and 67 s on the Pixel 6a.

The estimate does not depend on the image size. It covers the measured peaks at each model's native size, with the automatic attention and VAE settings: SDXS used 1.30 GiB and SD-Turbo on the CPU 2.66 GiB of process memory on an M4 Max; on five Android phones, loading and generating at 512x512 added at most 1.16 GiB to the app for SDXS, 2.26 GiB for SD-Turbo with TAESD and 2.64 GiB for SD-Turbo with its full VAE (estimates 1.30, 2.87 and 2.86 GiB); and 1024x1024 SDXL, SD 3.5 Large Turbo, FLUX and Z-Image stayed 0.5 to 1.6 GiB under it on Metal. SD 3.5 Medium, whose single file decodes with the full VAE, peaked 0.3 GiB above it (11.6 against 11.3 GiB). Larger sizes need more, especially on the CPU: SD-Turbo at 1024x1024 peaked at 4.7 GiB there.

Runtime logs#

stable-diffusion.cpp's and ggml's messages go to the handler of LlamaLogging.configure, each at its own level:

await LlamaLogging.configure(
  level: LlamaLogLevel.info,
  handler: (record) => print('[${record.level.name}] ${record.message}'),
);
final engine = await ImageGenerationEngine.load(model);

• A message has to pass both level and nativeLevel, so the runtime records from the stricter of the two, and nothing when either is none, the default. A load takes the levels it finds: configure logging before load.
• At debug the runtime logs each prompt verbatim, with its tokens. Keep that level out of logs that leave the device.
• Messages arrive after each load and generation and at dispose(), not while one runs: a worker isolate reads them from the runtime between its calls. Each starts with stable_diffusion: , and model file paths are replaced by roles, such as <checkpoint file>.
• The runtime keeps the most recent 256 KiB of messages. When one load or generation records more, the oldest are lost and one warning says how many. On an M4 Max an SDXS load recorded 28 messages at info and 40 at debug, and reading them took about 0.1 ms.
• The runtime no longer writes ggml's device messages to stderr, at any level, the default none included: its recorder is always registered, because the reason of a failed load comes from it. What a ggml backend writes to stderr itself still goes there.
• A stable_diffusion runtime older than stable-diffusion-native v0.2.0-2 has no log to read: it logs as before, to stderr only.

The core accepts only an Apple SD companion pinning its own runtime (v0.3.0): earlier runtimes have a smaller sd_ctx_params_t. Upgrade the core and llamadart_stable_diffusion_flutter together.

Errors#

Known limits#

• Runtime logs arrive after a load or generation, not during it. A failed load quotes what the runtime logged as errors, which for a file in the wrong role is only get sd version from file failed, and a failed generation does not quote its reason yet.
• Android runs on the CPU only: ggml Vulkan and OpenCL crashed or ran slower on the phones tried (stable-diffusion-native#2).
• iOS can abort a Metal command buffer under GPU pressure (seen once with SD-Turbo at four steps). This leaves ggml's Metal backend in an error state: from then on every generation on that engine fails until you dispose it and load the model again. A runtime older than stable-diffusion-native v0.2.0-2 has no log to detect this from, so its failures read as recoverable.
• On Linux a model on a Vulkan GPU is refused only when it fits neither the GPU nor the GPU plus system memory; one larger than the GPU's memory loads with a warning and runs slower. On Windows such a model is warned about and never refused, and the Windows CPU and integrated GPUs on Windows are not memory-checked. The Vulkan figures, device facts and device selection were validated under Vulkan on an NVIDIA L4 on Linux and Windows, one GPU each; the warning, refusal and two-GPU paths have not been validated on a physical GPU yet (#1024). Per-platform validation results, including timings, are in #779.
• The web has no image runtime yet (#780).

The basic app has a command-line image example, and the chat app has an image screen that downloads SDXS or SD-Turbo with TAESD from the same pinned files and generates on device. The command-line example also runs the desktop models from pinned downloads; both keep their pinned sources and settings in the app.

Local checkpoint sets#

ImageGenerationModel.localSafetensorsSet accepts a local directory containing model.safetensors.index.json, or that standard index itself, with 1..16 adjacent Safetensors shard files. This is a complete-checkpoint container recipe, not Diffusers Python pipeline/config discovery. Component overrides, Diffusers markers, extra model files, symlinks, unsafe shard paths, duplicate JSON keys/tensors, missing/misassigned tensors, malformed shapes/offsets and non-F16/F32 dtypes are rejected before native load.

The first recipe is the documented complete SD-Turbo F16/F32 tensor layout, identified by all 1,323 tensor names, dtypes and shapes. The fingerprint is a versioned recipe boundary, not a generic architecture-completeness detector or weight authenticity check. Its measured source is the complete local checkpoint derived for the format qualification; other matching weights are unmeasured. Only macOS CPU with pinned SD v0.3.0/upstream 0518b34, explicit flashAttention: false and vaeDirectConvolution: false is admitted. A loaded engine reports supportsLocalSafetensorsSets; it is false before load/after disposal. Editing, VAE tiling and seamless opt-ins remain refused for this path.

final model = ImageGenerationModel.localSafetensorsSet(
  localDirectoryOrStandardIndex,
  sha256: trustedPerFileHashes,
  cacheDirectory: appPrivateSetCache,
);
final engine = await ImageGenerationEngine.load(
  model,
  params: const ImageModelParams(
    device: ComputeDevice.cpu,
    flashAttention: false,
    vaeDirectConvolution: false,
  ),
);

The expected map includes the index and every shard's safe basename and SHA-256. Pin it from a trusted source independently of mutable input files. Every local file resolves through ModelFileStore; remote or outside-root redirects fail before a downloader acts. There is no remote index/repository/shard fetch and no credential forwarding. This route takes only the cancel token from ModelLoadOptions; its factory owns the checksum map and cache directory. Individual-file sources and custom legacy resolver/manager interfaces retain their existing behavior.

Reserve an app-private cache directory for owned sets. The loader copies and hashes into a unique partial stage, validates every header/index relationship, then atomically publishes by content identity. It re-verifies every cached file on reuse, including when the original files are no longer available. Never modify or remove published cache files while any engine uses them; clear this cache manually only after all engines are disposed. Source directory topology must remain stable during resolution/copy; concurrent malicious rename/symlink replacement is outside this app-owned-source contract. Original file contents may change after the verified copy. A precheck alone is not treated as a file lease or immutable snapshot.

Progress and memory accounting include the index, every shard header and all tensor bytes in deterministic basename order. Cancellation checks each stream chunk; failed/cancelled staging is removed. A complete published set remains cached if cancellation arrives afterward, and a corrupt cache is rejected instead of repaired under an active engine. Cleanup failures surface as errors. The recipe caps source files at 3 GiB including metadata; headers/index and nesting also have explicit bounds.

example/basic_app/bin/local_image_set.dart is a runnable local-only consumer. The maintained image-local-checkpoint-sets scenario builds two unchanged shards from the fixed complete checkpoint, compares public directory/index output to the pinned C settings, and checks source-removal lifetime, cache reuse, cancellation cleanup and recovery. See the provenance and procedure.

The first slice requires complete contiguous numbered shard basenames, model-00001-of-000NN.safetensors through model-000NN-of-000NN.safetensors, for N in 1..16. Arbitrary names, gaps and count aliases fail before manager or native work. This keeps caller-chosen basenames out of the existing native log path; no shared worker/redactor expansion is needed. The strict index mapping and tensor-layout fingerprint bind all native-visible tensor names; unsupported metadata/extra aliases use fixed error categories without quoting caller text.

Managed integrity, confinement and aggregate accounting apply only to localSafetensorsSet. Legacy ordinary explicit-role JSON forwarding remains unqualified and is refused on Metal because shard dtypes cannot be inspected; CPU forwarding does not acquire these guarantees. Unexpected process termination may leave uncommitted partial directories; these are never reused, and may be removed only after all processes using this private cache have stopped.

The fixed 128-square, one-step diagnostic output is an abstract green/white pattern rather than a recognizable fox. Native single-file, index, directory and public paths are byte-identical, so this qualifies container/cache-path equivalence only. It is not evidence of prompt fidelity or a recommended image-generation preset; no cause for that baseline output is asserted here.

Image editing#

ImageEditingRequest accepts an owned copy of decoded row-major RGB bytes and an optional one-channel binary mask. It leaves ImageGenerationRequest and text-to-image defaults unchanged. The output matches input dimensions: both axes must be multiples of 64 from 64 through 2048. The caller decodes PNG/JPEG, applies orientation, resizes and composites alpha before submission. No such transformations happen implicitly.

Check (await engine.capabilities).supportsImageEditing after loading. The bounded recipe requires stable-diffusion-native v0.3.0 (upstream 0518b34), a native-detected SD 2.x single checkpoint with a header-detected embedded full VAE on macOS CPU, flashAttention: false and vaeDirectConvolution: false. Component overrides, tiny decoders, other families, platforms/devices, VAE tiling and circular axes throw LlamaUnsupportedException before generation. SD-Turbo is the measured artifact; other SD 2.x checkpoints matching the recipe are source-compatible but unmeasured. This capability does not certify every prompt, resolution or checkpoint's output quality.

final input = ImageEditingPixels.rgb(width: width, height: height, pixels: rgb);
final request = ImageEditingRequest(
  prompt: 'a colorful painting of a house',
  initialImage: input,
  strength: 0.75,
  mask: ImageEditingPixels.mask(width: width, height: height, pixels: binaryMask),
  steps: 4,
  guidanceScale: 1,
  seed: 42,
);
final result = await engine.generateImage(request);

A mask value of 255 edits and 0 preserves the encoded latent. Other values are rejected; no implicit threshold is applied. Black output regions need not equal the input bytes because full-VAE encode/decode is lossy and convolutions cross mask boundaries. An omitted mask edits the whole image. strength must be finite and in (0, 1]; higher values allow larger changes, but step rounding can map nearby strengths to the same schedule. At 1 the runtime still encodes the input. This API does not load an inpainting-specific model or guarantee exact pixel compositing.

Inputs copy their supplied bytes and return copies from pixels. They remain owned through cancellation until the native call ends. Cancellation, concurrent generation and disposal use the same rules as text-to-image. Reusing a session for ordinary generation resets input/mask fields to native defaults. Run example/basic_app/bin/image_editing.dart with a local full checkpoint; it creates a decoded RGB fixture without a codec dependency. Maintainers can reproduce the independent C comparison, mask/strength measurements and cancellation checks in the qualification procedure.