High-level vs backend API

llamadart exposes two API layers:

• High-level API (LlamaEngine + ChatSession) for most application code.
• Backend API (LlamaBackend) for advanced runtime control.

Entrypoints#

Apps need only the first. Import backend.dart next to it to implement a custom backend or a test fake (implements LlamaBackend, BackendTextToSpeech), or to drive LiteRT-LM directly.

A custom backend that runs one runtime reports it with BackendRuntimeIdentity, which LlamaEngine.runtime returns, and then loads only that runtime's ModelSource.format. To load any explicit format, implement BackendModelFormatRouting.

Custom backends and test fakes declare URL-load progress as void Function(double progress)? onProgress in 0.14. Forward the callback's fractions without using its return value: it is a notification, not a load cancellation signal. Apply the same signature to BackendModelFormatRouting.modelLoadFromUrlAs and callback fields. See the migration guide.

The LlamaEngineBackendHooks members, such as modelHandle, are extension members: call them on an expression typed LlamaEngine. A call through dynamic throws NoSuchMethodError.

High-Level API#

Use this by default. It handles model lifecycle, template routing, streaming, and chat history management.

Key Components:

• LlamaEngine: Loads/unloads models, runs generation, and can produce embeddings.
• ChatSession: Keeps message history for multi-turn conversation flows.

Advantages:

• Simplicity: Work with message/content objects instead of low-level backend calls.
• Template-aware: Uses model chat templates and parsing behavior automatically.
• Tool support: Works with structured tool-call outputs.
• Embedding APIs: embed(...) and embedBatch(...) on the same engine.
• Next-token scores: scoreNextToken(...) returns next-token log-probabilities; see Next-token scores.
• State persistence: stateSaveFile(...) / stateLoadFile(...); see State persistence.

import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final LlamaEngine engine = await LlamaEngine.load(
    LlamaModel(ModelSource.path('model.gguf')),
  );

  try {
    final ChatSession session = ChatSession(engine)
      ..systemPrompt = 'You are a concise assistant.';

    final LlamaCompletion reply = await session.send(
      'Hello! Give me one sentence about local inference.',
    );
    print(reply.text);
  } finally {
    await engine.dispose();
  }
}

Low-Level API#

LlamaBackend gives direct access to model/context handles and raw generation streams.

Key Components:

• LlamaBackend: Exposes explicit model/context creation and byte-stream generation.

Advantages:

• Granular control: Manage handles and pipeline steps directly.
• Integration flexibility: Useful for specialized runtime integrations.
• Optional capabilities: Backends can expose extra interfaces from package:llamadart/backend.dart, such as BackendStatePersistence when a runtime supports native KV-cache snapshots.

import 'dart:convert';
import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final LlamaBackend backend = LlamaBackend();
  final ModelParams modelParams = const ModelParams();

  final int modelHandle = await backend.modelLoad('model.gguf', modelParams);
  final int contextHandle = await backend.contextCreate(
    modelHandle,
    modelParams,
  );

  try {
    final Stream<String> textStream = backend
        .generate(
          contextHandle,
          'Hello from low-level API',
          const GenerationParams(),
        )
        .transform(const Utf8Decoder());

    await for (final String text in textStream) {
      print(text);
    }
  } finally {
    await backend.contextFree(contextHandle);
    await backend.modelFree(modelHandle);
    await backend.dispose();
  }
}

Which should you choose?#

Start with the high-level API. Move down to LlamaBackend only when you need explicit handle-level control that LlamaEngine/ChatSession do not provide.