Tool calling with local models

Pass ToolDefinitions to engine.create or ChatSession.create. The model's chat template renders them, and the parser returns the model's calls as chunk.toolCalls. session.sendWithTools runs the calls with each tool's handler until the model answers; create and engine.create only return the calls, and your code runs them.

Automatic loops (sendWithTools and completeWithTools) require reliable runtime termination reporting. The pinned native LiteRT-LM v0.18.0 and Web @litert-lm/core@0.18.0 cannot distinguish normal completion from a per-request token cutoff. They throw LlamaUnsupportedException before starting the loop or modifying its history, including when resuming a turn or passing an empty tools list. Plain ChatSession.create and LlamaEngine.create remain available for manually managed completion; their LiteRT-LM stop finish reason does not prove the model reached EOS. Do not execute calls automatically based on that value or infer truncation from output length. See the owner runtime prerequisite.

Define a tool#

handler is optional: leave it out for a tool your app runs itself.

final weatherTool = ToolDefinition(
  name: 'get_weather',
  description: 'Get current weather for a city',
  parameters: [
    ToolParam.string('city', description: 'City name', required: true),
    ToolParam.enumType('unit', values: ['celsius', 'fahrenheit']),
  ],
  handler: (params) async {
    final city = params.getRequiredString('city');
    final unit = params.getString('unit') ?? 'celsius';
    return {'city': city, 'temperature': 22, 'unit': unit};
  },
);

Run the tool-call loop#

final engine = await LlamaEngine.load(
  LlamaModel(ModelSource.path('model.gguf')),
);
final session = ChatSession(engine);

final result = await session.sendWithTools(
  'What is the weather in Seoul?',
  tools: [weatherTool],
  maxRounds: 5,
);
switch (result.stopReason) {
  case LlamaToolLoopStopReason.completed:
    print(result.text);
  case LlamaToolLoopStopReason.unhandledToolCalls:
  case LlamaToolLoopStopReason.maxRounds:
  case LlamaToolLoopStopReason.contextExceeded:
  case LlamaToolLoopStopReason.truncated:
  case LlamaToolLoopStopReason.cancelled:
    print('Stopped (${result.stopReason.name}): ${result.pendingToolCalls}');
}
await engine.dispose();

Each round collects a reply. When it calls tools, sendWithTools runs the calls concurrently and adds one tool message per call, in call order, with a LlamaToolResultContent carrying the call's id and name; the next round continues the turn without a new user message. completeWithTools(parts) does the same for multimodal parts, and completeWithTools(const []) continues the current turn.

• No handler: a call to a tool without a handler runs onToolCall. Without onToolCall, the loop stops before running any call of that reply, with unhandledToolCalls; result.pendingToolCalls lists them. Add a tool message per call with session.addMessage(...), then call completeWithTools(const [], tools: ...) to continue. Answer every call, even to decline it: until then the history ends with the calls, and templates such as Ministral 3's reject a new user turn.
• Errors: an exception from a tool, a call to an unknown tool name, or arguments that are not a JSON object become the tool result {'error': message}, so the model can recover. That puts the exception's text into the prompt and the session history; pass onToolError to choose the result, for example to redact secrets, or rethrow from it to fail the loop.
• Bounds: after maxRounds tool rounds, further calls are returned unrun with maxRounds. Calls proposed from a prompt that did not fit the context budget (session.lastRequestFitContext == false) are not run (contextExceeded). A reply cut off at GenerationParams.maxTokens or the end of the context (finishReason length) stops the loop with truncated, without running its calls: its text may be an unfinished tool call or thinking, not an answer. A reported limit takes precedence over any parsed calls: even complete calls before a partial second parallel call are withheld, so result.pendingToolCalls is empty for that reply. result.completion keeps its text and thinking; raise maxTokens and send the turn again. The whole turn is rolled back, including earlier tool rounds; effects of tools that already ran cannot be undone.
• Cancel: engine.cancelGeneration(), model unload or replacement, and engine disposal stop the loop with cancelled. Running tools finish first. A partial answer stays as the turn's reply. If a reply also reports a generation limit, cancelled takes precedence and the incomplete turn is rolled back.
• toolChoice applies to the first request only, so ToolChoice.required forces one call and later rounds can answer.

History after a stop#

Every stop leaves the session ready for a new user turn, except unhandledToolCalls, which waits for your tool messages:

A rolled-back turn would otherwise end with unanswered calls or tool results, which some templates, such as Ministral 3's and Mistral Small 3.2's, cannot render before a new user turn. completeWithTools(const [], ...) continues the open turn, so its rollback also removes that turn's earlier messages, including the tool results you added. Messages that other code added during the loop stay, except those an addMessage override appends while the loop adds a message (or that a listener adds from inside that override): the rollback removes them with the turn, and they are neither in result.messages nor reported to onMessageAdded. Older turns that context trimming dropped stay dropped, as after create. A rollback edits the history without calling addMessage or reset, so a ChatSession subclass that mirrors the history should check result.rolledBack.

result.rolledBack tells whether the turn was removed. Tools that ran keep their side effects: result.messages holds every message of the turn, including their results, and onMessageAdded has already reported the ones the call added. To resume a rolled-back turn, add result.messages back, answer result.pendingToolCalls, and call completeWithTools(const [], tools: ...). After truncated, raise maxTokens and send the turn again instead: result.messages ends with the partial reply, and templates such as Ministral 3's can reject the next user turn after it.

On WebGPU a cancel ends generation with a stream error; the loop still reports it as cancelled.

To start a new chat while the loop runs, call engine.cancelGeneration(), await the loop, then call session.reset(): a reset while a reply is generating can leave that reply in the new chat (#888).

Run the calls yourself#

To stream each reply or control every call, collect the reply and add the results yourself:

final tools = [weatherTool];
final reply = await session.create(
  [const LlamaTextContent('What is the weather in Seoul?')],
  tools: tools,
).collect();
for (final call in reply.toolCalls) {
  session.addMessage(
    LlamaChatMessage.withContent(
      role: LlamaChatRole.tool,
      content: [
        LlamaToolResultContent(
          id: call.id,
          name: call.name,
          result: await weatherTool.invoke(call.arguments),
        ),
      ],
    ),
  );
}
if (reply.toolCalls.isNotEmpty) {
  print(await session.create(const [], tools: tools).text());
} else {
  print(reply.text);
}

• Tool calls arrive complete, with JSON arguments, in the final chunk, whose chunk.finishReason is LlamaFinishReason.toolCalls. collect() returns them as LlamaToolCallContents with decoded arguments (empty when they are not a JSON object; rawJson keeps the generated text).
• ChatSession stores the assistant's tool calls in its history. session.create(const []) continues from the tool results without a new user turn.
• With engine.create, append the collected completion.message (a LlamaToolCallContent per call) and the tool messages to your own list before the next call.
• Cap the rounds: a model can keep calling tools.

LlamaToolResultContent.result may be any JSON-compatible value. Non-string results are JSON-encoded into the prompt; strings pass through unchanged. A message with several results is rendered as one tool message per result.

For an OpenAI-compatible reference, see OpenAI-compatible server.

Import a JSON Schema#

Use ToolDefinition.fromJsonSchema when another library already provides the input schema. The factory owns a deep copy; toJsonSchema() returns a fresh copy. Its parameters list is empty, so integrations must read toJsonSchema().

final ToolDefinition searchTool = ToolDefinition.fromJsonSchema(
  name: 'search',
  description: 'Search selected collections',
  schema: {
    'type': 'object',
    'properties': {
      'query': {'type': 'string', 'minLength': 1},
      'collections': {
        'type': 'array',
        'items': {'type': 'string'},
        'minItems': 1,
      },
    },
    'required': ['query', 'collections'],
    'additionalProperties': false,
  },
  handler: (params) async => {
    'query': params.getRequiredString('query'),
    'collections': params.getList('collections'),
  },
);

The initial subset supports scalar types, nested objects and arrays, scalar enum/const, nullable type lists, anyOf, string length and array item bounds, and boolean or schema-valued additionalProperties. The root must be an object. Missing additionalProperties means true; the existing ToolParam constructor keeps its closed-property behavior.

References (local and remote), patterns, numeric bounds, oneOf/allOf, tuple arrays, container literals and unsupported constraint combinations throw LlamaUnsupportedException. Malformed schemas throw LlamaArgumentException. This API supports a bounded subset of JSON Schema.

toJsonSchema() and toJson() return the validated schema with object defaults made explicit: every object schema includes properties (empty when absent) and additionalProperties (true when absent).

Limits#

Imported schemas may come from untrusted sources, so the factory bounds them and throws LlamaUnsupportedException, naming the keyword or path and the limit, when a schema exceeds one:

The bound limit is llama.cpp's grammar repetition limit (MAX_REPETITION_THRESHOLD, 2000): the grammar parser of the llama.cpp release the native runtime ships (upstream v0.6.0) rejects a repetition count above 2000 and silently drops a larger upper bound.

The grammar for all tools in a request is checked again before generation on grammar-capable backends. It throws LlamaUnsupportedException when llama.cpp would refuse to load it: a repeated group whose symbols times its repetition count exceed 2000. Hermes, for example, puts every schema tool inside one repeated group, so about 2000 schema tools exceed it.

Active schema tools require grammar-constrained decoding. Native llama.cpp supports it; WebGPU works only when the selected grammar starts at the first token (usually ToolChoice.required). Lazy auto grammars and LiteRT-LM are rejected. ToolChoice.none can retain declarations for replay without enabling calls. Final parsed calls and direct invoke arguments are validated before handlers run. A schema constrains the input shape; the app still controls tool permissions and side effects.

Active schema tools cannot be combined with responseFormat, which would replace the tool grammar. Use separate requests. Undeclared calls and malformed tool envelopes fail the turn before handlers run. In an open object, an extra property cannot reuse a declared property name, so declared properties keep their own schema. The grammar does not exclude escaped spellings of a declared name (such as \u0063ity); final validation checks the decoded values against the schema. Where a declared name needs a JSON escape (such as a"b), the grammar also rejects extra names with a different escape at that position (such as a\nb). Open objects whose declared names would add more than 65,536 characters of exclusion rules to one grammar accept any extra name instead; for tools, final validation still checks declared properties.

In Hermes auto mode, a JSON object with a name key outside a <tool_call>, <tool> or <function…> envelope stays answer text unless it is a complete call to a declared tool, which is then validated like any other call. A reply cut off by maxTokens or the context size finishes with length (tool loops stop with truncated) rather than failing as a malformed call.

Known gaps: templates that force reasoning open (such as QwQ) cannot emit reasoning before a ToolChoice.required call, as with ToolParam tools; and required may only name declared properties.

On WebGPU, an automatic tool loop changes to auto after its first tool round. Hermes and Mistral then need lazy grammars, so their schema-backed automatic loops cannot finish on WebGPU. Use the generic non-lazy JSON format there, or manage a single required-tool request and its follow-up explicitly.

Generic JSON fallback#

For templates without a recognized tool-call format, and for Gemma 2/3/3n, llamadart keeps its generic JSON fallback when tools are passed with ToolChoice.auto or ToolChoice.required. It adds a JSON instruction, builds a grammar from the supplied tools for runtimes that support it, and parses output such as:

{"tool_call":{"name":"get_weather","arguments":{"city":"Paris"}}}

An ordinary answer uses {"response":"Hello!"}. The parser returns the answer or tool call through the same API as a model-specific format. Your application still decides whether to invoke the tool.

This fallback is an intentional extension to the tested llama.cpp versions (b10549 and v0.5.0, commit 7fe450e19). For the tested ChatML Hermes-3, SmolLM2 and SmolVLM templates, those versions render without tool definitions or a tool grammar and return JSON tool envelopes as ordinary text. Matching that behavior would remove llamadart's existing fallback. The template in the actual model file determines routing; a model family name alone does not.

What the fallback does not guarantee#

A valid JSON shape does not establish that a model chose the correct tool or arguments. Generic calling depends on instruction following and should be validated with the exact model and task.

Tool definitions are passed to the template, but a template that ignores its tools variable can omit names, descriptions and parameter documentation from the prompt. The generic instruction only explains the JSON envelope; it does not insert a tool catalog. The grammar can constrain names and argument shapes without explaining their meaning to the model. Inspect engine.chatTemplate(...); if the definitions are missing, provide them in your system instruction or use a template with native tool support.

ToolChoice.none disables the fallback tool instruction and tool grammar for these templates. Backend restrictions still apply: in particular, this fallback does not add grammar enforcement to LiteRT-LM. See Chat templates and output parsing for runtime routing and model families for formats.

Tool results on templates without a tool role#

Gemma 2/3/3n and older Mistral templates reject the tool role. When such a template fails to render a request with tool results, and accepts a tool result as a user message, llamadart renders the request again with each run of consecutive results as one user turn, such as:

{
  "tool_response": {
    "tool": "get_weather",
    "content": "{\"temperature_c\":21}",
    "tool_call_id": "call_1"
  }
}

The earlier tool calls appear as JSON in the model turn, with or without tools on the follow-up request. This restores the tool-result polyfill of llama.cpp's former minja engine (for example b6000); the tested b10549 fails the same request with the template's role-alternation error. A request that the template renders as given, and any template that renders the tool role itself, is unchanged. A user message directly after tool results still fails on these templates, as two user turns in a row (#1118).

Tool choice semantics#

• ToolChoice.none: disable tool calls for that request.
• ToolChoice.auto: model decides whether to call tools.
• ToolChoice.required: model must emit tool calls.

On Web with WebGPU (llama.cpp), the bridge applies a grammar from the first token and cannot wait for a tool-call trigger. ToolChoice.auto therefore skips a lazy tool-call grammar, and tool calls are parsed from the output best-effort. ToolChoice.required keeps a grammar that starts at the first token and fails early with LlamaUnsupportedException when the chat format's required-tool grammar is lazy.

Parser details, such as call validation against the supplied tools, are in Template engine internals.