llm-chat/MODULE.md
flemming-it 72f3037ebe
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s
feat: max_tokens and repeat_penalty inputs, truncated and stop_reason outputs, 0.4.0
A model caught in a repetition loop wrote until the flow's 300 s
timeout. max_tokens bounds the answer on every wire (Ollama
num_predict, OpenAI max_tokens, Anthropic max_tokens), and
truncated tells the caller whether the answer stopped at that
limit. repeat_penalty goes to Ollama and vLLM and is refused where
the backend has no such setting.

Signed-off-by: flemming-it <sf@flemming.it>
2026-11-12 15:49:50 +01:00

8.6 KiB

llm.chat

Generic LLM chat adapter. The building-block module any flow uses when it needs a one-shot LLM completion — classify, extract-fields, rewrite, decide — instead of rolling its own HTTP client. Speaks three wire formats, selected by the optional api input: Ollama (default), OpenAI Chat Completions (OpenAI, vLLM, compatible servers), and the Anthropic Messages API.

Capability

  • llm.chat@0.4.0

Inputs

Name Type Description
prompt text The user-facing prompt.
endpoint text Chat endpoint URL matching the selected api (see below).
model text Model identifier (e.g. qwen2.5:14b, claude-fable-5).
api_key text Optional API key. Never appears in outputs, logs, or events.
keep_alive text Ollama only: how long the server keeps the model resident after this request (4h, 30m, -1 = indefinitely). Empty = server default, which unloads after five minutes. Refused with any other api.
think text true or false: whether a reasoning model runs its thinking phase. Empty = server default (Ollama lets qwen3-type models think). Ollama gets think, vLLM chat_template_kwargs.enable_thinking; refused on the OpenAI wire; Anthropic takes false only. Measured on Ollama 0.40.9: qwen3.8:27b and gemma4:26b honour false; qwen3:30b-a3b thinks anyway and its thinking then arrives inside the reply, ending with </think>.
max_tokens text Upper bound on the answer length in tokens (whole number, at least 1). Ollama gets options.num_predict, the OpenAI wire max_tokens, Anthropic max_tokens (default 4096 when empty). Empty = no limit on Ollama and the OpenAI wire. Without a limit, a model caught in a repetition loop writes until the flow's timeout; with num_predict: 400, qwen3.8:27b on Ollama 0.40.9 stopped such a loop after 39 s.
repeat_penalty text Penalty on repeated tokens, 1.0 = neutral. Ollama gets options.repeat_penalty, vLLM repetition_penalty; refused on the OpenAI wire and for Anthropic.
system_prompt text Optional system message. Empty = use the model's default.
api text Optional wire format: ollama (default), openai, anthropic.

Wire formats (api)

api Endpoint shape Auth header Notes
ollama http://localhost:11434/api/chat Authorization: Bearer (optional) v0.1.0 behavior, unchanged. Model-digest probe via /api/show.
openai https://.../v1/chat/completions Authorization: Bearer OpenAI Chat Completions format — also spoken by vLLM and most self-hosted inference servers.
anthropic https://api.anthropic.com/v1/messages x-api-key + anthropic-version Messages API: system is a top-level field; max_tokens is mandatory; max_tokens sets it, default 4096.

Outputs

Name Type Description
response text The assistant's plain-text reply.
truncated text true when the answer stopped at the length limit, false when the model finished. Empty when the backend gave no stop reason. A truncated answer is still returned; the flow decides what to do with it.
stop_reason text The backend's own reason: Ollama done_reason, OpenAI finish_reason, Anthropic stop_reason. Empty when absent.
model_endpoint text The endpoint the response was generated against.
model_name text The model identifier as supplied to the LLM API.
model_digest text SHA-256 digest of the served Ollama model. Empty for openai/anthropic — those APIs expose no digest; the field is left empty rather than fabricated.
tokens_prompt text What the prompt cost, as the backend counted it. Empty when the answer carried no count.
tokens_completion text What the answer cost, same rule. Ollama counts in eval_count, the OpenAI shape in usage.completion_tokens, Anthropic in usage.output_tokens.

Together the three model_* outputs answer the audit question: "which exact model produced this response?" — that audit trail is the reason flows use this module rather than spawning their own HTTP calls.

Permissions

permissions:
  - "net: localhost"
  - "net: 127.0.0.1"
  - "net: api.openai.com"
  - "net: api.anthropic.com"

Loopback (local Ollama / vLLM) by default. Cloud endpoints require an operator-policy override in ~/.chain/config.yaml#security.max_permissions.

Why this module instead of inline HTTP

Three reasons:

  1. Audit. Every LLM call surfaces model_endpoint, model_name, model_digest as separate outputs that land in the hash-chained audit log alongside the response. A regulator-facing reproducibility check just compares the digest field to the model snapshot.
  2. Permission posture. The operator sees llm.chat in the installed modules + its declared net: list. A home-grown HTTP module would either hide its endpoints or be a fresh review surface every time.
  3. Endpoint portability. The endpoint, model, and wire format are flow inputs, not compile-time constants. The same flow runs against localhost:11434 in dev and a production vLLM cluster in prod just by swapping inputs.

Limits in v0.2.0

  • No streaming. The whole reply is buffered before the output step fires.
  • No tool-call / function-call surface. A flow needing tool use composes multiple llm.chat steps with prompt engineering, or uses MCP via the bridge.
  • Anthropic max_tokens is fixed at 4096 (the API requires the field; a configurable input lands when a flow needs it).

Example flows

name: classify-incoming
inputs:
  text: text
steps:
  - id: classify
    use: llm.chat@^0
    with:
      prompt: $inputs.text
      system_prompt: |
        Classify the text as one of: question, complaint,
        feedback, spam. Answer with the label only.
      endpoint: "http://localhost:11434/api/chat"
      model: "qwen2.5:14b"
outputs:
  category: $classify.response
  audit_model: $classify.model_digest

Against a vLLM (OpenAI-compatible) server:

steps:
  - id: classify
    use: llm.chat@^0
    with:
      api: "openai"
      endpoint: "http://inference.internal:8000/v1/chat/completions"
      model: "meta-llama/Llama-3.1-8B-Instruct"
      prompt: $inputs.text

Against the Anthropic Messages API:

steps:
  - id: classify
    use: llm.chat@^0
    with:
      api: "anthropic"
      endpoint: "https://api.anthropic.com/v1/messages"
      model: "claude-fable-5"
      api_key: $inputs.anthropic_key
      prompt: $inputs.text

Build

cargo build --release --target wasm32-wasip2
# Output: target/wasm32-wasip2/release/llm_chat.wasm

Keeping a local model resident

Ollama unloads a model five minutes after its last request. The next request then pays a full reload, which is 30 to 40 seconds for a large local model: long enough that an operator assumes something is broken.

keep_alive sets that window per request:

- id: answer
  use: llm.chat@^0.2
  with:
    endpoint: http://localhost:11434/api/chat
    model: qwen2.5:7b
    keep_alive: "4h"
    prompt: $inputs.question

Measured against a live Ollama (2026-10-07): without the input the model expired 5:02 after the request; with keep_alive: "90s", 92 seconds after. Note that Ollama keeps the value for the loaded instance, so a later request that omits keep_alive does not fall back to five minutes until the model has actually been unloaded.

-1 is sent as a number, not a string, because Ollama only reads the numeric form as "never unload".

Only Ollama has this setting. With api: openai, vllm or anthropic the input is refused rather than ignored: it expresses an expectation about the server, and silently dropping it would leave you paying the reload while believing it was handled. For vLLM and OpenAI-compatible servers, residency is configured on the server.