Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s
A model caught in a repetition loop wrote until the flow's 300 s timeout. max_tokens bounds the answer on every wire (Ollama num_predict, OpenAI max_tokens, Anthropic max_tokens), and truncated tells the caller whether the answer stopped at that limit. repeat_penalty goes to Ollama and vLLM and is refused where the backend has no such setting. Signed-off-by: flemming-it <sf@flemming.it>
183 lines
8.6 KiB
Markdown
183 lines
8.6 KiB
Markdown
# llm.chat
|
|
|
|
Generic LLM chat adapter. The building-block module any flow
|
|
uses when it needs a one-shot LLM completion — classify,
|
|
extract-fields, rewrite, decide — instead of rolling its own
|
|
HTTP client. Speaks three wire formats, selected by the
|
|
optional `api` input: Ollama (default), OpenAI Chat
|
|
Completions (OpenAI, vLLM, compatible servers), and the
|
|
Anthropic Messages API.
|
|
|
|
## Capability
|
|
|
|
- `llm.chat@0.4.0`
|
|
|
|
## Inputs
|
|
|
|
| Name | Type | Description |
|
|
| ---------------- | ---- | --------------------------------------------------------------- |
|
|
| `prompt` | text | The user-facing prompt. |
|
|
| `endpoint` | text | Chat endpoint URL matching the selected `api` (see below). |
|
|
| `model` | text | Model identifier (e.g. `qwen2.5:14b`, `claude-fable-5`). |
|
|
| `api_key` | text | Optional API key. Never appears in outputs, logs, or events. |
|
|
| `keep_alive` | text | Ollama only: how long the server keeps the model resident after this request (`4h`, `30m`, `-1` = indefinitely). Empty = server default, which unloads after five minutes. Refused with any other `api`. |
|
|
| `think` | text | `true` or `false`: whether a reasoning model runs its thinking phase. Empty = server default (Ollama lets qwen3-type models think). Ollama gets `think`, vLLM `chat_template_kwargs.enable_thinking`; refused on the OpenAI wire; Anthropic takes `false` only. Measured on Ollama 0.40.9: `qwen3.8:27b` and `gemma4:26b` honour `false`; `qwen3:30b-a3b` thinks anyway and its thinking then arrives inside the reply, ending with `</think>`. |
|
|
| `max_tokens` | text | Upper bound on the answer length in tokens (whole number, at least 1). Ollama gets `options.num_predict`, the OpenAI wire `max_tokens`, Anthropic `max_tokens` (default 4096 when empty). Empty = no limit on Ollama and the OpenAI wire. Without a limit, a model caught in a repetition loop writes until the flow's timeout; with `num_predict: 400`, `qwen3.8:27b` on Ollama 0.40.9 stopped such a loop after 39 s. |
|
|
| `repeat_penalty` | text | Penalty on repeated tokens, 1.0 = neutral. Ollama gets `options.repeat_penalty`, vLLM `repetition_penalty`; refused on the OpenAI wire and for Anthropic. |
|
|
| `system_prompt` | text | Optional system message. Empty = use the model's default. |
|
|
| `api` | text | Optional wire format: `ollama` (default), `openai`, `anthropic`. |
|
|
|
|
### Wire formats (`api`)
|
|
|
|
| `api` | Endpoint shape | Auth header | Notes |
|
|
| ----------- | --------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ |
|
|
| `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | v0.1.0 behavior, unchanged. Model-digest probe via `/api/show`. |
|
|
| `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI Chat Completions format — also spoken by vLLM and most self-hosted inference servers. |
|
|
| `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` is a top-level field; `max_tokens` is mandatory; `max_tokens` sets it, default 4096. |
|
|
|
|
## Outputs
|
|
|
|
| Name | Type | Description |
|
|
| ---------------- | ---- | ------------------------------------------------------------------- |
|
|
| `response` | text | The assistant's plain-text reply. |
|
|
| `truncated` | text | `true` when the answer stopped at the length limit, `false` when the model finished. Empty when the backend gave no stop reason. A truncated answer is still returned; the flow decides what to do with it. |
|
|
| `stop_reason` | text | The backend's own reason: Ollama `done_reason`, OpenAI `finish_reason`, Anthropic `stop_reason`. Empty when absent. |
|
|
| `model_endpoint` | text | The endpoint the response was generated against. |
|
|
| `model_name` | text | The model identifier as supplied to the LLM API. |
|
|
| `model_digest` | text | SHA-256 digest of the served Ollama model. Empty for `openai`/`anthropic` — those APIs expose no digest; the field is left empty rather than fabricated. |
|
|
| `tokens_prompt` | text | What the prompt cost, as the backend counted it. Empty when the answer carried no count. |
|
|
| `tokens_completion` | text | What the answer cost, same rule. Ollama counts in `eval_count`, the OpenAI shape in `usage.completion_tokens`, Anthropic in `usage.output_tokens`. |
|
|
|
|
Together the three `model_*` outputs answer the audit
|
|
question: "which exact model produced this response?" — that
|
|
audit trail is the reason flows use this module rather than
|
|
spawning their own HTTP calls.
|
|
|
|
## Permissions
|
|
|
|
```yaml
|
|
permissions:
|
|
- "net: localhost"
|
|
- "net: 127.0.0.1"
|
|
- "net: api.openai.com"
|
|
- "net: api.anthropic.com"
|
|
```
|
|
|
|
Loopback (local Ollama / vLLM) by default. Cloud endpoints require an
|
|
operator-policy override in `~/.chain/config.yaml#security.max_permissions`.
|
|
|
|
## Why this module instead of inline HTTP
|
|
|
|
Three reasons:
|
|
|
|
1. **Audit.** Every LLM call surfaces `model_endpoint`,
|
|
`model_name`, `model_digest` as separate outputs that land
|
|
in the hash-chained audit log alongside the response. A
|
|
regulator-facing reproducibility check just compares the
|
|
digest field to the model snapshot.
|
|
2. **Permission posture.** The operator sees `llm.chat` in
|
|
the installed modules + its declared `net:` list. A
|
|
home-grown HTTP module would either hide its endpoints
|
|
or be a fresh review surface every time.
|
|
3. **Endpoint portability.** The endpoint, model, and wire
|
|
format are flow inputs, not compile-time constants. The
|
|
same flow runs against `localhost:11434` in dev and a
|
|
production vLLM cluster in prod just by swapping inputs.
|
|
|
|
## Limits in v0.2.0
|
|
|
|
- No streaming. The whole reply is buffered before the output
|
|
step fires.
|
|
- No tool-call / function-call surface. A flow needing tool
|
|
use composes multiple `llm.chat` steps with prompt
|
|
engineering, or uses MCP via the bridge.
|
|
- Anthropic `max_tokens` is fixed at 4096 (the API requires
|
|
the field; a configurable input lands when a flow needs it).
|
|
|
|
## Example flows
|
|
|
|
```yaml
|
|
name: classify-incoming
|
|
inputs:
|
|
text: text
|
|
steps:
|
|
- id: classify
|
|
use: llm.chat@^0
|
|
with:
|
|
prompt: $inputs.text
|
|
system_prompt: |
|
|
Classify the text as one of: question, complaint,
|
|
feedback, spam. Answer with the label only.
|
|
endpoint: "http://localhost:11434/api/chat"
|
|
model: "qwen2.5:14b"
|
|
outputs:
|
|
category: $classify.response
|
|
audit_model: $classify.model_digest
|
|
```
|
|
|
|
Against a vLLM (OpenAI-compatible) server:
|
|
|
|
```yaml
|
|
steps:
|
|
- id: classify
|
|
use: llm.chat@^0
|
|
with:
|
|
api: "openai"
|
|
endpoint: "http://inference.internal:8000/v1/chat/completions"
|
|
model: "meta-llama/Llama-3.1-8B-Instruct"
|
|
prompt: $inputs.text
|
|
```
|
|
|
|
Against the Anthropic Messages API:
|
|
|
|
```yaml
|
|
steps:
|
|
- id: classify
|
|
use: llm.chat@^0
|
|
with:
|
|
api: "anthropic"
|
|
endpoint: "https://api.anthropic.com/v1/messages"
|
|
model: "claude-fable-5"
|
|
api_key: $inputs.anthropic_key
|
|
prompt: $inputs.text
|
|
```
|
|
|
|
## Build
|
|
|
|
```bash
|
|
cargo build --release --target wasm32-wasip2
|
|
# Output: target/wasm32-wasip2/release/llm_chat.wasm
|
|
```
|
|
|
|
## Keeping a local model resident
|
|
|
|
Ollama unloads a model five minutes after its last request. The next
|
|
request then pays a full reload, which is 30 to 40 seconds for a large
|
|
local model: long enough that an operator assumes something is broken.
|
|
|
|
`keep_alive` sets that window per request:
|
|
|
|
```yaml
|
|
- id: answer
|
|
use: llm.chat@^0.2
|
|
with:
|
|
endpoint: http://localhost:11434/api/chat
|
|
model: qwen2.5:7b
|
|
keep_alive: "4h"
|
|
prompt: $inputs.question
|
|
```
|
|
|
|
Measured against a live Ollama (2026-10-07): without the input the
|
|
model expired 5:02 after the request; with `keep_alive: "90s"`, 92
|
|
seconds after. Note that Ollama keeps the value for the loaded
|
|
instance, so a later request that omits `keep_alive` does not fall
|
|
back to five minutes until the model has actually been unloaded.
|
|
|
|
`-1` is sent as a number, not a string, because Ollama only reads the
|
|
numeric form as "never unload".
|
|
|
|
Only Ollama has this setting. With `api: openai`, `vllm` or
|
|
`anthropic` the input is **refused rather than ignored**: it expresses
|
|
an expectation about the server, and silently dropping it would leave
|
|
you paying the reload while believing it was handled. For vLLM and
|
|
OpenAI-compatible servers, residency is configured on the server.
|