Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s
Ollama unloads a model five minutes after its last request, and the next request then pays a full reload: 30 to 40 seconds for a large local model, long enough that an operator assumes something is broken. The stromnetz demonstrator runs on a notebook from around 12.10. and hits exactly that after any idle period; it works around it today by setting OLLAMA_KEEP_ALIVE on the server, which is not expressible in a flow. Measured against a live Ollama: without the input the model expired 5:02 after the request, with `keep_alive: "90s"` it expired after 92 seconds. Worth knowing for callers: Ollama keeps the value for the loaded instance, so a later request that omits keep_alive does not fall back to five minutes until the model has actually been unloaded. `-1` is sent as a number rather than a string, because Ollama reads only the numeric form as "never unload"; quoted it is parsed as a duration and the setting silently does nothing. With any other wire format the input is REFUSED, not ignored. It expresses an expectation about the server, and dropping it quietly would leave the caller paying the reload while believing it was handled. The message says what to do instead. Capability goes 0.2.0 to 0.2.1: the addition is optional and additive, so `@^0.2` and `@^0` both keep resolving, and an unset value leaves the request byte-identical to the previous version. Five guards, including that unset omits the field entirely and that the other wire formats never carry it. Mutation-checked three ways. Signed-off-by: flemming-it <sf@flemming.it>
176 lines
6.9 KiB
Markdown
176 lines
6.9 KiB
Markdown
# llm.chat
|
|
|
|
Generic LLM chat adapter. The building-block module any flow
|
|
uses when it needs a one-shot LLM completion — classify,
|
|
extract-fields, rewrite, decide — instead of rolling its own
|
|
HTTP client. Speaks three wire formats, selected by the
|
|
optional `api` input: Ollama (default), OpenAI Chat
|
|
Completions (OpenAI, vLLM, compatible servers), and the
|
|
Anthropic Messages API.
|
|
|
|
## Capability
|
|
|
|
- `llm.chat@0.2.0`
|
|
|
|
## Inputs
|
|
|
|
| Name | Type | Description |
|
|
| ---------------- | ---- | --------------------------------------------------------------- |
|
|
| `prompt` | text | The user-facing prompt. |
|
|
| `endpoint` | text | Chat endpoint URL matching the selected `api` (see below). |
|
|
| `model` | text | Model identifier (e.g. `qwen2.5:14b`, `claude-fable-5`). |
|
|
| `api_key` | text | Optional API key. Never appears in outputs, logs, or events. |
|
|
| `keep_alive` | text | Ollama only: how long the server keeps the model resident after this request (`4h`, `30m`, `-1` = indefinitely). Empty = server default, which unloads after five minutes. Refused with any other `api`. |
|
|
| `system_prompt` | text | Optional system message. Empty = use the model's default. |
|
|
| `api` | text | Optional wire format: `ollama` (default), `openai`, `anthropic`. |
|
|
|
|
### Wire formats (`api`)
|
|
|
|
| `api` | Endpoint shape | Auth header | Notes |
|
|
| ----------- | --------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ |
|
|
| `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | v0.1.0 behavior, unchanged. Model-digest probe via `/api/show`. |
|
|
| `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI Chat Completions format — also spoken by vLLM and most self-hosted inference servers. |
|
|
| `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` is a top-level field; `max_tokens` is mandatory and defaults to 4096. |
|
|
|
|
## Outputs
|
|
|
|
| Name | Type | Description |
|
|
| ---------------- | ---- | ------------------------------------------------------------------- |
|
|
| `response` | text | The assistant's plain-text reply. |
|
|
| `model_endpoint` | text | The endpoint the response was generated against. |
|
|
| `model_name` | text | The model identifier as supplied to the LLM API. |
|
|
| `model_digest` | text | SHA-256 digest of the served Ollama model. Empty for `openai`/`anthropic` — those APIs expose no digest; the field is left empty rather than fabricated. |
|
|
|
|
Together the three `model_*` outputs answer the audit
|
|
question: "which exact model produced this response?" — that
|
|
audit trail is the reason flows use this module rather than
|
|
spawning their own HTTP calls.
|
|
|
|
## Permissions
|
|
|
|
```yaml
|
|
permissions:
|
|
- "net: localhost"
|
|
- "net: 127.0.0.1"
|
|
- "net: api.openai.com"
|
|
- "net: api.anthropic.com"
|
|
```
|
|
|
|
Loopback (local Ollama / vLLM) by default. Cloud endpoints require an
|
|
operator-policy override in `~/.chain/config.yaml#security.max_permissions`.
|
|
|
|
## Why this module instead of inline HTTP
|
|
|
|
Three reasons:
|
|
|
|
1. **Audit.** Every LLM call surfaces `model_endpoint`,
|
|
`model_name`, `model_digest` as separate outputs that land
|
|
in the hash-chained audit log alongside the response. A
|
|
regulator-facing reproducibility check just compares the
|
|
digest field to the model snapshot.
|
|
2. **Permission posture.** The operator sees `llm.chat` in
|
|
the installed modules + its declared `net:` list. A
|
|
home-grown HTTP module would either hide its endpoints
|
|
or be a fresh review surface every time.
|
|
3. **Endpoint portability.** The endpoint, model, and wire
|
|
format are flow inputs, not compile-time constants. The
|
|
same flow runs against `localhost:11434` in dev and a
|
|
production vLLM cluster in prod just by swapping inputs.
|
|
|
|
## Limits in v0.2.0
|
|
|
|
- No streaming. The whole reply is buffered before the output
|
|
step fires.
|
|
- No tool-call / function-call surface. A flow needing tool
|
|
use composes multiple `llm.chat` steps with prompt
|
|
engineering, or uses MCP via the bridge.
|
|
- Anthropic `max_tokens` is fixed at 4096 (the API requires
|
|
the field; a configurable input lands when a flow needs it).
|
|
|
|
## Example flows
|
|
|
|
```yaml
|
|
name: classify-incoming
|
|
inputs:
|
|
text: text
|
|
steps:
|
|
- id: classify
|
|
use: llm.chat@^0
|
|
with:
|
|
prompt: $inputs.text
|
|
system_prompt: |
|
|
Classify the text as one of: question, complaint,
|
|
feedback, spam. Answer with the label only.
|
|
endpoint: "http://localhost:11434/api/chat"
|
|
model: "qwen2.5:14b"
|
|
outputs:
|
|
category: $classify.response
|
|
audit_model: $classify.model_digest
|
|
```
|
|
|
|
Against a vLLM (OpenAI-compatible) server:
|
|
|
|
```yaml
|
|
steps:
|
|
- id: classify
|
|
use: llm.chat@^0
|
|
with:
|
|
api: "openai"
|
|
endpoint: "http://inference.internal:8000/v1/chat/completions"
|
|
model: "meta-llama/Llama-3.1-8B-Instruct"
|
|
prompt: $inputs.text
|
|
```
|
|
|
|
Against the Anthropic Messages API:
|
|
|
|
```yaml
|
|
steps:
|
|
- id: classify
|
|
use: llm.chat@^0
|
|
with:
|
|
api: "anthropic"
|
|
endpoint: "https://api.anthropic.com/v1/messages"
|
|
model: "claude-fable-5"
|
|
api_key: $inputs.anthropic_key
|
|
prompt: $inputs.text
|
|
```
|
|
|
|
## Build
|
|
|
|
```bash
|
|
cargo build --release --target wasm32-wasip2
|
|
# Output: target/wasm32-wasip2/release/llm_chat.wasm
|
|
```
|
|
|
|
## Keeping a local model resident
|
|
|
|
Ollama unloads a model five minutes after its last request. The next
|
|
request then pays a full reload, which is 30 to 40 seconds for a large
|
|
local model: long enough that an operator assumes something is broken.
|
|
|
|
`keep_alive` sets that window per request:
|
|
|
|
```yaml
|
|
- id: answer
|
|
use: llm.chat@^0.2
|
|
with:
|
|
endpoint: http://localhost:11434/api/chat
|
|
model: qwen2.5:7b
|
|
keep_alive: "4h"
|
|
prompt: $inputs.question
|
|
```
|
|
|
|
Measured against a live Ollama (2026-10-07): without the input the
|
|
model expired 5:02 after the request; with `keep_alive: "90s"`, 92
|
|
seconds after. Note that Ollama keeps the value for the loaded
|
|
instance, so a later request that omits `keep_alive` does not fall
|
|
back to five minutes until the model has actually been unloaded.
|
|
|
|
`-1` is sent as a number, not a string, because Ollama only reads the
|
|
numeric form as "never unload".
|
|
|
|
Only Ollama has this setting. With `api: openai`, `vllm` or
|
|
`anthropic` the input is **refused rather than ignored**: it expresses
|
|
an expectation about the server, and silently dropping it would leave
|
|
you paying the reload while believing it was handled. For vLLM and
|
|
OpenAI-compatible servers, residency is configured on the server.
|