feat: max_tokens and repeat_penalty inputs, truncated and stop_reason outputs, 0.4.0
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s

A model caught in a repetition loop wrote until the flow's 300 s
timeout. max_tokens bounds the answer on every wire (Ollama
num_predict, OpenAI max_tokens, Anthropic max_tokens), and
truncated tells the caller whether the answer stopped at that
limit. repeat_penalty goes to Ollama and vLLM and is refused where
the backend has no such setting.

Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
flemming-it 2026-11-12 15:49:50 +01:00
parent 14c63083d7
commit 72f3037ebe
8 changed files with 361 additions and 9 deletions

View file

@ -10,7 +10,7 @@ Anthropic Messages API.
## Capability
- `llm.chat@0.3.0`
- `llm.chat@0.4.0`
## Inputs
@ -22,6 +22,8 @@ Anthropic Messages API.
| `api_key` | text | Optional API key. Never appears in outputs, logs, or events. |
| `keep_alive` | text | Ollama only: how long the server keeps the model resident after this request (`4h`, `30m`, `-1` = indefinitely). Empty = server default, which unloads after five minutes. Refused with any other `api`. |
| `think` | text | `true` or `false`: whether a reasoning model runs its thinking phase. Empty = server default (Ollama lets qwen3-type models think). Ollama gets `think`, vLLM `chat_template_kwargs.enable_thinking`; refused on the OpenAI wire; Anthropic takes `false` only. Measured on Ollama 0.40.9: `qwen3.8:27b` and `gemma4:26b` honour `false`; `qwen3:30b-a3b` thinks anyway and its thinking then arrives inside the reply, ending with `</think>`. |
| `max_tokens` | text | Upper bound on the answer length in tokens (whole number, at least 1). Ollama gets `options.num_predict`, the OpenAI wire `max_tokens`, Anthropic `max_tokens` (default 4096 when empty). Empty = no limit on Ollama and the OpenAI wire. Without a limit, a model caught in a repetition loop writes until the flow's timeout; with `num_predict: 400`, `qwen3.8:27b` on Ollama 0.40.9 stopped such a loop after 39 s. |
| `repeat_penalty` | text | Penalty on repeated tokens, 1.0 = neutral. Ollama gets `options.repeat_penalty`, vLLM `repetition_penalty`; refused on the OpenAI wire and for Anthropic. |
| `system_prompt` | text | Optional system message. Empty = use the model's default. |
| `api` | text | Optional wire format: `ollama` (default), `openai`, `anthropic`. |
@ -31,13 +33,15 @@ Anthropic Messages API.
| ----------- | --------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ |
| `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | v0.1.0 behavior, unchanged. Model-digest probe via `/api/show`. |
| `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI Chat Completions format — also spoken by vLLM and most self-hosted inference servers. |
| `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` is a top-level field; `max_tokens` is mandatory and defaults to 4096. |
| `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` is a top-level field; `max_tokens` is mandatory; `max_tokens` sets it, default 4096. |
## Outputs
| Name | Type | Description |
| ---------------- | ---- | ------------------------------------------------------------------- |
| `response` | text | The assistant's plain-text reply. |
| `truncated` | text | `true` when the answer stopped at the length limit, `false` when the model finished. Empty when the backend gave no stop reason. A truncated answer is still returned; the flow decides what to do with it. |
| `stop_reason` | text | The backend's own reason: Ollama `done_reason`, OpenAI `finish_reason`, Anthropic `stop_reason`. Empty when absent. |
| `model_endpoint` | text | The endpoint the response was generated against. |
| `model_name` | text | The model identifier as supplied to the LLM API. |
| `model_digest` | text | SHA-256 digest of the served Ollama model. Empty for `openai`/`anthropic` — those APIs expose no digest; the field is left empty rather than fabricated. |