feat: max_tokens and repeat_penalty inputs, truncated and stop_reason outputs, 0.4.0
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s

A model caught in a repetition loop wrote until the flow's 300 s
timeout. max_tokens bounds the answer on every wire (Ollama
num_predict, OpenAI max_tokens, Anthropic max_tokens), and
truncated tells the caller whether the answer stopped at that
limit. repeat_penalty goes to Ollama and vLLM and is refused where
the backend has no such setting.

Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
flemming-it 2026-11-12 15:49:50 +01:00
parent 14c63083d7
commit 72f3037ebe
8 changed files with 361 additions and 9 deletions

View file

@ -1,12 +1,12 @@
schema_version: 3
provider: chain
name: llm-chat
version: 0.3.1
version: 0.4.0
# Capability provided by this module.
provides:
- capability: llm.chat
version: 0.3.1
version: 0.4.0
# Declared inputs in verbose v3 form. Studio renders the
# description in the active locale as a tooltip on each
@ -108,6 +108,39 @@ inputs:
das einen Lauf wiederholbar. Nur Ollama und das
OpenAI-Format; Anthropic kennt keinen Seed und lehnt den
Eingang ab.
max_tokens:
type: text
description:
en: |
Upper bound on the answer length in tokens, a whole number of
at least 1. A model caught in a repetition loop otherwise
writes until the flow's timeout. Ollama receives
options.num_predict, the OpenAI wire max_tokens, Anthropic
max_tokens (default 4096 there when empty). Empty leaves no
limit on Ollama and the OpenAI wire. Whether the answer
stopped at the limit is reported in the output truncated.
de: |
Obergrenze für die Länge der Antwort in Token, ganze Zahl ab
1. Ein Modell in einer Wiederholungsschleife schreibt sonst
bis zur Zeitgrenze des Ablaufs. Ollama bekommt
options.num_predict, das OpenAI-Format max_tokens, Anthropic
max_tokens (dort 4096, wenn leer). Leer heißt bei Ollama und
im OpenAI-Format keine Grenze. Ob die Antwort an der Grenze
endete, steht in der Ausgabe truncated.
repeat_penalty:
type: text
description:
en: |
Penalty on repeated tokens, a number; 1.0 is neutral, values
such as 1.1 make loops less likely. Ollama receives
options.repeat_penalty, vLLM repetition_penalty. The OpenAI
wire and Anthropic have no such setting and refuse the input.
de: |
Abschlag auf wiederholte Token, eine Zahl; 1.0 ist neutral,
Werte wie 1.1 machen Schleifen seltener. Ollama bekommt
options.repeat_penalty, vLLM repetition_penalty. Das
OpenAI-Format und Anthropic kennen die Einstellung nicht und
lehnen den Eingang ab.
keep_alive:
type: text
description:
@ -167,6 +200,32 @@ outputs:
description:
en: The assistant's plain-text reply.
de: Die Klartext-Antwort des Modells.
truncated:
type: text
description:
en: |
"true" when the answer stopped at the length limit
(max_tokens) instead of where the model finished, "false"
when the model finished. Empty when the backend gave no
stop reason.
de: |
"true", wenn die Antwort an der Längengrenze (max_tokens)
endete statt dort, wo das Modell fertig war, "false", wenn
das Modell fertig war. Leer, wenn das Backend keinen Grund
für das Ende nannte.
stop_reason:
type: text
description:
en: |
Why the backend stopped writing, in its own words: Ollama
done_reason (stop, length), the OpenAI wire finish_reason
(stop, length), Anthropic stop_reason (end_turn,
max_tokens, ...). Empty when the answer did not say.
de: |
Warum das Backend aufhörte, in seinen eigenen Worten: Ollama
done_reason (stop, length), das OpenAI-Format finish_reason
(stop, length), Anthropic stop_reason (end_turn, max_tokens
u. a.). Leer, wenn die Antwort es nicht angab.
model_endpoint:
type: text
description: