feat: max_tokens and repeat_penalty inputs, truncated and stop_reason outputs, 0.4.0
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s
A model caught in a repetition loop wrote until the flow's 300 s timeout. max_tokens bounds the answer on every wire (Ollama num_predict, OpenAI max_tokens, Anthropic max_tokens), and truncated tells the caller whether the answer stopped at that limit. repeat_penalty goes to Ollama and vLLM and is refused where the backend has no such setting. Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
parent
14c63083d7
commit
72f3037ebe
8 changed files with 361 additions and 9 deletions
63
module.yaml
63
module.yaml
|
|
@ -1,12 +1,12 @@
|
|||
schema_version: 3
|
||||
provider: chain
|
||||
name: llm-chat
|
||||
version: 0.3.1
|
||||
version: 0.4.0
|
||||
|
||||
# Capability provided by this module.
|
||||
provides:
|
||||
- capability: llm.chat
|
||||
version: 0.3.1
|
||||
version: 0.4.0
|
||||
|
||||
# Declared inputs in verbose v3 form. Studio renders the
|
||||
# description in the active locale as a tooltip on each
|
||||
|
|
@ -108,6 +108,39 @@ inputs:
|
|||
das einen Lauf wiederholbar. Nur Ollama und das
|
||||
OpenAI-Format; Anthropic kennt keinen Seed und lehnt den
|
||||
Eingang ab.
|
||||
max_tokens:
|
||||
type: text
|
||||
description:
|
||||
en: |
|
||||
Upper bound on the answer length in tokens, a whole number of
|
||||
at least 1. A model caught in a repetition loop otherwise
|
||||
writes until the flow's timeout. Ollama receives
|
||||
options.num_predict, the OpenAI wire max_tokens, Anthropic
|
||||
max_tokens (default 4096 there when empty). Empty leaves no
|
||||
limit on Ollama and the OpenAI wire. Whether the answer
|
||||
stopped at the limit is reported in the output truncated.
|
||||
de: |
|
||||
Obergrenze für die Länge der Antwort in Token, ganze Zahl ab
|
||||
1. Ein Modell in einer Wiederholungsschleife schreibt sonst
|
||||
bis zur Zeitgrenze des Ablaufs. Ollama bekommt
|
||||
options.num_predict, das OpenAI-Format max_tokens, Anthropic
|
||||
max_tokens (dort 4096, wenn leer). Leer heißt bei Ollama und
|
||||
im OpenAI-Format keine Grenze. Ob die Antwort an der Grenze
|
||||
endete, steht in der Ausgabe truncated.
|
||||
repeat_penalty:
|
||||
type: text
|
||||
description:
|
||||
en: |
|
||||
Penalty on repeated tokens, a number; 1.0 is neutral, values
|
||||
such as 1.1 make loops less likely. Ollama receives
|
||||
options.repeat_penalty, vLLM repetition_penalty. The OpenAI
|
||||
wire and Anthropic have no such setting and refuse the input.
|
||||
de: |
|
||||
Abschlag auf wiederholte Token, eine Zahl; 1.0 ist neutral,
|
||||
Werte wie 1.1 machen Schleifen seltener. Ollama bekommt
|
||||
options.repeat_penalty, vLLM repetition_penalty. Das
|
||||
OpenAI-Format und Anthropic kennen die Einstellung nicht und
|
||||
lehnen den Eingang ab.
|
||||
keep_alive:
|
||||
type: text
|
||||
description:
|
||||
|
|
@ -167,6 +200,32 @@ outputs:
|
|||
description:
|
||||
en: The assistant's plain-text reply.
|
||||
de: Die Klartext-Antwort des Modells.
|
||||
truncated:
|
||||
type: text
|
||||
description:
|
||||
en: |
|
||||
"true" when the answer stopped at the length limit
|
||||
(max_tokens) instead of where the model finished, "false"
|
||||
when the model finished. Empty when the backend gave no
|
||||
stop reason.
|
||||
de: |
|
||||
"true", wenn die Antwort an der Längengrenze (max_tokens)
|
||||
endete statt dort, wo das Modell fertig war, "false", wenn
|
||||
das Modell fertig war. Leer, wenn das Backend keinen Grund
|
||||
für das Ende nannte.
|
||||
stop_reason:
|
||||
type: text
|
||||
description:
|
||||
en: |
|
||||
Why the backend stopped writing, in its own words: Ollama
|
||||
done_reason (stop, length), the OpenAI wire finish_reason
|
||||
(stop, length), Anthropic stop_reason (end_turn,
|
||||
max_tokens, ...). Empty when the answer did not say.
|
||||
de: |
|
||||
Warum das Backend aufhörte, in seinen eigenen Worten: Ollama
|
||||
done_reason (stop, length), das OpenAI-Format finish_reason
|
||||
(stop, length), Anthropic stop_reason (end_turn, max_tokens
|
||||
u. a.). Leer, wenn die Antwort es nicht angab.
|
||||
model_endpoint:
|
||||
type: text
|
||||
description:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue