feat: max_tokens and repeat_penalty inputs, truncated and stop_reason outputs, 0.4.0
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1s

A model caught in a repetition loop wrote until the flow's 300 s
timeout. max_tokens bounds the answer on every wire (Ollama
num_predict, OpenAI max_tokens, Anthropic max_tokens), and
truncated tells the caller whether the answer stopped at that
limit. repeat_penalty goes to Ollama and vLLM and is refused where
the backend has no such setting.

Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
flemming-it 2026-11-12 15:49:50 +01:00
parent 14c63083d7
commit 72f3037ebe
8 changed files with 361 additions and 9 deletions

2
Cargo.lock generated
View file

@ -138,7 +138,7 @@ checksum = "6cc46bac87ef8093eed6f272babb833b6443374399985ac8ed28471ee0918545"
[[package]] [[package]]
name = "llm_chat" name = "llm_chat"
version = "0.3.1" version = "0.4.0"
dependencies = [ dependencies = [
"chain-module-sdk", "chain-module-sdk",
"serde", "serde",

View file

@ -5,7 +5,7 @@
[package] [package]
name = "llm_chat" name = "llm_chat"
version = "0.3.1" version = "0.4.0"
edition = "2024" edition = "2024"
license = "Apache-2.0" license = "Apache-2.0"
publish = false publish = false

View file

@ -10,7 +10,7 @@ die Anthropic Messages API.
## Capability ## Capability
- `llm.chat@0.3.0` - `llm.chat@0.4.0`
## Eingaben ## Eingaben
@ -22,6 +22,8 @@ die Anthropic Messages API.
| `api_key` | text | Optionaler API-Key. Erscheint nie in Ausgaben, Logs oder Events. | | `api_key` | text | Optionaler API-Key. Erscheint nie in Ausgaben, Logs oder Events. |
| `keep_alive` | text | Nur Ollama: wie lange der Server das Modell nach dieser Anfrage geladen lässt (`4h`, `30m`, `-1` = unbegrenzt). Leer = Server-Voreinstellung, die nach fünf Minuten entlädt. Mit anderer `api` abgelehnt. | | `keep_alive` | text | Nur Ollama: wie lange der Server das Modell nach dieser Anfrage geladen lässt (`4h`, `30m`, `-1` = unbegrenzt). Leer = Server-Voreinstellung, die nach fünf Minuten entlädt. Mit anderer `api` abgelehnt. |
| `think` | text | `true` oder `false`: ob ein Reasoning-Modell seine Denkphase durchläuft. Leer = Voreinstellung des Servers (Ollama lässt Modelle wie qwen3 denken). Ollama bekommt `think`, vLLM `chat_template_kwargs.enable_thinking`; im OpenAI-Format abgelehnt; Anthropic nimmt nur `false`. Gemessen mit Ollama 0.40.9: `qwen3.8:27b` und `gemma4:26b` halten `false` ein; `qwen3:30b-a3b` denkt trotzdem, das Denken steht dann in der Antwort und endet mit `</think>`. | | `think` | text | `true` oder `false`: ob ein Reasoning-Modell seine Denkphase durchläuft. Leer = Voreinstellung des Servers (Ollama lässt Modelle wie qwen3 denken). Ollama bekommt `think`, vLLM `chat_template_kwargs.enable_thinking`; im OpenAI-Format abgelehnt; Anthropic nimmt nur `false`. Gemessen mit Ollama 0.40.9: `qwen3.8:27b` und `gemma4:26b` halten `false` ein; `qwen3:30b-a3b` denkt trotzdem, das Denken steht dann in der Antwort und endet mit `</think>`. |
| `max_tokens` | text | Obergrenze für die Länge der Antwort in Token (ganze Zahl ab 1). Ollama bekommt `options.num_predict`, das OpenAI-Format `max_tokens`, Anthropic `max_tokens` (4096, wenn leer). Leer = keine Grenze bei Ollama und im OpenAI-Format. Ohne Grenze schreibt ein Modell in einer Wiederholungsschleife bis zur Zeitgrenze des Ablaufs; mit `num_predict: 400` brach `qwen3.8:27b` unter Ollama 0.40.9 eine solche Schleife nach 39 s ab. |
| `repeat_penalty` | text | Abschlag auf wiederholte Token, 1.0 = neutral. Ollama bekommt `options.repeat_penalty`, vLLM `repetition_penalty`; im OpenAI-Format und bei Anthropic abgelehnt. |
| `system_prompt` | text | Optionale System-Nachricht. Leer = Default des Modells. | | `system_prompt` | text | Optionale System-Nachricht. Leer = Default des Modells. |
| `api` | text | Optionales Wire-Format: `ollama` (Default), `openai`, `anthropic`. | | `api` | text | Optionales Wire-Format: `ollama` (Default), `openai`, `anthropic`. |
@ -31,13 +33,15 @@ die Anthropic Messages API.
| ----------- | ----------------------------------------- | ---------------------------------- | ------------------------------------------------------------ | | ----------- | ----------------------------------------- | ---------------------------------- | ------------------------------------------------------------ |
| `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | Verhalten wie v0.1.0, unverändert. Modell-Digest-Probe via `/api/show`. | | `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | Verhalten wie v0.1.0, unverändert. Modell-Digest-Probe via `/api/show`. |
| `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI-Chat-Completions-Format — sprechen auch vLLM und die meisten selbst gehosteten Inferenz-Server. | | `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI-Chat-Completions-Format — sprechen auch vLLM und die meisten selbst gehosteten Inferenz-Server. |
| `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` ist Top-Level-Feld; `max_tokens` ist Pflicht und steht fest auf 4096. | | `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` ist Top-Level-Feld; `max_tokens` ist Pflicht; `max_tokens` setzt es, sonst 4096. |
## Ausgaben ## Ausgaben
| Name | Typ | Beschreibung | | Name | Typ | Beschreibung |
| ---------------- | ---- | --------------------------------------------------------------------- | | ---------------- | ---- | --------------------------------------------------------------------- |
| `response` | text | Die Antwort des Assistants als Plain-Text. | | `response` | text | Die Antwort des Assistants als Plain-Text. |
| `truncated` | text | `true`, wenn die Antwort an der Längengrenze endete, `false`, wenn das Modell fertig war. Leer, wenn das Backend keinen Grund nannte. Eine abgeschnittene Antwort wird trotzdem geliefert; der Ablauf entscheidet, was damit geschieht. |
| `stop_reason` | text | Der Grund des Backends: Ollama `done_reason`, OpenAI `finish_reason`, Anthropic `stop_reason`. Leer, wenn er fehlt. |
| `model_endpoint` | text | URL, gegen die die Antwort erzeugt wurde. | | `model_endpoint` | text | URL, gegen die die Antwort erzeugt wurde. |
| `model_name` | text | Modell-ID wie an die LLM-API gesendet. | | `model_name` | text | Modell-ID wie an die LLM-API gesendet. |
| `model_digest` | text | SHA-256-Digest des bedienenden Ollama-Modells. Leer bei `openai`/`anthropic` — diese APIs bieten keinen Digest; das Feld bleibt leer statt erfunden. | | `model_digest` | text | SHA-256-Digest des bedienenden Ollama-Modells. Leer bei `openai`/`anthropic` — diese APIs bieten keinen Digest; das Feld bleibt leer statt erfunden. |

View file

@ -10,7 +10,7 @@ Anthropic Messages API.
## Capability ## Capability
- `llm.chat@0.3.0` - `llm.chat@0.4.0`
## Inputs ## Inputs
@ -22,6 +22,8 @@ Anthropic Messages API.
| `api_key` | text | Optional API key. Never appears in outputs, logs, or events. | | `api_key` | text | Optional API key. Never appears in outputs, logs, or events. |
| `keep_alive` | text | Ollama only: how long the server keeps the model resident after this request (`4h`, `30m`, `-1` = indefinitely). Empty = server default, which unloads after five minutes. Refused with any other `api`. | | `keep_alive` | text | Ollama only: how long the server keeps the model resident after this request (`4h`, `30m`, `-1` = indefinitely). Empty = server default, which unloads after five minutes. Refused with any other `api`. |
| `think` | text | `true` or `false`: whether a reasoning model runs its thinking phase. Empty = server default (Ollama lets qwen3-type models think). Ollama gets `think`, vLLM `chat_template_kwargs.enable_thinking`; refused on the OpenAI wire; Anthropic takes `false` only. Measured on Ollama 0.40.9: `qwen3.8:27b` and `gemma4:26b` honour `false`; `qwen3:30b-a3b` thinks anyway and its thinking then arrives inside the reply, ending with `</think>`. | | `think` | text | `true` or `false`: whether a reasoning model runs its thinking phase. Empty = server default (Ollama lets qwen3-type models think). Ollama gets `think`, vLLM `chat_template_kwargs.enable_thinking`; refused on the OpenAI wire; Anthropic takes `false` only. Measured on Ollama 0.40.9: `qwen3.8:27b` and `gemma4:26b` honour `false`; `qwen3:30b-a3b` thinks anyway and its thinking then arrives inside the reply, ending with `</think>`. |
| `max_tokens` | text | Upper bound on the answer length in tokens (whole number, at least 1). Ollama gets `options.num_predict`, the OpenAI wire `max_tokens`, Anthropic `max_tokens` (default 4096 when empty). Empty = no limit on Ollama and the OpenAI wire. Without a limit, a model caught in a repetition loop writes until the flow's timeout; with `num_predict: 400`, `qwen3.8:27b` on Ollama 0.40.9 stopped such a loop after 39 s. |
| `repeat_penalty` | text | Penalty on repeated tokens, 1.0 = neutral. Ollama gets `options.repeat_penalty`, vLLM `repetition_penalty`; refused on the OpenAI wire and for Anthropic. |
| `system_prompt` | text | Optional system message. Empty = use the model's default. | | `system_prompt` | text | Optional system message. Empty = use the model's default. |
| `api` | text | Optional wire format: `ollama` (default), `openai`, `anthropic`. | | `api` | text | Optional wire format: `ollama` (default), `openai`, `anthropic`. |
@ -31,13 +33,15 @@ Anthropic Messages API.
| ----------- | --------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ | | ----------- | --------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ |
| `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | v0.1.0 behavior, unchanged. Model-digest probe via `/api/show`. | | `ollama` | `http://localhost:11434/api/chat` | `Authorization: Bearer` (optional) | v0.1.0 behavior, unchanged. Model-digest probe via `/api/show`. |
| `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI Chat Completions format — also spoken by vLLM and most self-hosted inference servers. | | `openai` | `https://.../v1/chat/completions` | `Authorization: Bearer` | OpenAI Chat Completions format — also spoken by vLLM and most self-hosted inference servers. |
| `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` is a top-level field; `max_tokens` is mandatory and defaults to 4096. | | `anthropic` | `https://api.anthropic.com/v1/messages` | `x-api-key` + `anthropic-version` | Messages API: `system` is a top-level field; `max_tokens` is mandatory; `max_tokens` sets it, default 4096. |
## Outputs ## Outputs
| Name | Type | Description | | Name | Type | Description |
| ---------------- | ---- | ------------------------------------------------------------------- | | ---------------- | ---- | ------------------------------------------------------------------- |
| `response` | text | The assistant's plain-text reply. | | `response` | text | The assistant's plain-text reply. |
| `truncated` | text | `true` when the answer stopped at the length limit, `false` when the model finished. Empty when the backend gave no stop reason. A truncated answer is still returned; the flow decides what to do with it. |
| `stop_reason` | text | The backend's own reason: Ollama `done_reason`, OpenAI `finish_reason`, Anthropic `stop_reason`. Empty when absent. |
| `model_endpoint` | text | The endpoint the response was generated against. | | `model_endpoint` | text | The endpoint the response was generated against. |
| `model_name` | text | The model identifier as supplied to the LLM API. | | `model_name` | text | The model identifier as supplied to the LLM API. |
| `model_digest` | text | SHA-256 digest of the served Ollama model. Empty for `openai`/`anthropic` — those APIs expose no digest; the field is left empty rather than fabricated. | | `model_digest` | text | SHA-256 digest of the served Ollama model. Empty for `openai`/`anthropic` — those APIs expose no digest; the field is left empty rather than fabricated. |

Binary file not shown.

View file

@ -1,12 +1,12 @@
schema_version: 3 schema_version: 3
provider: chain provider: chain
name: llm-chat name: llm-chat
version: 0.3.1 version: 0.4.0
# Capability provided by this module. # Capability provided by this module.
provides: provides:
- capability: llm.chat - capability: llm.chat
version: 0.3.1 version: 0.4.0
# Declared inputs in verbose v3 form. Studio renders the # Declared inputs in verbose v3 form. Studio renders the
# description in the active locale as a tooltip on each # description in the active locale as a tooltip on each
@ -108,6 +108,39 @@ inputs:
das einen Lauf wiederholbar. Nur Ollama und das das einen Lauf wiederholbar. Nur Ollama und das
OpenAI-Format; Anthropic kennt keinen Seed und lehnt den OpenAI-Format; Anthropic kennt keinen Seed und lehnt den
Eingang ab. Eingang ab.
max_tokens:
type: text
description:
en: |
Upper bound on the answer length in tokens, a whole number of
at least 1. A model caught in a repetition loop otherwise
writes until the flow's timeout. Ollama receives
options.num_predict, the OpenAI wire max_tokens, Anthropic
max_tokens (default 4096 there when empty). Empty leaves no
limit on Ollama and the OpenAI wire. Whether the answer
stopped at the limit is reported in the output truncated.
de: |
Obergrenze für die Länge der Antwort in Token, ganze Zahl ab
1. Ein Modell in einer Wiederholungsschleife schreibt sonst
bis zur Zeitgrenze des Ablaufs. Ollama bekommt
options.num_predict, das OpenAI-Format max_tokens, Anthropic
max_tokens (dort 4096, wenn leer). Leer heißt bei Ollama und
im OpenAI-Format keine Grenze. Ob die Antwort an der Grenze
endete, steht in der Ausgabe truncated.
repeat_penalty:
type: text
description:
en: |
Penalty on repeated tokens, a number; 1.0 is neutral, values
such as 1.1 make loops less likely. Ollama receives
options.repeat_penalty, vLLM repetition_penalty. The OpenAI
wire and Anthropic have no such setting and refuse the input.
de: |
Abschlag auf wiederholte Token, eine Zahl; 1.0 ist neutral,
Werte wie 1.1 machen Schleifen seltener. Ollama bekommt
options.repeat_penalty, vLLM repetition_penalty. Das
OpenAI-Format und Anthropic kennen die Einstellung nicht und
lehnen den Eingang ab.
keep_alive: keep_alive:
type: text type: text
description: description:
@ -167,6 +200,32 @@ outputs:
description: description:
en: The assistant's plain-text reply. en: The assistant's plain-text reply.
de: Die Klartext-Antwort des Modells. de: Die Klartext-Antwort des Modells.
truncated:
type: text
description:
en: |
"true" when the answer stopped at the length limit
(max_tokens) instead of where the model finished, "false"
when the model finished. Empty when the backend gave no
stop reason.
de: |
"true", wenn die Antwort an der Längengrenze (max_tokens)
endete statt dort, wo das Modell fertig war, "false", wenn
das Modell fertig war. Leer, wenn das Backend keinen Grund
für das Ende nannte.
stop_reason:
type: text
description:
en: |
Why the backend stopped writing, in its own words: Ollama
done_reason (stop, length), the OpenAI wire finish_reason
(stop, length), Anthropic stop_reason (end_turn,
max_tokens, ...). Empty when the answer did not say.
de: |
Warum das Backend aufhörte, in seinen eigenen Worten: Ollama
done_reason (stop, length), das OpenAI-Format finish_reason
(stop, length), Anthropic stop_reason (end_turn, max_tokens
u. a.). Leer, wenn die Antwort es nicht angab.
model_endpoint: model_endpoint:
type: text type: text
description: description:

View file

@ -48,6 +48,8 @@ pub fn invoke(_ctx: Context, inputs: Inputs) -> Result<Outputs, ModuleError> {
let top_p = sampling_input("top_p"); let top_p = sampling_input("top_p");
let top_k = sampling_input("top_k"); let top_k = sampling_input("top_k");
let seed = sampling_input("seed"); let seed = sampling_input("seed");
let max_tokens = sampling_input("max_tokens");
let repeat_penalty = sampling_input("repeat_penalty");
// Refused rather than ignored for any other wire format. The // Refused rather than ignored for any other wire format. The
// whole point of the input is an expectation about the server // whole point of the input is an expectation about the server
// ("the model stays loaded"); silently dropping it would leave // ("the model stays loaded"); silently dropping it would leave
@ -77,6 +79,8 @@ pub fn invoke(_ctx: Context, inputs: Inputs) -> Result<Outputs, ModuleError> {
top_p: &top_p, top_p: &top_p,
top_k: &top_k, top_k: &top_k,
seed: &seed, seed: &seed,
max_tokens: &max_tokens,
repeat_penalty: &repeat_penalty,
}, },
}; };
let result = crate::llm::chat_with_identity(&client, &params) let result = crate::llm::chat_with_identity(&client, &params)
@ -89,8 +93,13 @@ pub fn invoke(_ctx: Context, inputs: Inputs) -> Result<Outputs, ModuleError> {
Some((prompt, completion)) => (prompt.to_string(), completion.to_string()), Some((prompt, completion)) => (prompt.to_string(), completion.to_string()),
None => (String::new(), String::new()), None => (String::new(), String::new()),
}; };
// Empty where the backend gave no stop reason: "false" would
// claim the answer is complete without anyone having said so.
let truncated = result.truncated.map(|t| t.to_string()).unwrap_or_default();
Ok(Outputs::new() Ok(Outputs::new()
.with_text("response", result.response) .with_text("response", result.response)
.with_text("truncated", truncated)
.with_text("stop_reason", result.stop_reason.unwrap_or_default())
.with_text("model_endpoint", endpoint) .with_text("model_endpoint", endpoint)
.with_text("model_name", model) .with_text("model_name", model)
.with_text("model_digest", result.model_digest.unwrap_or_default()) .with_text("model_digest", result.model_digest.unwrap_or_default())

View file

@ -119,6 +119,12 @@ pub struct Sampling<'a> {
pub top_p: &'a str, pub top_p: &'a str,
pub top_k: &'a str, pub top_k: &'a str,
pub seed: &'a str, pub seed: &'a str,
/// Upper bound on the answer length in tokens. A model caught in
/// a repetition loop otherwise writes until the flow's timeout.
pub max_tokens: &'a str,
/// Penalty on repeated tokens (Ollama `repeat_penalty`, vLLM
/// `repetition_penalty`; 1.0 is neutral).
pub repeat_penalty: &'a str,
} }
/// One parsed sampling value. Numbers only; a value that is not a /// One parsed sampling value. Numbers only; a value that is not a
@ -160,6 +166,10 @@ struct OllamaOptions {
top_k: Option<SamplingValue>, top_k: Option<SamplingValue>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
seed: Option<SamplingValue>, seed: Option<SamplingValue>,
#[serde(skip_serializing_if = "Option::is_none")]
num_predict: Option<SamplingValue>,
#[serde(skip_serializing_if = "Option::is_none")]
repeat_penalty: Option<SamplingValue>,
} }
impl OllamaOptions { impl OllamaOptions {
@ -168,6 +178,24 @@ impl OllamaOptions {
&& self.top_p.is_none() && self.top_p.is_none()
&& self.top_k.is_none() && self.top_k.is_none()
&& self.seed.is_none() && self.seed.is_none()
&& self.num_predict.is_none()
&& self.repeat_penalty.is_none()
}
}
/// Parse `max_tokens`: a whole number of at least 1. Zero or a
/// negative count would mean "no answer" or, for Ollama, "no limit",
/// and neither is what a caller writing a limit asks for.
fn parse_max_tokens(raw: &str) -> Result<Option<u32>, LlmError> {
let raw = raw.trim();
if raw.is_empty() {
return Ok(None);
}
match raw.parse::<u32>() {
Ok(n) if n >= 1 => Ok(Some(n)),
_ => Err(LlmError::UnsupportedApi(format!(
"max_tokens must be a whole number of at least 1, got {raw:?}"
))),
} }
} }
@ -247,6 +275,11 @@ struct OpenAiRequest<'a> {
top_p: Option<SamplingValue>, top_p: Option<SamplingValue>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
seed: Option<SamplingValue>, seed: Option<SamplingValue>,
#[serde(skip_serializing_if = "Option::is_none")]
max_tokens: Option<u32>,
/// vLLM only: an extra sampling parameter of vLLM's server.
#[serde(skip_serializing_if = "Option::is_none")]
repetition_penalty: Option<SamplingValue>,
/// vLLM only: passed to the chat template, where qwen3-type /// vLLM only: passed to the chat template, where qwen3-type
/// templates read `enable_thinking`. /// templates read `enable_thinking`.
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
@ -310,6 +343,9 @@ pub fn build_ollama_body(p: &ChatParams) -> Result<String, LlmError> {
top_p: parse_sampling("top_p", p.sampling.top_p, false)?, top_p: parse_sampling("top_p", p.sampling.top_p, false)?,
top_k: parse_sampling("top_k", p.sampling.top_k, true)?, top_k: parse_sampling("top_k", p.sampling.top_k, true)?,
seed: parse_sampling("seed", p.sampling.seed, true)?, seed: parse_sampling("seed", p.sampling.seed, true)?,
num_predict: parse_max_tokens(p.sampling.max_tokens)?
.map(|n| SamplingValue::Int(i64::from(n))),
repeat_penalty: parse_sampling("repeat_penalty", p.sampling.repeat_penalty, false)?,
}; };
let req = OllamaRequest { let req = OllamaRequest {
model: p.model, model: p.model,
@ -351,6 +387,16 @@ pub fn build_openai_body(p: &ChatParams) -> Result<String, LlmError> {
)); ));
} }
}; };
// OpenAI's own server has no repetition penalty with this
// meaning (frequency_penalty counts differently); vLLM has one.
let repetition_penalty = parse_sampling("repeat_penalty", p.sampling.repeat_penalty, false)?;
if repetition_penalty.is_some() && p.api != Api::Vllm {
return Err(LlmError::UnsupportedApi(
"repeat_penalty is not part of the OpenAI chat wire; use \
api: ollama or vllm, or drop the input"
.to_string(),
));
}
let req = OpenAiRequest { let req = OpenAiRequest {
model: p.model, model: p.model,
messages: build_messages(p), messages: build_messages(p),
@ -358,6 +404,8 @@ pub fn build_openai_body(p: &ChatParams) -> Result<String, LlmError> {
temperature: parse_sampling("temperature", p.sampling.temperature, false)?, temperature: parse_sampling("temperature", p.sampling.temperature, false)?,
top_p: parse_sampling("top_p", p.sampling.top_p, false)?, top_p: parse_sampling("top_p", p.sampling.top_p, false)?,
seed: parse_sampling("seed", p.sampling.seed, true)?, seed: parse_sampling("seed", p.sampling.seed, true)?,
max_tokens: parse_max_tokens(p.sampling.max_tokens)?,
repetition_penalty,
chat_template_kwargs, chat_template_kwargs,
}; };
Ok(serde_json::to_string(&req).unwrap_or_else(|_| String::from("{}"))) Ok(serde_json::to_string(&req).unwrap_or_else(|_| String::from("{}")))
@ -388,9 +436,17 @@ pub fn build_anthropic_body(p: &ChatParams) -> Result<String, LlmError> {
.to_string(), .to_string(),
)); ));
} }
if !p.sampling.repeat_penalty.trim().is_empty() {
return Err(LlmError::UnsupportedApi(
"repeat_penalty is not part of the Anthropic Messages API; use \
api: ollama or vllm, or drop the input"
.to_string(),
));
}
let req = AnthropicRequest { let req = AnthropicRequest {
model: p.model, model: p.model,
max_tokens: ANTHROPIC_DEFAULT_MAX_TOKENS, max_tokens: parse_max_tokens(p.sampling.max_tokens)?
.unwrap_or(ANTHROPIC_DEFAULT_MAX_TOKENS),
system: if p.system_prompt.is_empty() { system: if p.system_prompt.is_empty() {
None None
} else { } else {
@ -454,6 +510,33 @@ pub struct ChatWithIdentity {
/// (prompt, completion). `None` where the answer carries no /// (prompt, completion). `None` where the answer carries no
/// count, which is honest about a number nobody measured. /// count, which is honest about a number nobody measured.
pub tokens: Option<(u64, u64)>, pub tokens: Option<(u64, u64)>,
/// Why the backend stopped writing, as the backend said it
/// (`stop`, `length`, `end_turn`, `max_tokens`, ...). `None`
/// where the answer did not say.
pub stop_reason: Option<String>,
/// Whether the answer ended at the length limit rather than
/// where the model finished. `None` where the backend gave no
/// reason, because "not cut off" would be a guess.
pub truncated: Option<bool>,
}
/// The stop reason of one answer and whether it means "cut off at
/// the length limit". Ollama writes `done_reason` (`stop`, `length`),
/// the OpenAI shape `choices[0].finish_reason` (`stop`, `length`),
/// Anthropic `stop_reason` (`end_turn`, `max_tokens`, ...).
pub fn extract_stop(api: Api, body: &str) -> (Option<String>, Option<bool>) {
let Ok(v) = serde_json::from_str::<serde_json::Value>(body) else {
return (None, None);
};
let (reason, length_word) = match api {
Api::Ollama => (v.get("done_reason"), "length"),
Api::Openai | Api::Vllm => (v.pointer("/choices/0/finish_reason"), "length"),
Api::Anthropic => (v.get("stop_reason"), "max_tokens"),
};
match reason.and_then(|r| r.as_str()).filter(|r| !r.is_empty()) {
Some(r) => (Some(r.to_string()), Some(r == length_word)),
None => (None, None),
}
} }
/// The token counts of one answer, per wire format. /// The token counts of one answer, per wire format.
@ -522,10 +605,13 @@ pub fn chat_with_identity<C: LlmClient>(
Api::Openai | Api::Vllm | Api::Anthropic => None, Api::Openai | Api::Vllm | Api::Anthropic => None,
}; };
let tokens = extract_tokens(p.api, &response_body); let tokens = extract_tokens(p.api, &response_body);
let (stop_reason, truncated) = extract_stop(p.api, &response_body);
Ok(ChatWithIdentity { Ok(ChatWithIdentity {
response, response,
model_digest, model_digest,
tokens, tokens,
stop_reason,
truncated,
}) })
} }
@ -1350,3 +1436,193 @@ mod think_tests {
assert!(!body.contains("think"), "{body}"); assert!(!body.contains("think"), "{body}");
} }
} }
#[cfg(test)]
mod length_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
fn p(api: Api, sampling: Sampling<'_>) -> ChatParams<'_> {
ChatParams {
api,
endpoint: "http://x/api/chat",
model: "qwen3.8:27b",
api_key: "",
system_prompt: "",
prompt: "hello",
keep_alive: "",
think: "",
sampling,
}
}
fn body(api: Api, sampling: Sampling<'_>) -> Result<serde_json::Value, LlmError> {
let params = p(api, sampling);
let raw = match api {
Api::Ollama => build_ollama_body(&params),
Api::Openai | Api::Vllm => build_openai_body(&params),
Api::Anthropic => build_anthropic_body(&params),
}?;
Ok(serde_json::from_str(&raw).unwrap())
}
fn limited(max_tokens: &str) -> Sampling<'_> {
Sampling {
max_tokens,
..Sampling::default()
}
}
/// Each wire spells the limit differently; every one must carry it.
#[test]
fn every_wire_carries_the_length_limit() {
assert_eq!(
body(Api::Ollama, limited("400")).unwrap()["options"]["num_predict"],
400
);
assert_eq!(
body(Api::Openai, limited("400")).unwrap()["max_tokens"],
400
);
assert_eq!(body(Api::Vllm, limited("400")).unwrap()["max_tokens"], 400);
assert_eq!(
body(Api::Anthropic, limited("400")).unwrap()["max_tokens"],
400
);
}
/// Unset leaves the bodies as before: no limit on Ollama and the
/// OpenAI wire, the mandatory default on Anthropic.
#[test]
fn unset_keeps_the_previous_bodies() {
let v = body(Api::Ollama, Sampling::default()).unwrap();
assert!(v.get("options").is_none(), "{v}");
let v = body(Api::Vllm, Sampling::default()).unwrap();
assert!(
v.get("max_tokens").is_none() && v.get("repetition_penalty").is_none(),
"{v}"
);
assert_eq!(
body(Api::Anthropic, Sampling::default()).unwrap()["max_tokens"],
4096
);
}
/// Zero would be "no answer" and, for Ollama's -1, "no limit";
/// a caller writing a limit means neither.
#[test]
fn a_limit_below_one_or_not_a_number_is_refused() {
for raw in ["0", "-1", "12.5", "viel"] {
for api in [Api::Ollama, Api::Openai, Api::Anthropic] {
let err = body(api, limited(raw)).unwrap_err();
assert!(
err.to_string().contains("max_tokens"),
"{raw} {api:?}: {err}"
);
}
}
}
#[test]
fn repeat_penalty_goes_where_the_backend_reads_it() {
let s = Sampling {
repeat_penalty: "1.15",
..Sampling::default()
};
assert_eq!(
body(Api::Ollama, s).unwrap()["options"]["repeat_penalty"],
1.15
);
assert_eq!(body(Api::Vllm, s).unwrap()["repetition_penalty"], 1.15);
// No equivalent: refused rather than dropped.
for api in [Api::Openai, Api::Anthropic] {
let err = body(api, s).unwrap_err();
assert!(err.to_string().contains("repeat_penalty"), "{api:?}: {err}");
}
}
/// The case baigle hit: the answer stopped at the limit. The
/// caller must be able to tell that from a finished answer.
#[test]
fn a_cut_off_answer_is_reported_per_wire() {
let cases = [
(
Api::Ollama,
r#"{"message":{"content":"§ 566"},"done_reason":"length"}"#,
"length",
true,
),
(
Api::Ollama,
r#"{"message":{"content":"ok"},"done_reason":"stop"}"#,
"stop",
false,
),
(
Api::Vllm,
r#"{"choices":[{"message":{"content":"x"},"finish_reason":"length"}]}"#,
"length",
true,
),
(
Api::Openai,
r#"{"choices":[{"message":{"content":"x"},"finish_reason":"stop"}]}"#,
"stop",
false,
),
(
Api::Anthropic,
r#"{"content":[{"text":"x"}],"stop_reason":"max_tokens"}"#,
"max_tokens",
true,
),
(
Api::Anthropic,
r#"{"content":[{"text":"x"}],"stop_reason":"end_turn"}"#,
"end_turn",
false,
),
];
for (api, raw, reason, cut) in cases {
let (r, t) = extract_stop(api, raw);
assert_eq!(r.as_deref(), Some(reason), "{api:?}");
assert_eq!(t, Some(cut), "{api:?}");
}
}
/// No reason in the answer is no claim either way.
#[test]
fn no_stop_reason_claims_nothing() {
assert_eq!(
extract_stop(Api::Ollama, r#"{"message":{"content":"x"}}"#),
(None, None)
);
assert_eq!(extract_stop(Api::Openai, "not json"), (None, None));
}
#[test]
fn the_whole_call_reports_the_cut() {
struct One(&'static str);
impl LlmClient for One {
fn post_json(
&self,
url: &str,
_: &str,
_: &[(&'static str, String)],
) -> Result<String, LlmError> {
if url.ends_with("/api/show") {
Err(LlmError::Status(404))
} else {
Ok(self.0.to_string())
}
}
}
let client = One(
r#"{"message":{"content":"§ 556 BGB § 566 BGB"},"done_reason":"length","eval_count":400}"#,
);
let out = chat_with_identity(&client, &p(Api::Ollama, limited("400"))).unwrap();
assert_eq!(out.truncated, Some(true));
assert_eq!(out.stop_reason.as_deref(), Some("length"));
assert_eq!(out.response, "§ 556 BGB § 566 BGB");
}
}