A model caught in a repetition loop wrote until the flow's 300 s
timeout. max_tokens bounds the answer on every wire (Ollama
num_predict, OpenAI max_tokens, Anthropic max_tokens), and
truncated tells the caller whether the answer stopped at that
limit. repeat_penalty goes to Ollama and vLLM and is refused where
the backend has no such setting.
Signed-off-by: flemming-it <sf@flemming.it>
Three dialects spell the same two numbers differently: Ollama counts
in `prompt_eval_count` and `eval_count`, the OpenAI shape in
`usage.prompt_tokens` and `usage.completion_tokens`, Anthropic in
`usage.input_tokens` and `usage.output_tokens`. All three now reach
the caller as `tokens_prompt` and `tokens_completion`, which the hub
lifts into the audit log by itself.
An answer that carries no count leaves both fields empty rather than
reporting a zero: a zero is a measurement, and this is the absence of
one. Half a count is reported as far as it goes.
The cost of a model call was the one part of the audit question
"which model produced this, and what did it cost" that nothing
answered (model pool M3).
Signed-off-by: flemming-it <sf@flemming.it>
Ollama unloads a model five minutes after its last request, and the
next request then pays a full reload: 30 to 40 seconds for a large
local model, long enough that an operator assumes something is broken.
The stromnetz demonstrator runs on a notebook from around 12.10. and
hits exactly that after any idle period; it works around it today by
setting OLLAMA_KEEP_ALIVE on the server, which is not expressible in a
flow.
Measured against a live Ollama: without the input the model expired
5:02 after the request, with `keep_alive: "90s"` it expired after 92
seconds. Worth knowing for callers: Ollama keeps the value for the
loaded instance, so a later request that omits keep_alive does not
fall back to five minutes until the model has actually been unloaded.
`-1` is sent as a number rather than a string, because Ollama reads
only the numeric form as "never unload"; quoted it is parsed as a
duration and the setting silently does nothing.
With any other wire format the input is REFUSED, not ignored. It
expresses an expectation about the server, and dropping it quietly
would leave the caller paying the reload while believing it was
handled. The message says what to do instead.
Capability goes 0.2.0 to 0.2.1: the addition is optional and additive,
so `@^0.2` and `@^0` both keep resolving, and an unset value leaves
the request byte-identical to the previous version.
Five guards, including that unset omits the field entirely and that
the other wire formats never carry it. Mutation-checked three ways.
Signed-off-by: flemming-it <sf@flemming.it>
New optional input api: ollama|openai|anthropic (default ollama —
existing flows unchanged).
- openai: OpenAI Chat Completions format (/v1/chat/completions),
Bearer auth — targets vLLM and compatible self-hosted servers.
- anthropic: Messages API (/v1/messages), x-api-key +
anthropic-version headers, top-level system field, mandatory
max_tokens (fixed 4096).
- Audit outputs unchanged: model_endpoint/model_name always set;
model_digest stays Ollama-only (no digest API on openai/anthropic,
field is left empty rather than fabricated).
- api_key is only ever placed in auth headers; never in outputs,
errors, or audit events (verified against the event log).
- No streaming, no tool calls.
Bump module + capability version to 0.2.0. Wire-format unit tests
for request serialization and response parsing against fixed JSON
fixtures; live smoke green on the ollama path and on the openai
path against an OpenAI-compatible local endpoint.
Signed-off-by: flemming-it <sf@flemming.it>
Explains the 'why this module instead of inline HTTP' answer
operators ask when they see llm.chat for the first time:
audit, permission posture, endpoint portability.
Signed-off-by: flemming-it <sf@flemming.it>