feat: keep_alive input so a local model stays resident
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s

Ollama unloads a model five minutes after its last request, and the
next request then pays a full reload: 30 to 40 seconds for a large
local model, long enough that an operator assumes something is broken.
The stromnetz demonstrator runs on a notebook from around 12.10. and
hits exactly that after any idle period; it works around it today by
setting OLLAMA_KEEP_ALIVE on the server, which is not expressible in a
flow.

Measured against a live Ollama: without the input the model expired
5:02 after the request, with `keep_alive: "90s"` it expired after 92
seconds. Worth knowing for callers: Ollama keeps the value for the
loaded instance, so a later request that omits keep_alive does not
fall back to five minutes until the model has actually been unloaded.

`-1` is sent as a number rather than a string, because Ollama reads
only the numeric form as "never unload"; quoted it is parsed as a
duration and the setting silently does nothing.

With any other wire format the input is REFUSED, not ignored. It
expresses an expectation about the server, and dropping it quietly
would leave the caller paying the reload while believing it was
handled. The message says what to do instead.

Capability goes 0.2.0 to 0.2.1: the addition is optional and additive,
so `@^0.2` and `@^0` both keep resolving, and an unset value leaves
the request byte-identical to the previous version.

Five guards, including that unset omits the field entirely and that
the other wire formats never carry it. Mutation-checked three ways.

Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
flemming-it 2026-10-07 05:49:16 +02:00
parent dd7b419d37
commit 9483f196a1
8 changed files with 230 additions and 4 deletions

View file

@ -1,12 +1,12 @@
schema_version: 3
provider: chain
name: llm-chat
version: 0.2.0
version: 0.2.1
# Capability provided by this module.
provides:
- capability: llm.chat
version: 0.2.0
version: 0.2.1
# Declared inputs in verbose v3 form. Studio renders the
# description in the active locale as a tooltip on each
@ -67,6 +67,27 @@ inputs:
"Authorization: Bearer", bei anthropic als "x-api-key"
gesendet. Leer = kein Auth-Header. Erscheint nie in
Ausgaben, Logs oder Audit-Events.
keep_alive:
type: text
description:
en: |
Ollama only: how long the server keeps the model resident
after this request, e.g. "4h", "30m", or "-1" for
indefinitely. Empty leaves the server default, which unloads
after five minutes, so the next first request pays a full
model reload (30 to 40 seconds for a large local model).
Setting this with any other api is refused rather than
ignored, because the input expresses an expectation the
other backends cannot meet.
de: |
Nur für Ollama: wie lange der Server das Modell nach dieser
Anfrage geladen lässt, z. B. "4h", "30m" oder "-1" für
unbegrenzt. Leer bedeutet Server-Voreinstellung, die nach
fünf Minuten entlädt; die nächste erste Anfrage zahlt dann
den vollen Modell-Neustart (30 bis 40 Sekunden bei einem
großen lokalen Modell). Mit einer anderen api wird der Wert
abgelehnt statt ignoriert, weil er eine Erwartung ausdrückt,
die die anderen Backends nicht erfüllen können.
system_prompt:
type: text
description: