Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s
Ollama unloads a model five minutes after its last request, and the next request then pays a full reload: 30 to 40 seconds for a large local model, long enough that an operator assumes something is broken. The stromnetz demonstrator runs on a notebook from around 12.10. and hits exactly that after any idle period; it works around it today by setting OLLAMA_KEEP_ALIVE on the server, which is not expressible in a flow. Measured against a live Ollama: without the input the model expired 5:02 after the request, with `keep_alive: "90s"` it expired after 92 seconds. Worth knowing for callers: Ollama keeps the value for the loaded instance, so a later request that omits keep_alive does not fall back to five minutes until the model has actually been unloaded. `-1` is sent as a number rather than a string, because Ollama reads only the numeric form as "never unload"; quoted it is parsed as a duration and the setting silently does nothing. With any other wire format the input is REFUSED, not ignored. It expresses an expectation about the server, and dropping it quietly would leave the caller paying the reload while believing it was handled. The message says what to do instead. Capability goes 0.2.0 to 0.2.1: the addition is optional and additive, so `@^0.2` and `@^0` both keep resolving, and an unset value leaves the request byte-identical to the previous version. Five guards, including that unset omits the field entirely and that the other wire formats never carry it. Mutation-checked three ways. Signed-off-by: flemming-it <sf@flemming.it>
145 lines
5.5 KiB
YAML
145 lines
5.5 KiB
YAML
schema_version: 3
|
|
provider: chain
|
|
name: llm-chat
|
|
version: 0.2.1
|
|
|
|
# Capability provided by this module.
|
|
provides:
|
|
- capability: llm.chat
|
|
version: 0.2.1
|
|
|
|
# Declared inputs in verbose v3 form. Studio renders the
|
|
# description in the active locale as a tooltip on each
|
|
# port; the editor falls back to the English description
|
|
# when no peer is configured.
|
|
inputs:
|
|
prompt:
|
|
type: text
|
|
description:
|
|
en: The user-facing prompt text the model should respond to.
|
|
de: Der vom Modell zu beantwortende Prompt-Text.
|
|
endpoint:
|
|
type: text
|
|
description:
|
|
en: |
|
|
Chat endpoint URL matching the selected api:
|
|
ollama "http://localhost:11434/api/chat",
|
|
openai "https://.../v1/chat/completions" (vLLM etc.),
|
|
anthropic "https://api.anthropic.com/v1/messages".
|
|
de: |
|
|
Chat-Endpunkt-URL passend zur gewählten api:
|
|
ollama "http://localhost:11434/api/chat",
|
|
openai "https://.../v1/chat/completions" (vLLM u. a.),
|
|
anthropic "https://api.anthropic.com/v1/messages".
|
|
api:
|
|
type: text
|
|
description:
|
|
en: |
|
|
Optional wire format: "ollama" (default), "vllm"
|
|
(self-hosted vLLM), "openai" (OpenAI or other
|
|
OpenAI-compatible servers such as LM Studio), or
|
|
"anthropic" (Messages API). vllm and openai speak the
|
|
same wire; the separate value exists so you can write
|
|
what you mean and get vLLM-specific hints. Empty = ollama.
|
|
de: |
|
|
Optionales Wire-Format: "ollama" (Default), "vllm"
|
|
(selbst gehostetes vLLM), "openai" (OpenAI oder andere
|
|
OpenAI-kompatible Server wie LM Studio) oder "anthropic"
|
|
(Messages API). vllm und openai sprechen dieselbe Wire;
|
|
der eigene Wert existiert, damit Du schreibst, was Du
|
|
meinst, und vLLM-spezifische Hinweise bekommst.
|
|
Leer = ollama.
|
|
model:
|
|
type: text
|
|
description:
|
|
en: Model identifier as registered with the endpoint, e.g. "qwen2.5:14b" or "llama3.1:8b".
|
|
de: Modell-Identifier wie beim Endpunkt registriert, z. B. "qwen2.5:14b" oder "llama3.1:8b".
|
|
api_key:
|
|
type: text
|
|
description:
|
|
en: |
|
|
Optional API key. Sent as "Authorization: Bearer" for
|
|
ollama/openai and as "x-api-key" for anthropic. Empty
|
|
string means no auth header is sent. Never appears in
|
|
outputs, logs, or audit events.
|
|
de: |
|
|
Optionaler API-Key. Bei ollama/openai als
|
|
"Authorization: Bearer", bei anthropic als "x-api-key"
|
|
gesendet. Leer = kein Auth-Header. Erscheint nie in
|
|
Ausgaben, Logs oder Audit-Events.
|
|
keep_alive:
|
|
type: text
|
|
description:
|
|
en: |
|
|
Ollama only: how long the server keeps the model resident
|
|
after this request, e.g. "4h", "30m", or "-1" for
|
|
indefinitely. Empty leaves the server default, which unloads
|
|
after five minutes, so the next first request pays a full
|
|
model reload (30 to 40 seconds for a large local model).
|
|
Setting this with any other api is refused rather than
|
|
ignored, because the input expresses an expectation the
|
|
other backends cannot meet.
|
|
de: |
|
|
Nur für Ollama: wie lange der Server das Modell nach dieser
|
|
Anfrage geladen lässt, z. B. "4h", "30m" oder "-1" für
|
|
unbegrenzt. Leer bedeutet Server-Voreinstellung, die nach
|
|
fünf Minuten entlädt; die nächste erste Anfrage zahlt dann
|
|
den vollen Modell-Neustart (30 bis 40 Sekunden bei einem
|
|
großen lokalen Modell). Mit einer anderen api wird der Wert
|
|
abgelehnt statt ignoriert, weil er eine Erwartung ausdrückt,
|
|
die die anderen Backends nicht erfüllen können.
|
|
system_prompt:
|
|
type: text
|
|
description:
|
|
en: |
|
|
Optional system prompt. Empty string omits the system
|
|
message and lets the model use its built-in default.
|
|
de: |
|
|
Optionaler System-Prompt. Leer = das Modell verwendet
|
|
seinen eingebauten Default.
|
|
|
|
# Outputs produced.
|
|
outputs:
|
|
response:
|
|
type: text
|
|
description:
|
|
en: The assistant's plain-text reply.
|
|
de: Die Klartext-Antwort des Modells.
|
|
model_endpoint:
|
|
type: text
|
|
description:
|
|
en: The endpoint the response was generated against. Used for audit correlation.
|
|
de: Der Endpunkt, gegen den die Antwort erzeugt wurde — für Audit-Korrelation.
|
|
model_name:
|
|
type: text
|
|
description:
|
|
en: The model identifier as supplied to the LLM API.
|
|
de: Der an die LLM-API übergebene Modell-Identifier.
|
|
model_digest:
|
|
type: text
|
|
description:
|
|
en: |
|
|
SHA-256 digest of the served Ollama model, when reachable.
|
|
Empty for non-Ollama endpoints (OpenAI / Anthropic do not
|
|
expose a digest API). Together with model_endpoint and
|
|
model_name this answers the audit question "which exact
|
|
model produced this response?"
|
|
de: |
|
|
SHA-256-Digest des bedienten Ollama-Modells, wenn erreichbar.
|
|
Leer für Nicht-Ollama-Endpunkte (OpenAI / Anthropic stellen
|
|
keine Digest-API bereit). Zusammen mit model_endpoint und
|
|
model_name beantwortet das die Audit-Frage "welches genaue
|
|
Modell hat diese Antwort erzeugt?"
|
|
|
|
# Permissions required.
|
|
#
|
|
# The module makes outbound HTTP to the configured endpoint.
|
|
# Default declarations cover loopback (Ollama) and the common
|
|
# cloud providers. Operators with different endpoints fork
|
|
# module.yaml in Phase 0.5; operator-config-driven permission
|
|
# overrides arrive in Phase 1+.
|
|
permissions:
|
|
- "net: localhost"
|
|
- "net: 127.0.0.1"
|
|
- "net: api.openai.com"
|
|
- "net: api.anthropic.com"
|