text.anonymize.legal-de 0.1.0: WASM bridge to the judge-ner host service
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1m20s

NER anonymization of German legal texts (plain text + DOCX) via the
JuraNER model by Harshil Darji (MIT). The transformer model runs in
the judge-ner host service; this module forwards documents over
loopback HTTP and maps responses onto the capability contract.
This commit is contained in:
flemming-it 2026-07-10 12:44:32 +02:00
commit 3e018f528a
15 changed files with 1781 additions and 0 deletions

102
MODULE.md Normal file
View file

@ -0,0 +1,102 @@
# text.anonymize.legal-de
NER-based anonymization of German legal texts (JuraNER). Bridge
module: the transformers model runs in the **judge-ner host service**
(`fai_judge/host_services/judge-ner/`); this WASM module forwards the
document over loopback HTTP and maps the response onto the capability
contract.
## Capability
- `text.anonymize.legal-de@0.1.0`
## Inputs
| Name | Type | Description |
| ---------- | ----- | ----------- |
| `request` | json | `{"mode":"text"\|"docx","options":{...}}` — see below. |
| `document` | bytes | UTF-8 plain text (mode=text) or a DOCX file (mode=docx). |
| `endpoint` | text | Optional judge-ner base URL. Default `http://127.0.0.1:8756`. |
### `request.options` (all optional)
| Key | Default | Meaning |
| --- | ------- | ------- |
| `importance_levels` | `["High"]` | Entity classes to anonymize (`High`/`Mid`/`Low`). |
| `confidence_threshold` | `0.8` | Minimum NER confidence (0.01.0). |
| `manual_phrases` | `[]` | Additional phrases to redact (label `RED`). |
| `maintain_consistency` | `true` | Same surface form gets the same ⟦TYPE#⟧ token everywhere. |
| `remove_rubrum` | `false` | Detect the rubrum block and replace it with a placeholder. |
## Outputs
| Name | Type | Description |
| ---------- | ----- | ----------- |
| `result` | json | `{"replacements": [...], "text": "..."}``text` (the anonymized text) only for mode=text. `statistics` and `rubrum` are passed through from the service when present. |
| `document` | bytes | The anonymized document: UTF-8 text (`text/plain`) for mode=text, the rewritten DOCX for mode=docx. |
Each `replacements` entry:
`{text, label, label_description, start, end, score, line_number,
anonymized}`; DOCX mode additionally carries `run_id_start`,
`char_in_start`, `run_id_end`, `char_in_end` addressing the exact
`w:t` nodes of the original document. Entries with
`"anonymized": null` were detected but not replaced (importance level
not selected).
## Permissions & required services
```yaml
permissions:
- "net: localhost"
- "net: 127.0.0.1"
requires_services:
- judge-ner
```
The hub refuses to dispatch a step to this module unless the operator
declared a `judge-ner` service endpoint in `~/.chain/config.yaml`
(`services:`). Note the first judge-ner start downloads the JuraNER
model from HuggingFace — until then the service answers 503 and this
module fails with a clear "NER model not available" error.
## Setting up the judge-ner host service
The NER model (JuraNER by Harshil Darji, MIT — transformers + torch)
cannot run inside the WASM sandbox; it runs as a host service.
```bash
# 1. Start the service (Docker; ~1 GB model download on first start,
# persisted in the named volume — later starts are offline-capable)
docker run -d --name judge-ner \
-p 8756:8756 \
-v judge-ner-models:/models \
judge-ner
# 2. Register it with the hub (~/.chain/config.yaml)
# services:
# judge-ner:
# endpoint: http://127.0.0.1:8756
# health_check:
# url: http://127.0.0.1:8756/health
# 3. Verify
curl -s localhost:8756/health # {"status":"ok","model_loaded":true}
chain service status judge-ner
```
Source and image build: `fai_judge/host_services/judge-ner/`
(`docker build -t judge-ner .`).
## Errors
- Malformed `request`, non-UTF-8 text document, or a 4xx from the
service → `invalid input: ...`
- Service unreachable, 5xx, malformed service response →
`internal error: judge-ner at <url> ...`
## Build
```bash
cargo test
cargo build --release --target wasm32-wasip2
```