flemming-it
3e018f528a
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1m20s
NER anonymization of German legal texts (plain text + DOCX) via the JuraNER model by Harshil Darji (MIT). The transformer model runs in the judge-ner host service; this module forwards documents over loopback HTTP and maps responses onto the capability contract.
102 lines
3.5 KiB
Markdown
102 lines
3.5 KiB
Markdown
# text.anonymize.legal-de
|
||
|
||
NER-based anonymization of German legal texts (JuraNER). Bridge
|
||
module: the transformers model runs in the **judge-ner host service**
|
||
(`fai_judge/host_services/judge-ner/`); this WASM module forwards the
|
||
document over loopback HTTP and maps the response onto the capability
|
||
contract.
|
||
|
||
## Capability
|
||
|
||
- `text.anonymize.legal-de@0.1.0`
|
||
|
||
## Inputs
|
||
|
||
| Name | Type | Description |
|
||
| ---------- | ----- | ----------- |
|
||
| `request` | json | `{"mode":"text"\|"docx","options":{...}}` — see below. |
|
||
| `document` | bytes | UTF-8 plain text (mode=text) or a DOCX file (mode=docx). |
|
||
| `endpoint` | text | Optional judge-ner base URL. Default `http://127.0.0.1:8756`. |
|
||
|
||
### `request.options` (all optional)
|
||
|
||
| Key | Default | Meaning |
|
||
| --- | ------- | ------- |
|
||
| `importance_levels` | `["High"]` | Entity classes to anonymize (`High`/`Mid`/`Low`). |
|
||
| `confidence_threshold` | `0.8` | Minimum NER confidence (0.0–1.0). |
|
||
| `manual_phrases` | `[]` | Additional phrases to redact (label `RED`). |
|
||
| `maintain_consistency` | `true` | Same surface form gets the same ⟦TYPE#⟧ token everywhere. |
|
||
| `remove_rubrum` | `false` | Detect the rubrum block and replace it with a placeholder. |
|
||
|
||
## Outputs
|
||
|
||
| Name | Type | Description |
|
||
| ---------- | ----- | ----------- |
|
||
| `result` | json | `{"replacements": [...], "text": "..."}` — `text` (the anonymized text) only for mode=text. `statistics` and `rubrum` are passed through from the service when present. |
|
||
| `document` | bytes | The anonymized document: UTF-8 text (`text/plain`) for mode=text, the rewritten DOCX for mode=docx. |
|
||
|
||
Each `replacements` entry:
|
||
`{text, label, label_description, start, end, score, line_number,
|
||
anonymized}`; DOCX mode additionally carries `run_id_start`,
|
||
`char_in_start`, `run_id_end`, `char_in_end` addressing the exact
|
||
`w:t` nodes of the original document. Entries with
|
||
`"anonymized": null` were detected but not replaced (importance level
|
||
not selected).
|
||
|
||
## Permissions & required services
|
||
|
||
```yaml
|
||
permissions:
|
||
- "net: localhost"
|
||
- "net: 127.0.0.1"
|
||
requires_services:
|
||
- judge-ner
|
||
```
|
||
|
||
The hub refuses to dispatch a step to this module unless the operator
|
||
declared a `judge-ner` service endpoint in `~/.chain/config.yaml`
|
||
(`services:`). Note the first judge-ner start downloads the JuraNER
|
||
model from HuggingFace — until then the service answers 503 and this
|
||
module fails with a clear "NER model not available" error.
|
||
|
||
## Setting up the judge-ner host service
|
||
|
||
The NER model (JuraNER by Harshil Darji, MIT — transformers + torch)
|
||
cannot run inside the WASM sandbox; it runs as a host service.
|
||
|
||
```bash
|
||
# 1. Start the service (Docker; ~1 GB model download on first start,
|
||
# persisted in the named volume — later starts are offline-capable)
|
||
docker run -d --name judge-ner \
|
||
-p 8756:8756 \
|
||
-v judge-ner-models:/models \
|
||
judge-ner
|
||
|
||
# 2. Register it with the hub (~/.chain/config.yaml)
|
||
# services:
|
||
# judge-ner:
|
||
# endpoint: http://127.0.0.1:8756
|
||
# health_check:
|
||
# url: http://127.0.0.1:8756/health
|
||
|
||
# 3. Verify
|
||
curl -s localhost:8756/health # {"status":"ok","model_loaded":true}
|
||
chain service status judge-ner
|
||
```
|
||
|
||
Source and image build: `fai_judge/host_services/judge-ner/`
|
||
(`docker build -t judge-ner .`).
|
||
|
||
## Errors
|
||
|
||
- Malformed `request`, non-UTF-8 text document, or a 4xx from the
|
||
service → `invalid input: ...`
|
||
- Service unreachable, 5xx, malformed service response →
|
||
`internal error: judge-ner at <url> ...`
|
||
|
||
## Build
|
||
|
||
```bash
|
||
cargo test
|
||
cargo build --release --target wasm32-wasip2
|
||
```
|