text.anonymize.legal-de 0.1.0: WASM bridge to the judge-ner host service
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1m20s
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 1m20s
NER anonymization of German legal texts (plain text + DOCX) via the JuraNER model by Harshil Darji (MIT). The transformer model runs in the judge-ner host service; this module forwards documents over loopback HTTP and maps responses onto the capability contract.
This commit is contained in:
commit
3e018f528a
15 changed files with 1781 additions and 0 deletions
102
MODULE.md
Normal file
102
MODULE.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
# text.anonymize.legal-de
|
||||
|
||||
NER-based anonymization of German legal texts (JuraNER). Bridge
|
||||
module: the transformers model runs in the **judge-ner host service**
|
||||
(`fai_judge/host_services/judge-ner/`); this WASM module forwards the
|
||||
document over loopback HTTP and maps the response onto the capability
|
||||
contract.
|
||||
|
||||
## Capability
|
||||
|
||||
- `text.anonymize.legal-de@0.1.0`
|
||||
|
||||
## Inputs
|
||||
|
||||
| Name | Type | Description |
|
||||
| ---------- | ----- | ----------- |
|
||||
| `request` | json | `{"mode":"text"\|"docx","options":{...}}` — see below. |
|
||||
| `document` | bytes | UTF-8 plain text (mode=text) or a DOCX file (mode=docx). |
|
||||
| `endpoint` | text | Optional judge-ner base URL. Default `http://127.0.0.1:8756`. |
|
||||
|
||||
### `request.options` (all optional)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
| --- | ------- | ------- |
|
||||
| `importance_levels` | `["High"]` | Entity classes to anonymize (`High`/`Mid`/`Low`). |
|
||||
| `confidence_threshold` | `0.8` | Minimum NER confidence (0.0–1.0). |
|
||||
| `manual_phrases` | `[]` | Additional phrases to redact (label `RED`). |
|
||||
| `maintain_consistency` | `true` | Same surface form gets the same ⟦TYPE#⟧ token everywhere. |
|
||||
| `remove_rubrum` | `false` | Detect the rubrum block and replace it with a placeholder. |
|
||||
|
||||
## Outputs
|
||||
|
||||
| Name | Type | Description |
|
||||
| ---------- | ----- | ----------- |
|
||||
| `result` | json | `{"replacements": [...], "text": "..."}` — `text` (the anonymized text) only for mode=text. `statistics` and `rubrum` are passed through from the service when present. |
|
||||
| `document` | bytes | The anonymized document: UTF-8 text (`text/plain`) for mode=text, the rewritten DOCX for mode=docx. |
|
||||
|
||||
Each `replacements` entry:
|
||||
`{text, label, label_description, start, end, score, line_number,
|
||||
anonymized}`; DOCX mode additionally carries `run_id_start`,
|
||||
`char_in_start`, `run_id_end`, `char_in_end` addressing the exact
|
||||
`w:t` nodes of the original document. Entries with
|
||||
`"anonymized": null` were detected but not replaced (importance level
|
||||
not selected).
|
||||
|
||||
## Permissions & required services
|
||||
|
||||
```yaml
|
||||
permissions:
|
||||
- "net: localhost"
|
||||
- "net: 127.0.0.1"
|
||||
requires_services:
|
||||
- judge-ner
|
||||
```
|
||||
|
||||
The hub refuses to dispatch a step to this module unless the operator
|
||||
declared a `judge-ner` service endpoint in `~/.chain/config.yaml`
|
||||
(`services:`). Note the first judge-ner start downloads the JuraNER
|
||||
model from HuggingFace — until then the service answers 503 and this
|
||||
module fails with a clear "NER model not available" error.
|
||||
|
||||
## Setting up the judge-ner host service
|
||||
|
||||
The NER model (JuraNER by Harshil Darji, MIT — transformers + torch)
|
||||
cannot run inside the WASM sandbox; it runs as a host service.
|
||||
|
||||
```bash
|
||||
# 1. Start the service (Docker; ~1 GB model download on first start,
|
||||
# persisted in the named volume — later starts are offline-capable)
|
||||
docker run -d --name judge-ner \
|
||||
-p 8756:8756 \
|
||||
-v judge-ner-models:/models \
|
||||
judge-ner
|
||||
|
||||
# 2. Register it with the hub (~/.chain/config.yaml)
|
||||
# services:
|
||||
# judge-ner:
|
||||
# endpoint: http://127.0.0.1:8756
|
||||
# health_check:
|
||||
# url: http://127.0.0.1:8756/health
|
||||
|
||||
# 3. Verify
|
||||
curl -s localhost:8756/health # {"status":"ok","model_loaded":true}
|
||||
chain service status judge-ner
|
||||
```
|
||||
|
||||
Source and image build: `fai_judge/host_services/judge-ner/`
|
||||
(`docker build -t judge-ner .`).
|
||||
|
||||
## Errors
|
||||
|
||||
- Malformed `request`, non-UTF-8 text document, or a 4xx from the
|
||||
service → `invalid input: ...`
|
||||
- Service unreachable, 5xx, malformed service response →
|
||||
`internal error: judge-ner at <url> ...`
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
cargo test
|
||||
cargo build --release --target wasm32-wasip2
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue