feat: text.anonymize v0.1.0 — regex-based PII redaction
First release. Pure-Rust, in-WASM, regex-only, declares no
permissions. Suitable as a first redaction pass after
text.extract before any cloud-LLM step.
Detection categories:
EMAIL RFC-5321-ish local@domain, IDN-aware.
PHONE International (+CC …) and DE national
(030 …, 0151-…) shapes, 7..20 raw digits.
IBAN Word-bounded [A-Z]{2}\d{2}[A-Z0-9]{11,30}.
Structural only — MOD-97 checksum
deliberately skipped so partial / truncated
tokens in running text still get redacted.
BIC 8 or 11 uppercase alnum.
IPV4 Four 0..255 octets, dot-separated.
GERMAN_TAX_ID 11 consecutive digits, word-bounded.
CUSTOM Operator-supplied bare terms from the
newline-separated `custom_terms` input,
matched whole-word case-insensitive.
Token shape: ⟦TYPE_N⟧ — U+27E6 / U+27E7 mathematical white
square brackets. Distinct from any plain ASCII `[…]` already
present in source text (Markdown links, legal citations,
code blocks) so a reviewer never has to guess which `[…]`
is a redaction.
Outputs:
anonymized text Input with PII replaced by ⟦TYPE_N⟧.
Counter restarts at 1 per type so the
tokens stay operator-readable.
report json { redactions: [{type, token, original,
offset}…], counts: { TYPE: n, … } }.
Full original-text reconstruction is
possible from this — the GDPR
Art. 32(1)(a) "ability to undo"
requirement.
Quality bar (7 unit tests):
* email round-trip
* IBAN + BIC don't eat each other
* three phone-number shapes redact
* IPv4 only matches valid 0..255 octets
* custom_terms case-insensitive
* no double-redaction on overlapping patterns
* per-category counter resets correctly
Built artefact: target/wasm32-wasip2/release/text_anonymize.wasm
(~180 KiB stripped).
NER for free-text names / organisations / locations is the
v0.2.0 plan once a benchmarked ONNX model is selected; the
operator's `custom_terms` field is the v0.1.0 escape hatch.
Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
commit
734d8f6e6f
8 changed files with 1164 additions and 0 deletions
74
MODULE.de.md
Normal file
74
MODULE.de.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# text.anonymize
|
||||
|
||||
Regex-basierte PII-Anonymisierung für reinen Text. Erkennt
|
||||
typische Personenbezugs-Muster und ersetzt sie durch stabile
|
||||
Tokens der Form `⟦TYPE_N⟧`. Zusätzlich entsteht ein JSON-Bericht,
|
||||
mit dem ein Folge-Schritt die Vollständigkeit prüfen kann.
|
||||
|
||||
## Capability
|
||||
|
||||
- `text.anonymize@0.1.0`
|
||||
|
||||
## Eingaben
|
||||
|
||||
| Name | Typ | Beschreibung |
|
||||
| -------------- | ---- | ---------------------------------------------------------------------------------- |
|
||||
| `text` | text | UTF-8-String, der anonymisiert werden soll. |
|
||||
| `custom_terms` | text | Optional. Newline-getrennte Liste zusätzlicher Begriffe (ganz-Wort, Groß/Klein-egal). |
|
||||
|
||||
## Ausgaben
|
||||
|
||||
| Name | Typ | Beschreibung |
|
||||
| ------------ | ---- | -------------------------------------------------------------------------------------------- |
|
||||
| `anonymized` | text | Der Eingabetext mit PII durch `⟦TYPE_N⟧`-Tokens ersetzt. |
|
||||
| `report` | json | `{ redactions: [{ type, token, original, offset }…], counts: { TYPE: n, … } }`. |
|
||||
|
||||
## Erkannte Kategorien
|
||||
|
||||
| Typ | Muster |
|
||||
| --------------- | ------------------------------------------------------------------------------- |
|
||||
| `EMAIL` | RFC-5321-nahe `local@domain`, IDN-Domains zulässig. |
|
||||
| `PHONE` | International (`+49 …`) oder deutsch-national (`030 …`, `0151-…`). |
|
||||
| `IBAN` | Wortgrenzen, `[A-Z]{2}\d{2}[A-Z0-9]{11,30}`. Nur Struktur — Prüfsumme bewusst nicht. |
|
||||
| `BIC` | 8 oder 11 Großbuchstaben/Ziffern, wortbegrenzt. |
|
||||
| `IPV4` | Vier 0..255-Oktette, punktgetrennt. |
|
||||
| `GERMAN_TAX_ID` | 11 zusammenhängende Ziffern, wortbegrenzt. |
|
||||
| `CUSTOM` | Jede Zeile aus `custom_terms` als ganzes Wort, Groß/Klein-egal. |
|
||||
|
||||
NER kommt erst in v0.2.0 — sobald wir ein gebenchmarktes
|
||||
ONNX-Modell für Namen + Organisationen ausgewählt haben.
|
||||
|
||||
## Berechtigungen
|
||||
|
||||
Keine. Reines Rust, regex-basiert, in-WASM. Kein Dateisystem,
|
||||
kein Netz, kein LLM-Aufruf.
|
||||
|
||||
## Beispiel-Flow
|
||||
|
||||
```yaml
|
||||
name: redact-case-file
|
||||
inputs:
|
||||
document: bytes
|
||||
steps:
|
||||
- id: extract
|
||||
use: text.extract@^0
|
||||
with:
|
||||
document: $inputs.document
|
||||
- id: redact
|
||||
use: text.anonymize@^0
|
||||
with:
|
||||
text: $extract.extracted.pages[*].text
|
||||
custom_terms: |
|
||||
Mustermann
|
||||
Musterfrau
|
||||
outputs:
|
||||
redacted: $redact.anonymized
|
||||
audit_report: $redact.report
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
cargo build --release --target wasm32-wasip2
|
||||
# Ausgabe: target/wasm32-wasip2/release/text_anonymize.wasm
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue