feat: text.anonymize v0.1.0 — regex-based PII redaction
First release. Pure-Rust, in-WASM, regex-only, declares no
permissions. Suitable as a first redaction pass after
text.extract before any cloud-LLM step.
Detection categories:
EMAIL RFC-5321-ish local@domain, IDN-aware.
PHONE International (+CC …) and DE national
(030 …, 0151-…) shapes, 7..20 raw digits.
IBAN Word-bounded [A-Z]{2}\d{2}[A-Z0-9]{11,30}.
Structural only — MOD-97 checksum
deliberately skipped so partial / truncated
tokens in running text still get redacted.
BIC 8 or 11 uppercase alnum.
IPV4 Four 0..255 octets, dot-separated.
GERMAN_TAX_ID 11 consecutive digits, word-bounded.
CUSTOM Operator-supplied bare terms from the
newline-separated `custom_terms` input,
matched whole-word case-insensitive.
Token shape: ⟦TYPE_N⟧ — U+27E6 / U+27E7 mathematical white
square brackets. Distinct from any plain ASCII `[…]` already
present in source text (Markdown links, legal citations,
code blocks) so a reviewer never has to guess which `[…]`
is a redaction.
Outputs:
anonymized text Input with PII replaced by ⟦TYPE_N⟧.
Counter restarts at 1 per type so the
tokens stay operator-readable.
report json { redactions: [{type, token, original,
offset}…], counts: { TYPE: n, … } }.
Full original-text reconstruction is
possible from this — the GDPR
Art. 32(1)(a) "ability to undo"
requirement.
Quality bar (7 unit tests):
* email round-trip
* IBAN + BIC don't eat each other
* three phone-number shapes redact
* IPv4 only matches valid 0..255 octets
* custom_terms case-insensitive
* no double-redaction on overlapping patterns
* per-category counter resets correctly
Built artefact: target/wasm32-wasip2/release/text_anonymize.wasm
(~180 KiB stripped).
NER for free-text names / organisations / locations is the
v0.2.0 plan once a benchmarked ONNX model is selected; the
operator's `custom_terms` field is the v0.1.0 escape hatch.
Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
commit
734d8f6e6f
8 changed files with 1164 additions and 0 deletions
37
Cargo.toml
Normal file
37
Cargo.toml
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# Standalone Cargo.toml — targets wasm32-wasip2.
|
||||
#
|
||||
# Build with:
|
||||
# cargo build --release --target wasm32-wasip2
|
||||
|
||||
[package]
|
||||
name = "text_anonymize"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
authors = ["Dr. Stefan Flemming <platform@flemming.ai>"]
|
||||
license = "Apache-2.0"
|
||||
publish = false
|
||||
description = "F∆I module — regex-based PII anonymization for plain text"
|
||||
repository = "https://git.flemming.ai/fai-modules/text-anonymize"
|
||||
rust-version = "1.85"
|
||||
|
||||
[lib]
|
||||
crate-type = ["cdylib", "rlib"]
|
||||
|
||||
[dependencies]
|
||||
fai-module-sdk = { git = "https://git.flemming.ai/fai/module-sdk.git", branch = "main" }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
# Regex without the unicode-perl / unicode-script / unicode-segment
|
||||
# features keeps the wasm payload <300kB. The patterns this module
|
||||
# uses (email, phone, IBAN, BIC, IPv4, German tax id, name list)
|
||||
# are ASCII-class enough to live without the full Unicode tables.
|
||||
regex = { version = "1.10", default-features = false, features = ["std", "unicode-case", "unicode-perl"] }
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json = "1"
|
||||
|
||||
[profile.release]
|
||||
opt-level = "s"
|
||||
lto = true
|
||||
codegen-units = 1
|
||||
strip = true
|
||||
Loading…
Add table
Add a link
Reference in a new issue