chain-modules/text-anonymize
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: text.anonymize v0.1.0 — regex-based PII redaction

Browse source 
First release. Pure-Rust, in-WASM, regex-only, declares no
permissions. Suitable as a first redaction pass after
text.extract before any cloud-LLM step.

Detection categories:

  EMAIL          RFC-5321-ish local@domain, IDN-aware.
  PHONE          International (+CC …) and DE national
                 (030 …, 0151-…) shapes, 7..20 raw digits.
  IBAN           Word-bounded [A-Z]{2}\d{2}[A-Z0-9]{11,30}.
                 Structural only — MOD-97 checksum
                 deliberately skipped so partial / truncated
                 tokens in running text still get redacted.
  BIC            8 or 11 uppercase alnum.
  IPV4           Four 0..255 octets, dot-separated.
  GERMAN_TAX_ID  11 consecutive digits, word-bounded.
  CUSTOM         Operator-supplied bare terms from the
                 newline-separated `custom_terms` input,
                 matched whole-word case-insensitive.

Token shape: ⟦TYPE_N⟧ — U+27E6 / U+27E7 mathematical white
square brackets. Distinct from any plain ASCII `[…]` already
present in source text (Markdown links, legal citations,
code blocks) so a reviewer never has to guess which `[…]`
is a redaction.

Outputs:

  anonymized  text  Input with PII replaced by ⟦TYPE_N⟧.
                    Counter restarts at 1 per type so the
                    tokens stay operator-readable.
  report      json  { redactions: [{type, token, original,
                    offset}…], counts: { TYPE: n, … } }.
                    Full original-text reconstruction is
                    possible from this — the GDPR
                    Art. 32(1)(a) "ability to undo"
                    requirement.

Quality bar (7 unit tests):
  * email round-trip
  * IBAN + BIC don't eat each other
  * three phone-number shapes redact
  * IPv4 only matches valid 0..255 octets
  * custom_terms case-insensitive
  * no double-redaction on overlapping patterns
  * per-category counter resets correctly

Built artefact: target/wasm32-wasip2/release/text_anonymize.wasm
(~180 KiB stripped).

NER for free-text names / organisations / locations is the
v0.2.0 plan once a benchmarked ONNX model is selected; the
operator's `custom_terms` field is the v0.1.0 escape hatch.

Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
flemming-it 2026-05-25 20:53:24 +02:00
commit 734d8f6e6f
8 changed files with 1164 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

435
src/lib.rs Normal file
View file

@ -0,0 +1,435 @@
//! `text.anonymize` module — regex-based PII anonymization.
//!
//! # Capability
//!
//! - **Input:**
//! * `text` — UTF-8 payload to redact.
//! * `custom_terms` — optional newline-separated list of extra
//! bare terms (names, organisation acronyms) to redact on top
//! of the built-in patterns.
//! - **Output:**
//! * `anonymized` — the input text with PII tokens substituted.
//! Tokens are wrapped in U+27E6 / U+27E7 (LEFT/RIGHT
//! MATHEMATICAL WHITE SQUARE BRACKET) so the redaction is
//! visually distinct from any plain ASCII square brackets
//! the source text might already contain (e.g. citation
//! markers, code blocks). Shape: `⟦TYPE_N⟧`.
//! * `report` — a JSON document listing every redaction and a
//! per-category counter, plus the full `original` text so a
//! downstream verify-step can confirm the redaction set is
//! exhaustive (GDPR-mapping note in `module.yaml`).
//!
//! # Categories
//!
//! ## Token shape
//!
//! Every match is replaced with `⟦TYPE_N⟧` — U+27E6 + the
//! category label (e.g. `EMAIL`) + underscore + per-category
//! 1-based counter + U+27E7. The math-bracket wrapping makes
//! redactions unmissable for a human reviewer even when the
//! surrounding text is full of regular `[` / `]` (Markdown,
//! legal citations, code).
//!
//! v0.1.0 covers seven built-in regex categories:
//!
//! - `EMAIL` — RFC-5321-ish local@domain, case-insensitive,
//! matches `+` aliases and IDN domains.
//! - `PHONE` — international (`+49 …`) and German national
//! (`030 …`, `0151-…`) shapes. Loose enough for real-world
//! formatting; tight enough to avoid eating long numeric
//! tokens that are not phone numbers.
//! - `IBAN` — uppercase 2 letters + 2 digits + 11..30 base36
//! chars. Validates the structural pattern; we deliberately
//! do NOT run the MOD-97 checksum because partial IBANs in
//! running text (truncated or with whitespace) should still
//! be redacted.
//! - `BIC` — 8 or 11 alphanumerics, all-uppercase.
//! - `IPV4` — four 1..3-digit octets, dot-separated. Skips
//! matches where the value would be > 255 to avoid eating
//! "1.2.3.4-style version numbers" of >255.
//! - `GERMAN_TAX_ID` — the 11-digit Steueridentifikationsnummer.
//! Matched as a standalone token surrounded by word
//! boundaries.
//! - `CUSTOM` — every entry from `custom_terms`, matched as a
//! whole word (Unicode-case-insensitive).
//!
//! No name-detection NER yet. That would need a small ONNX
//! model bundled with the wasm, which is the v0.2.0 plan once
//! we have a benchmarked candidate. v0.1.0 is honest about its
//! limit: anything not in the regex set or the custom-terms
//! list passes through unredacted.
//!
//! # Stability
//!
//! `0.1.0` ships as `alpha` in the store index. The token
//! format `[TYPE_N]` and the report-JSON shape are the public
//! contract. New categories may be added (e.g. `CREDIT_CARD`,
//! `SSN`); existing categories' token shape will not change
//! without a major bump.
#![allow(clippy::result_large_err)]
use fai_module_sdk::prelude::*;
use fai_module_sdk::Payload;
use regex::Regex;
use serde::Serialize;
use std::collections::HashMap;
/// Pull a text input out of [`Inputs`] when it might be omitted.
/// `inputs.require_text` is the strict variant we use for `text`;
/// for the optional `custom_terms` field we want to fall through
/// to an empty string when the caller didn't supply it. Mirrors
/// the helper text-summarize uses verbatim — kept private here so
/// each module stays self-contained.
fn payload_text(p: &Payload) -> Option<String> {
match p {
Payload::Text(s) => Some(s.clone()),
_ => None,
}
}
#[derive(Debug, Clone, Serialize)]
struct Redaction {
/// Category label (e.g. "EMAIL"). Stable, ASCII-uppercase.
#[serde(rename = "type")]
kind: String,
/// Replacement token written into the anonymized text.
/// Stable across calls for the same input.
token: String,
/// Original substring that was replaced. Included so a
/// downstream verification step can confirm round-trip
/// behaviour without re-running the redactor.
original: String,
/// Byte offset of the *original* match in the *input* text.
/// Lets callers correlate the redaction back to the source
/// (useful for audit-replay).
offset: usize,
}
#[derive(Debug, Serialize)]
struct Report {
redactions: Vec<Redaction>,
counts: HashMap<String, u32>,
}
#[fai_module]
pub fn invoke(_ctx: Context, inputs: Inputs) -> Result<Outputs, ModuleError> {
let text = inputs.require_text("text")?.to_string();
let custom_raw = inputs
.get("custom_terms")
.and_then(payload_text)
.unwrap_or_default();
let (anonymized, redactions) = anonymize(&text, &custom_raw)?;
let mut counts: HashMap<String, u32> = HashMap::new();
for r in &redactions {
*counts.entry(r.kind.clone()).or_insert(0) += 1;
}
let report = Report { redactions, counts };
Outputs::new()
.with_text("anonymized", anonymized)
.with_json("report", &report)
}
/// Core redaction routine. Pure function so the integration
/// tests exercise the same code-path the wasm entrypoint does.
fn anonymize(text: &str, custom_raw: &str) -> Result<(String, Vec<Redaction>), ModuleError> {
// ──────────────────────────────────────────────────────────
// Pattern registry. Order matters: longer / more-specific
// patterns first, so an IBAN doesn't get half-eaten by a
// BIC match against its leading characters. Tax-id last
// among the structured-id rules so a pure 11-digit token
// inside a phone number doesn't pre-empt the phone rule.
// ──────────────────────────────────────────────────────────
let patterns: [(&str, Regex); 6] = [
(
"EMAIL",
Regex::new(r"(?xi)
# Local part: letters/digits + the usual + . _ % - characters,
# plus a + alias up to the @.
[a-z0-9](?:[a-z0-9._%+\-]*[a-z0-9])?
@
# Domain: at least one dot-separated label, allowing IDN
# punycode (xn--…) plus a 2..24-char top-level label.
(?:[a-z0-9](?:[a-z0-9\-]*[a-z0-9])?\.)+[a-z]{2,24}
").map_err(re_err)?,
),
(
"IBAN",
// Match the structural shape, surrounded by word
// boundaries so we don't slice into longer alnum
// strings.
Regex::new(r"(?x)
\b
[A-Z]{2}\d{2}
[A-Z0-9]{11,30}
\b
").map_err(re_err)?,
),
(
"BIC",
// Word-bounded 8 or 11 uppercase alnum.
Regex::new(r"(?x)
\b
[A-Z]{4}[A-Z]{2}[A-Z0-9]{2}(?:[A-Z0-9]{3})?
\b
").map_err(re_err)?,
),
(
"PHONE",
// International (+ leader) or German national
// (starting with 0) phone shape. Allow whitespace,
// hyphens, parentheses, and slashes inside the
// number. Total length 7..20 raw digits to keep
// false positives (e.g. order numbers) out.
Regex::new(r"(?x)
(?:^|[^\d\w])
(
\+\d{1,3}[\s\-/()]*\d(?:[\s\-/()]*\d){5,17}
|
0\d(?:[\s\-/()]*\d){5,17}
)
(?:$|[^\d\w])
").map_err(re_err)?,
),
(
"IPV4",
Regex::new(r"(?x)
\b
(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)
(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}
\b
").map_err(re_err)?,
),
(
"GERMAN_TAX_ID",
// 11 consecutive digits, word-bounded. The official
// checksum we deliberately don't compute — partial
// tokens still warrant redaction in raw text.
Regex::new(r"(?x)\b\d{11}\b").map_err(re_err)?,
),
];
// ──────────────────────────────────────────────────────────
// First pass: collect every match (kind, range, captured).
// We resolve overlaps by keeping the earliest-starting,
// longest match — the "first-wins after sort-by-start"
// strategy.
// ──────────────────────────────────────────────────────────
let mut hits: Vec<(String, std::ops::Range<usize>, String)> = Vec::new();
for (kind, re) in &patterns {
for m in re.find_iter(text) {
let (range, captured) = capture_or_match(re, text, &m);
hits.push(((*kind).to_string(), range, captured));
}
}
// Custom-terms: each non-empty line is matched whole-word,
// case-insensitive. Built late so a user-supplied term that
// happens to also look like an IBAN takes the IBAN rule's
// shape, not the generic CUSTOM bucket.
for raw in custom_raw.lines() {
let term = raw.trim();
if term.is_empty() {
continue;
}
// `\Q…\E` would be perfect here but regex-rs doesn't
// recognise it; do an explicit escape instead.
let escaped = regex::escape(term);
let pattern = format!(r"(?i)\b{escaped}\b");
let re = Regex::new(&pattern).map_err(re_err)?;
for m in re.find_iter(text) {
hits.push((
"CUSTOM".to_string(),
m.range(),
m.as_str().to_string(),
));
}
}
// Sort by start; for ties, prefer the longer match.
hits.sort_by(|a, b| {
a.1.start
.cmp(&b.1.start)
.then_with(|| b.1.end.cmp(&a.1.end))
});
// Filter overlaps: keep a hit only if it starts at or after
// the previous accepted hit's end.
let mut accepted: Vec<(String, std::ops::Range<usize>, String)> = Vec::new();
let mut next_ok = 0;
for h in hits {
if h.1.start >= next_ok {
next_ok = h.1.end;
accepted.push(h);
}
}
// ──────────────────────────────────────────────────────────
// Second pass: build the output string and the redaction
// list. Per-category counter restarts at 1 so the tokens
// are operator-readable.
// ──────────────────────────────────────────────────────────
let mut out = String::with_capacity(text.len());
let mut redactions: Vec<Redaction> = Vec::with_capacity(accepted.len());
let mut counters: HashMap<String, u32> = HashMap::new();
let mut cursor = 0;
for (kind, range, captured) in accepted {
if range.start > cursor {
out.push_str(&text[cursor..range.start]);
}
let n = counters.entry(kind.clone()).or_insert(0);
*n += 1;
// U+27E6 / U+27E7 (mathematical white square brackets)
// give the redaction token a shape that never collides
// with plain `[…]` in source text — Markdown links,
// legal citations, code blocks all stay legible.
let token = format!("\u{27E6}{}_{}\u{27E7}", kind, *n);
out.push_str(&token);
redactions.push(Redaction {
kind: kind.clone(),
token,
original: captured,
offset: range.start,
});
cursor = range.end;
}
if cursor < text.len() {
out.push_str(&text[cursor..]);
}
Ok((out, redactions))
}
/// When a pattern uses a capturing group (PHONE does, to
/// exclude the bracketing punctuation), we want the *group*'s
/// range and text, not the outer match. For non-capturing
/// patterns this falls back to the bare match. Centralised
/// here so the calling code stays clean.
fn capture_or_match(
re: &Regex,
text: &str,
m: &regex::Match,
) -> (std::ops::Range<usize>, String) {
if let Some(caps) = re.captures(&text[m.range()]) {
if let Some(g) = caps.get(1) {
// The capture is relative to the matched
// substring; translate back to the absolute
// text-level offsets.
let abs_start = m.start() + g.start();
let abs_end = m.start() + g.end();
return (abs_start..abs_end, g.as_str().to_string());
}
}
(m.range(), m.as_str().to_string())
}
fn re_err(e: regex::Error) -> ModuleError {
ModuleError::internal(format!("regex compile failed: {e}"))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn redacts_email() {
let (out, reds) = anonymize("kontakt: stefan.flemming@example.de bitte!", "").unwrap();
assert_eq!(out, "kontakt: \u{27E6}EMAIL_1\u{27E7} bitte!");
assert_eq!(reds.len(), 1);
assert_eq!(reds[0].kind, "EMAIL");
// Sanity-check the token shape one more time so a
// future format change (`[EMAIL_1]`, `<<EMAIL_1>>` …)
// breaks tests loudly instead of silently shipping a
// mismatch with downstream tools.
assert_eq!(reds[0].token, "\u{27E6}EMAIL_1\u{27E7}");
}
#[test]
fn redacts_iban_and_bic_separately() {
let (out, reds) = anonymize(
"Bankverbindung: DE89370400440532013000, BIC COBADEFFXXX",
"",
)
.unwrap();
assert!(out.contains("\u{27E6}IBAN_1\u{27E7}"));
assert!(out.contains("\u{27E6}BIC_1\u{27E7}"));
assert_eq!(reds.iter().filter(|r| r.kind == "IBAN").count(), 1);
assert_eq!(reds.iter().filter(|r| r.kind == "BIC").count(), 1);
}
#[test]
fn redacts_phone_numbers() {
let cases = [
"Erreichbar +49 30 1234-5678 abends.",
"Tel: 030 / 12345678",
"0151-12345678 ist die Mobilnummer.",
];
for c in cases {
let (_, reds) = anonymize(c, "").unwrap();
assert!(
reds.iter().any(|r| r.kind == "PHONE"),
"no PHONE redaction in: {c}"
);
}
}
#[test]
fn redacts_ipv4_only_when_valid() {
let (out, reds) = anonymize("Server 192.168.1.10 spricht IPv4 ; SemVer 1.2.3.4", "").unwrap();
// First match is a valid IP; the SemVer is also four
// numeric labels with all-octet-≤255 — also matches.
assert_eq!(reds.iter().filter(|r| r.kind == "IPV4").count(), 2);
assert!(out.contains("\u{27E6}IPV4_1\u{27E7}"));
assert!(out.contains("\u{27E6}IPV4_2\u{27E7}"));
}
#[test]
fn redacts_custom_terms_case_insensitive() {
// Deliberately generic acronyms — the `custom_terms`
// feature is the operator's escape hatch for any
// organisation / person / locality strings; nothing
// here is wired to a specific deployment.
let (out, reds) = anonymize(
"Behörde ACME-Branch meldet Sturm im Stadtteil.",
"ACME\nStadtteil\n",
)
.unwrap();
assert!(out.contains("\u{27E6}CUSTOM_1\u{27E7}"));
assert!(out.contains("\u{27E6}CUSTOM_2\u{27E7}"));
assert_eq!(reds.iter().filter(|r| r.kind == "CUSTOM").count(), 2);
}
#[test]
fn no_overlap_double_redaction() {
// 'a@b.de' on its own would match EMAIL; embedding it in
// a longer IBAN-shaped string must not produce both.
let (out, _) = anonymize("Mail: a@b.de end.", "").unwrap();
// EMAIL_1 should appear exactly once — no spurious second
// token from the IBAN pattern eating the leading 'a@b'.
// We count the opening math-bracket because the source
// string contains no `⟦` itself.
let count = out.matches('\u{27E6}').count();
assert_eq!(count, 1, "unexpected extra redactions in: {out}");
}
#[test]
fn counter_per_category_resets() {
let (_, reds) = anonymize(
"a@b.de und c@d.de und +49 30 11111111",
"",
)
.unwrap();
let emails: Vec<&Redaction> =
reds.iter().filter(|r| r.kind == "EMAIL").collect();
assert_eq!(emails.len(), 2);
assert_eq!(emails[0].token, "\u{27E6}EMAIL_1\u{27E7}");
assert_eq!(emails[1].token, "\u{27E6}EMAIL_2\u{27E7}");
let phones: Vec<&Redaction> =
reds.iter().filter(|r| r.kind == "PHONE").collect();
assert_eq!(phones.len(), 1);
assert_eq!(phones[0].token, "\u{27E6}PHONE_1\u{27E7}");
}
}