Compare commits
No commits in common. "fdb5deaa8bb8afa15f579744b0df4075022e7fd6" and "1078367d6a06f6d50502742d3f178c92c6c5c2ca" have entirely different histories.
fdb5deaa8b
...
1078367d6a
3 changed files with 0 additions and 198 deletions
96
MODULE.de.md
96
MODULE.de.md
|
|
@ -1,96 +0,0 @@
|
||||||
# text.extract
|
|
||||||
|
|
||||||
Reine Textextraktion aus PDF- und DOCX-Dokumenten. Pure-Rust,
|
|
||||||
in-WASM, ohne deklarierte Berechtigungen. Der kanonische erste
|
|
||||||
Schritt in Flows, die Rohtext brauchen — anonymisieren,
|
|
||||||
zusammenfassen, übersetzen, per LLM klassifizieren,
|
|
||||||
Audit-redigieren.
|
|
||||||
|
|
||||||
## Capability
|
|
||||||
|
|
||||||
- `text.extract@0.1.0`
|
|
||||||
|
|
||||||
## Eingaben
|
|
||||||
|
|
||||||
| Name | Typ | Beschreibung |
|
|
||||||
| ---------- | ----- | ------------------------------------------------------- |
|
|
||||||
| `document` | bytes | Das zu extrahierende Dokument. MIME-Typen siehe unten. |
|
|
||||||
|
|
||||||
Akzeptierte MIME-Typen (deklariert in
|
|
||||||
`module.yaml#accepts_mime`, von Studio als File-Picker-Filter
|
|
||||||
gelesen):
|
|
||||||
|
|
||||||
- `application/pdf`
|
|
||||||
- `application/x-pdf`
|
|
||||||
- `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
|
|
||||||
|
|
||||||
Andere MIME-Typen werden mit `invalid_input` abgelehnt — vor
|
|
||||||
jedem Parsen.
|
|
||||||
|
|
||||||
## Ausgabe
|
|
||||||
|
|
||||||
| Name | Typ | Beschreibung |
|
|
||||||
| ----------- | ---- | --------------------------------------------- |
|
|
||||||
| `extracted` | json | `ExtractedDocument`-Schema siehe unten. |
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"engine": "pdf-extract",
|
|
||||||
"engine_version": "0.7",
|
|
||||||
"pages": [
|
|
||||||
{ "number": 1, "text": "…" }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`confidence` und `bbox` pro Seite sind im Schema für eine
|
|
||||||
spätere Service-Mode-Variante (pdfium / mupdf via
|
|
||||||
Host-Services) reserviert. v0.1.0 setzt diese Felder nicht;
|
|
||||||
das Schema akzeptiert sie als optional.
|
|
||||||
|
|
||||||
## Berechtigungen
|
|
||||||
|
|
||||||
Keine. Reine Rust-Pipeline (`pdf-extract`, `zip`, `quick-xml`).
|
|
||||||
Kein Dateisystem, kein Netz. `permissions: []` in
|
|
||||||
`module.yaml` macht das explizit; der Hub erzwingt es.
|
|
||||||
|
|
||||||
## Grenzen in v0.1.0
|
|
||||||
|
|
||||||
- **Layout-naives PDF.** Mehrspaltige oder tabellenlastige
|
|
||||||
Dokumente können umsortierten Text liefern. Ein-Spalten-
|
|
||||||
Fließtext ist solide.
|
|
||||||
- **Kein OCR.** Gescannte (nur-Bild-)PDFs liefern pro Seite
|
|
||||||
leeren Text. Künftiges `text.ocr@^0` als Vor-Schritt.
|
|
||||||
- **Nur DOCX bei Office.** `.doc` (Altformat), `.rtf`, `.odt`
|
|
||||||
noch nicht unterstützt.
|
|
||||||
|
|
||||||
`1.0.0` setzt entweder die Service-Mode-Variante
|
|
||||||
(pdfium-Qualität + Layout) voraus oder einen
|
|
||||||
nachgewiesen-guten Extraktions-Score auf einem
|
|
||||||
repräsentativen Korpus.
|
|
||||||
|
|
||||||
## Beispiel-Flow
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
name: extract-anonymise
|
|
||||||
inputs:
|
|
||||||
document: bytes
|
|
||||||
steps:
|
|
||||||
- id: extract
|
|
||||||
use: text.extract@^0
|
|
||||||
with:
|
|
||||||
document: $inputs.document
|
|
||||||
- id: redact
|
|
||||||
use: text.anonymize@^0
|
|
||||||
with:
|
|
||||||
text: $extract.extracted.pages[*].text
|
|
||||||
outputs:
|
|
||||||
redacted: $redact.anonymized
|
|
||||||
```
|
|
||||||
|
|
||||||
## Build
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cargo build --release --target wasm32-wasip2
|
|
||||||
# Ausgabe: target/wasm32-wasip2/release/text_extract.wasm
|
|
||||||
```
|
|
||||||
92
MODULE.md
92
MODULE.md
|
|
@ -1,92 +0,0 @@
|
||||||
# text.extract
|
|
||||||
|
|
||||||
Plain-text extraction from PDF and DOCX documents. Pure-Rust,
|
|
||||||
in-WASM, no permissions required. The canonical first step in
|
|
||||||
any flow that needs raw text — anonymise, summarise, translate,
|
|
||||||
LLM-classify, audit-redact.
|
|
||||||
|
|
||||||
## Capability
|
|
||||||
|
|
||||||
- `text.extract@0.1.0`
|
|
||||||
|
|
||||||
## Inputs
|
|
||||||
|
|
||||||
| Name | Type | Description |
|
|
||||||
| ---------- | ----- | ------------------------------------------------------ |
|
|
||||||
| `document` | bytes | The document to extract. Accepted MIME types below. |
|
|
||||||
|
|
||||||
Accepted MIME types (declared in `module.yaml#accepts_mime`,
|
|
||||||
read by Studio as the file-picker filter):
|
|
||||||
|
|
||||||
- `application/pdf`
|
|
||||||
- `application/x-pdf`
|
|
||||||
- `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
|
|
||||||
|
|
||||||
Other MIME types are rejected with a `invalid_input` error
|
|
||||||
before any parsing happens.
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
| Name | Type | Description |
|
|
||||||
| ----------- | ---- | ------------------------------------------------- |
|
|
||||||
| `extracted` | json | `ExtractedDocument` schema below. |
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"engine": "pdf-extract",
|
|
||||||
"engine_version": "0.7",
|
|
||||||
"pages": [
|
|
||||||
{ "number": 1, "text": "…" }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Per-page `confidence` and `bbox` fields are reserved in the
|
|
||||||
schema for a future service-mode variant (pdfium / mupdf via
|
|
||||||
host services) that surfaces layout information. v0.1.0 does
|
|
||||||
not populate them; the schema accepts them as optional.
|
|
||||||
|
|
||||||
## Permissions
|
|
||||||
|
|
||||||
None. Pure-Rust pipeline (`pdf-extract`, `zip`, `quick-xml`).
|
|
||||||
No filesystem, no network. The empty `permissions: []` in
|
|
||||||
`module.yaml` makes that explicit; the hub enforces it.
|
|
||||||
|
|
||||||
## Limits in v0.1.0
|
|
||||||
|
|
||||||
- **Layout-naive PDF.** Multi-column or table-heavy documents
|
|
||||||
may produce reordered text. Single-column body text is solid.
|
|
||||||
- **No OCR.** Scanned (image-only) PDFs return empty text per
|
|
||||||
page. Pair with a future `text.ocr@^0` step for those.
|
|
||||||
- **DOCX-only Office.** `.doc` (legacy binary), `.rtf`, `.odt`
|
|
||||||
are not yet supported.
|
|
||||||
|
|
||||||
Bumping to `1.0.0` requires either the service-mode variant
|
|
||||||
(pdfium-quality output + layout) or a confirmed-good extraction
|
|
||||||
score on a representative corpus.
|
|
||||||
|
|
||||||
## Example flow
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
name: extract-anonymise
|
|
||||||
inputs:
|
|
||||||
document: bytes
|
|
||||||
steps:
|
|
||||||
- id: extract
|
|
||||||
use: text.extract@^0
|
|
||||||
with:
|
|
||||||
document: $inputs.document
|
|
||||||
- id: redact
|
|
||||||
use: text.anonymize@^0
|
|
||||||
with:
|
|
||||||
text: $extract.extracted.pages[*].text
|
|
||||||
outputs:
|
|
||||||
redacted: $redact.anonymized
|
|
||||||
```
|
|
||||||
|
|
||||||
## Build
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cargo build --release --target wasm32-wasip2
|
|
||||||
# Output: target/wasm32-wasip2/release/text_extract.wasm
|
|
||||||
```
|
|
||||||
10
module.yaml
10
module.yaml
|
|
@ -29,13 +29,3 @@ outputs:
|
||||||
# zip, quick-xml). No filesystem, no network. The empty list makes
|
# zip, quick-xml). No filesystem, no network. The empty list makes
|
||||||
# this explicit; the hub enforces it.
|
# this explicit; the hub enforces it.
|
||||||
permissions: []
|
permissions: []
|
||||||
|
|
||||||
# MIME-type allow-list for the `document` input. Studio reads this
|
|
||||||
# through ModuleInfoResponse.accepts_mime and uses it as the OS
|
|
||||||
# file-picker filter so operators don't accidentally pick a `.png`
|
|
||||||
# for a PDF-extract step. Same list documented inline next to
|
|
||||||
# `inputs.document` above; kept synchronised by hand for now.
|
|
||||||
accepts_mime:
|
|
||||||
- application/pdf
|
|
||||||
- application/x-pdf
|
|
||||||
- application/vnd.openxmlformats-officedocument.wordprocessingml.document
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue