feat: initial text-extract v0.1.0 (text.extract@0.1.0)
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s
First real F∆I capability module. Ships as alpha in the store
index — installable, but layout-naive PDF extraction. Bumps to
1.0.0 require either the planned service-mode variant
(host-services-backed pdfium/mupdf) or confirmed-good
extraction quality on a representative document set.
Capability surface:
Inputs: document : bytes (PDF or DOCX)
Outputs: extracted : json (ExtractedDocument)
Permissions: none — pure in-WASM, no filesystem, no network
Output schema:
{
"engine": "pdf-extract" | "zip+quick-xml",
"engine_version": "<version>",
"pages": [{ "number": <u32>, "text": <string>,
"confidence?": <f64>, "bbox?": {x,y,w,h} }]
}
The optional confidence and bbox fields are reserved for the
future service-mode variant. The host-services infrastructure
that backs pdfium/mupdf does not exist yet (Phase 0.5+); when
it lands, text-extract v0.2.0 will populate these fields without
an API break.
Engines:
PDF: pdf-extract 0.7 (pure Rust, MIT-licensed). Single-column
documents extract reliably. Multi-column or table-heavy
layouts may produce reordered text — documented in
README as a known limitation of the in-WASM v0.1.0 path.
DOCX: ad-hoc parser using the zip and quick-xml crates. Walks
word/document.xml, concatenates run-level text, inserts a
newline at each paragraph boundary. Tables, footnotes,
and embedded objects are not extracted in v0.1.0.
The module is the first reference user of fai-module-sdk. The
public surface is one #[fai_module] function — no wit_bindgen,
no Guest impl, no allow attributes. Testing surface is the same
function via crate-type = ["cdylib", "rlib"]; the SDK macro
gates its WASM glue on target_arch = "wasm32" so host integration
tests work out of the box.
Tests: 6 unit + 3 integration, all green. The DOCX integration
test builds a minimal valid DOCX in memory and verifies the
JSON output schema. PDF integration coverage will land in a
follow-up once a representative test fixture is bundled.
The Forgejo CI workflow clones fai/module-sdk into a sibling
location to satisfy the path-based dev dependency. The path
will switch to a versioned git tag once SDK releases stabilize.
Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
commit
cc72db0421
11 changed files with 1580 additions and 0 deletions
85
README.md
Normal file
85
README.md
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
# text-extract
|
||||
|
||||
F∆I module — extract plain text from PDF and DOCX documents.
|
||||
|
||||
## Capability
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Capability | `text.extract@0.1.0` |
|
||||
| Inputs | `document: bytes` (PDF or DOCX) |
|
||||
| Outputs | `extracted: json` (ExtractedDocument) |
|
||||
| Permissions | _none_ — pure in-WASM, no filesystem, no network |
|
||||
| Status (in store index) | `alpha` |
|
||||
|
||||
## Output schema
|
||||
|
||||
```json
|
||||
{
|
||||
"engine": "pdf-extract",
|
||||
"engine_version": "0.7",
|
||||
"pages": [
|
||||
{ "number": 1, "text": "..." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`confidence` and `bbox` per page are reserved for a future
|
||||
service-mode variant (host-services-backed pdfium/mupdf engine).
|
||||
v0.1.0 does not populate them; the schema accepts them as
|
||||
optional so that v0.2.0 can extend without an interface break.
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
cargo build --release --target wasm32-wasip2
|
||||
# Artifact: target/wasm32-wasip2/release/text_extract.wasm
|
||||
```
|
||||
|
||||
The build depends on a sibling checkout of `fai/module-sdk`:
|
||||
|
||||
```
|
||||
~/fai/
|
||||
├── module_sdk/ # cloned from fai/module-sdk
|
||||
└── fai_modules/
|
||||
└── text-extract/ # this repo
|
||||
```
|
||||
|
||||
This is a temporary path-based dependency. Once `fai-module-sdk`
|
||||
is published to a registry or a stable git tag, this module's
|
||||
`Cargo.toml` will switch over. The Forgejo CI workflow checks out
|
||||
the SDK repo into a sibling directory before building.
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
cargo test # 9 tests: 6 unit + 3 integration
|
||||
cargo test --release # same, optimized
|
||||
```
|
||||
|
||||
The DOCX path is covered end-to-end by an integration test that
|
||||
builds a minimal valid DOCX in memory and feeds it through
|
||||
`invoke`. The PDF path is covered by unit tests on the page
|
||||
splitter; full PDF coverage will land once the platform's hub
|
||||
loads this wasm component as part of its store-install
|
||||
integration test.
|
||||
|
||||
## Limitations of v0.1.0
|
||||
|
||||
- **Layout-naive PDF parsing.** Multi-column layouts and tables
|
||||
may produce reordered text. Single-column documents work
|
||||
reliably.
|
||||
- **DOCX text only.** Tables, footnotes, headers/footers and
|
||||
embedded objects are not yet extracted. Plain paragraph text
|
||||
with run-level concatenation is the v0.1.0 surface.
|
||||
- **No OCR.** Image-only PDFs return empty text per page.
|
||||
An OCR variant will arrive as a separate module
|
||||
(`text.ocr@x.y.z`) once the host-services infrastructure
|
||||
for tesseract is in place.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0. See `LICENSE`.
|
||||
|
||||
Author: Dr. Stefan Flemming, Flemming.AI <platform@flemming.ai>
|
||||
Repository: https://git.flemming.ws/fai-modules/text-extract
|
||||
Loading…
Add table
Add a link
Reference in a new issue