feat: initial text-extract v0.1.0 (text.extract@0.1.0)
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s

First real F∆I capability module. Ships as alpha in the store
index — installable, but layout-naive PDF extraction. Bumps to
1.0.0 require either the planned service-mode variant
(host-services-backed pdfium/mupdf) or confirmed-good
extraction quality on a representative document set.

Capability surface:

  Inputs:   document : bytes  (PDF or DOCX)
  Outputs:  extracted : json  (ExtractedDocument)
  Permissions: none — pure in-WASM, no filesystem, no network

Output schema:

  {
    "engine": "pdf-extract" | "zip+quick-xml",
    "engine_version": "<version>",
    "pages": [{ "number": <u32>, "text": <string>,
                "confidence?": <f64>, "bbox?": {x,y,w,h} }]
  }

The optional confidence and bbox fields are reserved for the
future service-mode variant. The host-services infrastructure
that backs pdfium/mupdf does not exist yet (Phase 0.5+); when
it lands, text-extract v0.2.0 will populate these fields without
an API break.

Engines:

  PDF:  pdf-extract 0.7 (pure Rust, MIT-licensed). Single-column
        documents extract reliably. Multi-column or table-heavy
        layouts may produce reordered text — documented in
        README as a known limitation of the in-WASM v0.1.0 path.

  DOCX: ad-hoc parser using the zip and quick-xml crates. Walks
        word/document.xml, concatenates run-level text, inserts a
        newline at each paragraph boundary. Tables, footnotes,
        and embedded objects are not extracted in v0.1.0.

The module is the first reference user of fai-module-sdk. The
public surface is one #[fai_module] function — no wit_bindgen,
no Guest impl, no allow attributes. Testing surface is the same
function via crate-type = ["cdylib", "rlib"]; the SDK macro
gates its WASM glue on target_arch = "wasm32" so host integration
tests work out of the box.

Tests: 6 unit + 3 integration, all green. The DOCX integration
test builds a minimal valid DOCX in memory and verifies the
JSON output schema. PDF integration coverage will land in a
follow-up once a representative test fixture is bundled.

The Forgejo CI workflow clones fai/module-sdk into a sibling
location to satisfy the path-based dev dependency. The path
will switch to a versioned git tag once SDK releases stabilize.

Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
flemming-it 2026-05-01 16:15:11 +02:00
commit cc72db0421
11 changed files with 1580 additions and 0 deletions

135
tests/docx_roundtrip.rs Normal file
View file

@ -0,0 +1,135 @@
//! End-to-end test for the DOCX path.
//!
//! Builds a minimal valid DOCX in memory, hands it to the
//! `invoke` entry point, and asserts the JSON output matches the
//! expected schema. The DOCX is a ZIP archive containing
//! `word/document.xml` plus the minimum `[Content_Types].xml`
//! that Office tools expect — that is the bare floor for a
//! parser to accept the file.
//!
//! The PDF path is covered by unit tests on the page splitter;
//! end-to-end PDF coverage will land once the platform's hub
//! integration test loads the wasm component and feeds it a
//! real PDF fixture.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::io::Write;
use fai_module_sdk::prelude::*;
use serde::Deserialize;
use text_extract::invoke;
#[derive(Debug, Deserialize)]
struct ExtractedDocument {
engine: String,
engine_version: String,
pages: Vec<Page>,
}
#[derive(Debug, Deserialize)]
struct Page {
number: u32,
text: String,
}
const DOCX_MIME: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
const CONTENT_TYPES_XML: &str = r#"<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
<Default Extension="xml" ContentType="application/xml"/>
<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>
</Types>"#;
const RELS_XML: &str = r#"<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>
</Relationships>"#;
const DOCUMENT_XML: &str = r#"<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">
<w:body>
<w:p><w:r><w:t>The quick brown fox</w:t></w:r></w:p>
<w:p><w:r><w:t xml:space="preserve">jumps over </w:t></w:r><w:r><w:t>the lazy dog.</w:t></w:r></w:p>
</w:body>
</w:document>"#;
fn build_minimal_docx() -> Vec<u8> {
let mut buf = Vec::new();
{
let mut w = zip::ZipWriter::new(std::io::Cursor::new(&mut buf));
let opts: zip::write::SimpleFileOptions = zip::write::SimpleFileOptions::default()
.compression_method(zip::CompressionMethod::Deflated);
w.start_file("[Content_Types].xml", opts).unwrap();
w.write_all(CONTENT_TYPES_XML.as_bytes()).unwrap();
w.start_file("_rels/.rels", opts).unwrap();
w.write_all(RELS_XML.as_bytes()).unwrap();
w.start_file("word/document.xml", opts).unwrap();
w.write_all(DOCUMENT_XML.as_bytes()).unwrap();
w.finish().unwrap();
}
buf
}
fn dummy_context() -> Context {
Context {
invocation_id: "test".into(),
capability_namespace: "text".into(),
capability_name: "extract".into(),
deadline_ms: 0,
}
}
#[test]
fn docx_round_trip_produces_expected_schema() {
let docx = build_minimal_docx();
let inputs = Inputs::from_pairs([("document", Payload::bytes(DOCX_MIME, docx))]);
let outputs = invoke(dummy_context(), inputs).expect("invoke ok");
let pairs = outputs.into_pairs();
assert_eq!(pairs.len(), 1, "exactly one output expected");
assert_eq!(pairs[0].0, "extracted");
let raw = match &pairs[0].1 {
Payload::Json(s) => s,
other => panic!("expected Json payload, got {other:?}"),
};
let doc: ExtractedDocument = serde_json::from_str(raw).expect("valid JSON");
assert_eq!(doc.engine, "zip+quick-xml");
assert_eq!(doc.engine_version, "0.1");
assert_eq!(doc.pages.len(), 1);
assert_eq!(doc.pages[0].number, 1);
assert!(
doc.pages[0].text.contains("quick brown fox"),
"missing first paragraph: {}",
doc.pages[0].text
);
assert!(
doc.pages[0].text.contains("jumps over the lazy dog"),
"missing second paragraph: {}",
doc.pages[0].text
);
}
#[test]
fn unsupported_mime_type_returns_invalid_input() {
let inputs =
Inputs::from_pairs([("document", Payload::bytes("application/zip", vec![0u8; 32]))]);
let err = invoke(dummy_context(), inputs).expect_err("must fail");
let msg = format!("{err}");
assert!(msg.contains("unsupported MIME type"), "msg = {msg}");
}
#[test]
fn missing_document_input_returns_invalid_input() {
let inputs = Inputs::default();
let err = invoke(dummy_context(), inputs).expect_err("must fail");
let msg = format!("{err}");
assert!(msg.contains("missing input"), "msg = {msg}");
}