feat: initial text-extract v0.1.0 (text.extract@0.1.0)
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s
Some checks failed
CI / Linux x86_64 (Forgejo) (push) Failing after 2s
First real F∆I capability module. Ships as alpha in the store
index — installable, but layout-naive PDF extraction. Bumps to
1.0.0 require either the planned service-mode variant
(host-services-backed pdfium/mupdf) or confirmed-good
extraction quality on a representative document set.
Capability surface:
Inputs: document : bytes (PDF or DOCX)
Outputs: extracted : json (ExtractedDocument)
Permissions: none — pure in-WASM, no filesystem, no network
Output schema:
{
"engine": "pdf-extract" | "zip+quick-xml",
"engine_version": "<version>",
"pages": [{ "number": <u32>, "text": <string>,
"confidence?": <f64>, "bbox?": {x,y,w,h} }]
}
The optional confidence and bbox fields are reserved for the
future service-mode variant. The host-services infrastructure
that backs pdfium/mupdf does not exist yet (Phase 0.5+); when
it lands, text-extract v0.2.0 will populate these fields without
an API break.
Engines:
PDF: pdf-extract 0.7 (pure Rust, MIT-licensed). Single-column
documents extract reliably. Multi-column or table-heavy
layouts may produce reordered text — documented in
README as a known limitation of the in-WASM v0.1.0 path.
DOCX: ad-hoc parser using the zip and quick-xml crates. Walks
word/document.xml, concatenates run-level text, inserts a
newline at each paragraph boundary. Tables, footnotes,
and embedded objects are not extracted in v0.1.0.
The module is the first reference user of fai-module-sdk. The
public surface is one #[fai_module] function — no wit_bindgen,
no Guest impl, no allow attributes. Testing surface is the same
function via crate-type = ["cdylib", "rlib"]; the SDK macro
gates its WASM glue on target_arch = "wasm32" so host integration
tests work out of the box.
Tests: 6 unit + 3 integration, all green. The DOCX integration
test builds a minimal valid DOCX in memory and verifies the
JSON output schema. PDF integration coverage will land in a
follow-up once a representative test fixture is bundled.
The Forgejo CI workflow clones fai/module-sdk into a sibling
location to satisfy the path-based dev dependency. The path
will switch to a versioned git tag once SDK releases stabilize.
Signed-off-by: flemming-it <sf@flemming.it>
This commit is contained in:
commit
cc72db0421
11 changed files with 1580 additions and 0 deletions
88
.forgejo/workflows/ci.yml
Normal file
88
.forgejo/workflows/ci.yml
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
RUSTFLAGS: "-D warnings"
|
||||
RUST_TOOLCHAIN: "1.86"
|
||||
|
||||
jobs:
|
||||
ci:
|
||||
name: Linux x86_64 (Forgejo)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
# Manual external-URL checkout — see fai/platform CI for
|
||||
# background. Lays this repo at $GITHUB_WORKSPACE.
|
||||
- name: Checkout text-extract via external URL
|
||||
run: |
|
||||
set -eu
|
||||
mkdir -p "$GITHUB_WORKSPACE"
|
||||
cd "$GITHUB_WORKSPACE"
|
||||
git init -q
|
||||
git remote add origin \
|
||||
"https://x-access-token:${GITHUB_TOKEN}@git.flemming.ws/${GITHUB_REPOSITORY}.git"
|
||||
git fetch --depth=1 origin "$GITHUB_SHA"
|
||||
git checkout -q FETCH_HEAD
|
||||
|
||||
# text-extract depends on fai-module-sdk via a sibling-path
|
||||
# dependency: ../../module_sdk/crates/fai-module-sdk
|
||||
# We materialize that layout by cloning module-sdk two
|
||||
# levels up from $GITHUB_WORKSPACE. This is a transient
|
||||
# arrangement until the SDK is published as a versioned
|
||||
# git tag or registry crate.
|
||||
- name: Checkout fai/module-sdk into sibling layout
|
||||
run: |
|
||||
set -eu
|
||||
parent="$(dirname "$GITHUB_WORKSPACE")"
|
||||
grandparent="$(dirname "$parent")"
|
||||
sdk_dir="$grandparent/module_sdk"
|
||||
mkdir -p "$sdk_dir"
|
||||
cd "$sdk_dir"
|
||||
git init -q
|
||||
git remote add origin \
|
||||
"https://x-access-token:${GITHUB_TOKEN}@git.flemming.ws/fai/module-sdk.git"
|
||||
git fetch --depth=1 origin main
|
||||
git checkout -q FETCH_HEAD
|
||||
echo "module-sdk checked out at: $sdk_dir"
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
apt-get update -qq
|
||||
apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
ca-certificates \
|
||||
build-essential \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
git
|
||||
|
||||
- name: Install Rust toolchain
|
||||
run: |
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
|
||||
| sh -s -- -y \
|
||||
--profile minimal \
|
||||
--default-toolchain "$RUST_TOOLCHAIN" \
|
||||
--target wasm32-wasip2 \
|
||||
--component rustfmt \
|
||||
--component clippy
|
||||
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Cargo fmt --check
|
||||
run: cargo fmt --all -- --check
|
||||
|
||||
- name: Cargo clippy
|
||||
run: cargo clippy --all-targets -- -D warnings
|
||||
|
||||
- name: Cargo build (host)
|
||||
run: cargo build --all-targets
|
||||
|
||||
- name: Cargo test (host)
|
||||
run: cargo test --all-targets
|
||||
|
||||
- name: Cargo build (wasm32-wasip2)
|
||||
run: cargo build --release --target wasm32-wasip2
|
||||
Loading…
Add table
Add a link
Reference in a new issue