Skip to content

feat(ofd): convert fixed-layout documents to Markdown - #209

Open
lihongjie0209 wants to merge 3 commits into
firecrawl:mainfrom
lihongjie0209:feat/ofd-markdown-208
Open

lihongjie0209 wants to merge 3 commits into
firecrawl:mainfrom
lihongjie0209:feat/ofd-markdown-208

Conversation

@lihongjie0209

Copy link
Copy Markdown

Summary

  • Recognize OFD packages and .ofd paths across the Rust, Node.js, Python, and Wasm APIs.
  • Parse document bodies, pages, templates, and Unicode TextCode runs using anydoc's bounded ZIP/XML readers, without depending on ofdrw.
  • Preserve outline and large-font headings, bold/italic text, printed lists, bordered table grids, and URI links in the shared document model and Markdown output.
  • Report encrypted packages and image-only pages explicitly; document the fixed-layout inference limits.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features --locked --offline -- -D warnings
  • cargo test --locked --offline
  • npm ci && npm run build && npm test in node/ (22 tests; generated declarations checked)
  • Built the Python 3.13 wheel with maturin and ran python -m unittest discover -s python/tests (13 tests)
  • cargo clippy -p anydoc-wasm --target wasm32-unknown-unknown --locked --offline -- -D warnings
  • wasm-pack build wasm --release --target web --scope firecrawl and node --test wasm/test.mjs (8 tests)

Scope notes

OFD is fixed-layout: larger-font headings and complete bordered table grids use conservative layout inference. Borderless/irregular tables, images, vector drawings, non-link annotations, and OCR are not converted. Image-only pages return NeedsOcr rather than incomplete Markdown.

Closes #208

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 24 files

Reply to a comment to ask cubic a question or push back. It learns from your replies.
Click Fix with cubic on a comment, or reply @cubic-dev-ai fix this, and cubic writes the fix.

View guided diff | Re-trigger cubic

Comment thread node/README.md Resolved
Comment thread src/formats/ofd.rs Resolved
Comment thread src/formats/ofd.rs Resolved
Comment thread node/cli.js Resolved
Comment thread wasm/src/typescript.rs Resolved
Comment thread src/lib.rs Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 13 files (changes from recent commits).

Confidence score: 4/5

  • In python/anydoc/__init__.py, a valid PDF with %PDF- beyond the 1,024-byte detection window is rejected instead of being sent to hosted Parse. Preserve the path-based format fallback for this case.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="python/anydoc/__init__.py">

<violation number="1" location="python/anydoc/__init__.py:88">
P2: This guard drops the native path extension fallback: a valid `.pdf` whose `%PDF-` header is beyond the 1,024-byte content-detection window is rejected instead of sent to hosted Parse. Fall back to `format_from_path(path)` as the native path API does.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread python/anydoc/__init__.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: convert OFD documents to Markdown

1 participant