Skip to content

Add opt-in structured XML exemption for Europass CV PDFs - #658

Merged
iobrado merged 9 commits into
release/10.6.0from
feature/pdf-embedded-xml-exemption
Sep 25, 2026
Merged

iobrado merged 9 commits into
release/10.6.0from
feature/pdf-embedded-xml-exemption

Conversation

@iobrado

@iobrado iobrado commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

This adds an opt-in exemption so the PDF scanner stops rejecting Europass CVs, whose builder attaches a machine-readable XML copy of every CV it exports.

Why

  • A real Europass export embeds attachment.xml, a <Candidate> document in the http://www.europass.eu/1.0 namespace.
  • The file carries no /Subtype, /AFRelationship or /AF, so only the payload bytes can identify it.
  • /EmbeddedFile was only exempt for C2PA manifests, so every Europass CV came back as validationFilePdfUnsafe.

Embedded-file verifier refactor

  • Rename C2paManifestVerifier to EmbeddedFileVerifier, which keeps only the structural walk.
  • Move the JUMBF/C2PA check unchanged into C2paPayloadValidator.
  • Add EmbeddedPayloadValidatorInterface (maxBytes(), accepts($payload, $name)).
  • Require every embedded stream to pass at least one enabled validator.
  • Keep all existing defences: qpdf-only, /ObjStm, /RF, duplicate objects, /Filter, the endstream extent check and the backstop.

File specification names

  • Read /F and /UF from the innermost dictionary that contains the /EF, with a single linear pass over the brackets.
  • Treat the name as optional, since C2PA Filespecs often have none.
  • Reject a name that is unreadable, duplicated, non-ASCII, or different between /F and /UF.
  • Reject one stream reached under two different names, which closes "verified as attachment.xml, saved as payload.exe".

Structured XML exemption

  • Add StructuredXmlPayloadValidator, which checks in this order:
    • An exact, case-sensitive name on the allowlist (attachment.xml, Europass-XML-Attachment.xml)
    • The 5 MB cap
    • Strict UTF-8 with no control bytes
    • No <!DOCTYPE or <!ENTITY in the raw bytes
    • A full XMLReader parse with LIBXML_NONET
    • The allowlisted root element and namespace
  • Add Config::FILE_UPLOAD_PDF_STRUCTURED_XML_ALLOWLIST and FILE_UPLOAD_PDF_STRUCTURED_XML_MAX_BYTES.
  • Add the fileSecurityPdfAllowStructuredXml filter, registered in Filters.php.
  • The filter is off by default, needs exactly true and needs qpdf, the same as fileSecurityPdfAllowC2pa.
  • Add an xmlreader row to the File security diagnostics tab.
add_filter('es_forms_validation_file_security_pdf_allow_structured_xml', '__return_true');

Deliberately not exempted

  • LibreOffice hybrid PDFs (Original.odt), which are ZIPs that can carry macros.
  • PDF Portfolios and LaTeX attachfile/embedfile.
  • Factur-X, ZUGFeRD and XRechnung invoices. Adding them later only needs allowlist entries, once real samples confirm the details.

Accepted residual risk

  • Base64 data inside the Europass XML (a photo, and legacy attached documents) passes through as inert text.
  • No reader extracts or opens it. /JS, /Launch, /RichMedia and the other blocked keys still apply to the whole PDF.

Tests

  • verify-c2pa.php and verify-pdf-scanner.php changed only in class names, and every existing case passes.
  • The new tests/security/verify-structured-xml.php harness has 59 checks, and its synthetic fixtures mean no real CV is committed.
  • I disabled each key defence in turn to confirm that at least one check fails without it.
  • A real Europass export is accepted with the filter on, and rejected by default, with C2PA only, with a truthy 1, and without qpdf.
  • composer test, bun run lintJs and bun run lintStyle all pass.

Changelog (for the release)

  • Added fileSecurityPdfAllowStructuredXml filter. When enabled, the PDF scanner accepts Europass CVs, whose only dangerous key is the embedded machine-readable CV XML. The file name must be on an allowlist and the payload must parse as the expected Europass document with no DTD. Disabled by default and requires qpdf, like fileSecurityPdfAllowC2pa.

Known follow-ups

  • Check the legacy SkillsPassport format against a real 2013–2020 export.
  • The pre-commit hook fails on commits that contain only tests/ PHP files, because PHPStan excludes tests/.

🤖 Generated with Claude Code

iobrado and others added 2 commits September 24, 2026 10:44
C2paManifestVerifier did two jobs: resolve every embedded stream the way
a PDF reader would, and judge whether the bytes are a C2PA manifest. The
structural walk becomes EmbeddedFileVerifier and the JUMBF logic moves
unchanged into C2paPayloadValidator, behind a new
EmbeddedPayloadValidatorInterface. Every embedded stream must now be
accepted by at least one supplied validator.

The walk also reads each file specification's /F and /UF name, taken
from the innermost dictionary around the /EF, so validators can match
on it. Names are optional, since C2PA signers often write neither. A name
that is present but unreadable, duplicated, non-ASCII, or that differs
between /F and /UF rejects the file, and so does one stream reached under
two different names.

No behaviour change for C2PA: the existing harnesses pass unmodified
apart from class names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Europass CV builder attaches a machine-readable copy of every CV it
exports as an embedded XML file, so the PDF scanner rejected all Europass
CVs. The real export has no /Subtype, /AFRelationship or /AF, so only the
payload bytes can identify it.

StructuredXmlPayloadValidator accepts an embedded file only when its
file specification name is an exact key in
FILE_UPLOAD_PDF_STRUCTURED_XML_ALLOWLIST (attachment.xml,
Europass-XML-Attachment.xml) and the payload:
- is at most 5 MB,
- is valid UTF-8 with no control bytes,
- contains no DOCTYPE or ENTITY,
- parses completely with XMLReader under LIBXML_NONET,
- has the allowlisted root element and namespace.

Opt-in through the new fileSecurityPdfAllowStructuredXml filter, which
is off by default, compared with === true and requires qpdf, exactly as
fileSecurityPdfAllowC2pa does. The diagnostics tab gains an xmlreader row.

tests/security/verify-structured-xml.php is a dependency-free harness.
Its fixtures are synthetic PDFs shaped like a real Europass export, so
no real CV is committed. It covers:
- name tricks,
- spoofed roots and namespaces,
- XXE and DTDs,
- appended payloads,
- executables under an allowlisted name,
- mixed attachments,
- the size cap and cost bounds,
- the scanner's decision with and without the filter and qpdf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
iobrado and others added 4 commits September 24, 2026 12:54
- EmbeddedFileVerifier::fileSpecName() now returns null (not false) for an
  unreadable /F or /UF, so a C2PA-only site no longer rejects a valid
  manifest just because its file name is non-ASCII, empty, or the /F and
  /UF values disagree. Only the XML validator needs a name, and it already
  rejects null.
- Extract PDF literal/hex string reading (skipLiteral, readAsciiText) into
  a new PdfStrings helper, and the /EF owner-dictionary lookup into
  innermostOpenDictionary(), to keep EmbeddedFileVerifier focused on
  structure.
- Config::FILE_UPLOAD_PDF_STRUCTURED_XML_ALLOWLIST entries use named
  root/namespace keys instead of positional tuples.
- Add regression checks in verify-c2pa.php for the name-handling fix
  (absent, ASCII, non-ASCII UTF-16, mismatched, and empty names).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- EmbeddedFileVerifier: replace the by-reference innermostOpenDictionary()
  walk with owningDictionaryOffsets(), a pure single-pass lookup over all
  /EF offsets in a region.
- Document that name pinning across two file specifications applies
  whichever payload validators are enabled, and add C2PA coverage for a
  stream reached under one name twice vs. two different names.
- FileSecurityDiagnostics: mark the xmlreader extension row optional,
  matching the qpdf row.
- Config: collapse the C2PA/structured-XML exemption docblock to one
  paragraph pointing at EmbeddedFileVerifier and its validators, rather
  than repeating their per-class docs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move the shared "structure only, not a safety proof" warning to the
EmbeddedPayloadValidatorInterface docblock instead of repeating it in
both C2paPayloadValidator and StructuredXmlPayloadValidator, and
shorten several comments in EmbeddedFileVerifier and
StructuredXmlPayloadValidator that restated what the adjacent code
already shows. No behavioural change; all three security test
harnesses still pass (57/22/56 checks).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iobrado
iobrado changed the base branch from main to release/10.6.0 September 24, 2026 13:01
@iobrado
iobrado marked this pull request as ready for review September 24, 2026 13:02
@iobrado
iobrado requested a review from a team September 24, 2026 13:02

@goranalkovic-infinum goranalkovic-infinum left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Europass exception looks well scoped, and keeping it off by default is a good choice. I’d request two changes before merging:

  1. Fix repeated scanning of nested dictionaries. The new filename lookup repeatedly scans the same content when PDF dictionaries are nested. With a roughly 1 MB PDF containing 100 nested dictionaries, verification increased from 0.09s to 8.48s after qpdf processing. This affects the existing C2PA exception too and could let repeated uploads tie up server workers. Please collect filenames in one pass or cap the processing work, and add a regression test.

  2. Run the security tests in CI. composer test doesn’t execute the three security harnesses, so GitHub can stay green even if attachment validation breaks. Please run them in CI with generated fixtures, qpdf, and the required PHP extensions.

@piqusy piqusy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced both open findings locally (PHP 8.4.25, qpdf 12.4.1, xmlreader loaded). All three harnesses pass on this branch. Requesting changes on the same two points, with measurements.

1. Nested dictionaries re-scan the region per level (perf regression)

collectEmbeddedFileReferences() in src/Validation/FileSecurity/EmbeddedFileVerifier.php caches owner names by $ownerStart, so a body where every nesting level owns an /EF re-runs readDictionary() plus fileSpecName() (a mask() byte loop and two preg_match_all passes) over the remaining region for each level. The "read each owner once" comment only holds when owner dictionaries are small. The inline-filespecs case in the harness stays linear; this shape does not. It also affects C2PA-only opt-ins, because name reading happens regardless of which validators are enabled.

Measured on the same 0.95 MB body with 100 nested dictionaries and one embedded stream, identical verdict on both sides:

code time
C2paManifestVerifier (release/10.6.0) 0.08s
EmbeddedFileVerifier (this branch) 7.41s

That is ~93x. Fix direction: record each << extent during the existing bracket walk and derive the name in one pass, or cap cumulative name-scan work and fail closed to name = null. The second keeps semantics intact: the C2PA validator ignores names, and the XML validator then rejects, which is the same treatment a readable-but-unverified name already gets. Please also add a timed nested-dictionary case to tests/security/verify-structured-xml.php; the cost section currently covers inline filespecs and many references to one stream, not nesting.

2. Security harnesses run nowhere in CI

.github/workflows/ci.yml runs composer lint and node lint only. composer test excludes tests/, and the pre-commit hook skips tests-only PHP commits (noted in the PR description). These harnesses are the only executable tests for this code, and the Playwright E2E never reaches the PDF verdicts. Please add a CI job that runs the three harnesses directly:

php tests/security/verify-c2pa.php
php tests/security/verify-structured-xml.php
php tests/security/verify-pdf-scanner.php

They need no vendor directory, they need dom, xmlreader and qpdf, and the qpdf-dependent checks skip cleanly when qpdf is absent, so a plain setup-php runner works.

Verified clean on this branch

Name pinning (one stream reached under two names rejects), /F + /UF disagree, duplicate and non-ASCII rejection, UTF-16BE hex names, XXE / DOCTYPE / SettingContent-ms / wrong-namespace rejection, DTD search on raw bytes before libxml, LIBXML_NONET with no LIBXML_NOENT, and the existing /RF, /ObjStm, /Filter and endstream checks. array_any is fine given "php": ">=8.4" in composer.json.

iobrado and others added 3 commits September 25, 2026 14:28
Each nesting level that owns an /EF re-read the dictionaries inside it,
so a ~1 MB body with 100 nested Filespecs took 7.5s to verify. Name
reads are now capped at twice the region; owners past the cap go
unnamed, which the structured XML validator rejects and C2PA ignores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
composer test never executes tests/security, so attachment validation
could break with CI green. The new job installs qpdf, generates the
fixtures, runs all three harnesses and fails on any SKIP, so a missing
qpdf or fixture cannot pass silently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An owner left unnamed by the name budget also fails the one-stream,
one-name rule when the same stream is reached from a named owner, so
the cap can reject a C2PA-only upload too, not just a structured XML
one. PdfStrings has a single caller, so its header no longer claims
several scanners share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@piqusy piqusy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both review findings are addressed and I verified the fixes locally (PHP 8.4.25, qpdf 12.4.1, xmlreader loaded), so approving.

1. Name-reading cap (afb61859). Measured on the same 0.95 MB body with 100 nested dictionaries and one embedded stream that reproduced the original regression:

code time verdict
release/10.6.0 0.08s accepted
f144724 7.41s accepted
fa56125 0.27s rejected, fail-closed

Cost is linear again. The new 100 nested Filespecs around 1 MB case in tests/security/verify-structured-xml.php passes in 0.27s against a 1.0s budget and pins the fail-closed verdict, so a future regression fails the harness. I also checked the edge where an exhausted budget makes the read limit drop below the owner start: readDictionary() returns null immediately, the capped path records an unnamed owner, and the $limit === strlen($region) guard still rejects a genuinely unreadable uncapped owner. One behavior change worth restating for the record: a stream reached from both a named owner and a capped owner now fails the one-stream, one-name rule, including for C2PA-only opt-ins. Fail-closed and deliberate, fine by me.

2. CI coverage (250ea3dd). The security job installs the right extensions and qpdf, generates fixtures through the existing tests/security/generate-test-files.sh, and fails on both harness failures (bash -e plus pipefail) and skips (the grep '^SKIP' guard). Verified the YAML parses and all three harnesses pass on this head.

Two non-blocking nits are filed in #659; happy either way, they can ride in a follow-up PR if you agree they are worth it.

@iobrado
iobrado merged commit be0c6d9 into release/10.6.0 Sep 25, 2026
3 checks passed
@iobrado
iobrado deleted the feature/pdf-embedded-xml-exemption branch September 25, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants