Skip to content

V2: Merge multiple files (CSV/Excel, PDF, Word, PowerPoint) through chat orchestration and workflows #1619

Open

Description

@paullizer

Summary

Add a Merge capability to the V2 experience that combines several files into one. It starts with CSV and Excel files that share a structure (a few files interactively in chat, ~100+ files as durable workflow work), then extends to reconciling differing spreadsheet structures, PDFs, Word documents and PowerPoint decks.

User Value

Users regularly receive the same report from many sources (regions, months, teams) and need one combined file. Today SimpleChat cannot do this: native tabular analysis is single-source by contract, the tabular stack has no union/append operation, workflows only publish JSON, and no code concatenates PDFs, Word documents or slide decks. Asking a model to "write a combined CSV" is slow, expensive and lossy for real data.

Proposed Behavior

  • Merge is another capability alongside Analyze and Compare, not a replacement for anything.
    • Reason: tabular_merge (and later document_merge) combines authorized sources deterministically into a retained result. No model rewrites rows; counts are exact and verifiable.
    • Render: the existing render_file creates the downloadable file (CSV, XLSX, later PDF/DOCX/PPTX).
    • Gather (optional): document_search finds the files when the user describes them; Phase 2 adds tabular_inspect to read headers for reconciliation.
  • V2 chat: users just ask ("merge these three CSVs into one Excel file with a source column"). The orchestrator plans Gather → Merge → Render and the user reviews the plan as usual.
  • V2 workflows: a task can use the new Merge document action for large, scheduled or File Sync-triggered merges, with per-file checkpoints and CSV/XLSX outputs. Chat can propose a merge workflow when a request exceeds inline limits, including a Create & run once option for one-off jobs.
  • Admin control: a new merge entry in Document Actions (enabled by default like Analyze/Compare) with chat/workflow document and row limits.
  • Formatting choices: PowerPoint and Word merges support "keep source formatting" (default) or "use the first file's theme/styles", chosen per request.
  • Classic UI is unchanged; there is no Document Explorer merge button.

Acceptance Criteria

  • Phase 0 — Tracking issue and umbrella design doc (docs/explanation/features/V2_FILE_MERGE.md).
  • Phase 1 — Same-structure CSV/Excel merge in V2 chat: tabular_merge (by-name or exact-order schema policies, optional sheet, Source File column), merge document action with admin limits, exact-schema CSV/XLSX export profiles, V2 plan card support, tests and docs.
  • Phase 2 — Reconcile differing structures: tabular_inspect, union and mapped policies, header row, sheet modes, dedupe/sort, honest exclusion of incompatible files, AI-suggested column mapping (tabular_column_mapping_v1) with a review-first flow.
  • Phase 3 — Large and recurring merges (100+ files) in V2 workflows: Merge task with selected/all/recent targets and per-source checkpoints, CSV/XLSX workflow outputs and workspace publication, chat proposal hand-off with Create & run once, V2 workflow editor and workflow assistant support.
  • Phase 4 — Document assembly foundation (document_merge, document_assembly_v1, lineage-pinned Render profiles), PDF merge, and Excel workbook with one tab per source file.
  • Phase 5 — Word (DOCX) merge: keep source styles (default) or use the first document's styles.
  • Phase 6 — PowerPoint (PPTX) merge: keep source formatting (default) or use the first deck's theme, with a section per source deck.
  • Phase 7 — Scale and security hardening (zip bombs, XXE, encrypted/macro files, external relationships, formula injection), user guide, release notes.
  • Every phase ships functional tests, documentation, a version bump and release notes.

Notes

Activity

  1. paullizer commented on Oct 3, 2026

    @paullizer
    ContributorAuthor

    Status: every phase is implemented; the PR is next

    All seven phases are implemented on branch paullizer-multi-file-merge, which isn't pushed yet. It is synced with paullizer-react-v2-ui (f1aeef1, v0.261.233) and renumbered above it, so this work ships as v0.261.234 – v0.261.242.

    Phase Version What shipped
    1 0.261.234 tabular_merge in V2 chat for same-structure CSV/Excel files; the merge document action with chat and workflow file and row limits; exact-schema CSV/XLSX export profiles; plan card wording
    2 0.261.235 tabular_inspect; union, mapped and alias column policies; header rows, every-sheet mode, exclusion, duplicate removal and sorting; AI-prepared column mapping (tabular_column_mapping_v1) that code checks before any row is copied
    3 0.261.236 Workflow Merge files task (selected files, every matching file, recent files, or the files File Sync changed; 100 files and 1,000,000 rows by default) in the V2 editor; chat-proposed merge workflows
    4 0.261.237 Document merge engine; PDF merges (a bookmark per file; scripts and program-launching actions removed) and one workbook with a sheet per file, as workflow merge types
    5 0.261.238 Word merges, keeping each document's styles or using the first document's
    6 0.261.239 PowerPoint merges, keeping each deck's look or the first deck's theme, with a section per deck
    7 0.261.240 document_merge in chat for every kind: the merge step retains a document_assembly_v1 description, and Render (assembled_document_v1) assembles the same files again and delivers the file only if it is byte-identical. Hardening: DDE fields refused, template links removed, PowerPoint program/macro actions removed, XML with a DOCTYPE or a non-UTF-8/16 encoding refused before parsing, ZIP size guards, and no live formulas in CSV/XLSX output
    — 0.261.241 When a request doesn't say, chat asks whether merged spreadsheets go on one sheet or a sheet per file
    — 0.261.242 Merges too large to keep in memory spill into the container's scratch directory (/sc-temp-files), like file renders since 0.261.233

    Validation

    • The merge engine, capability, workflow, hardening, lifecycle and UI suites pass, including 100 files × 1,000 rows merged in one workflow run.
    • After syncing with paullizer-react-v2-ui, a regression sweep of 465 orchestration, workflow, merge, export, docs and admin test files found no new failures. 79 files fail the same way on paullizer-react-v2-ui without this work, and one test that races on a shared cache when shards run in parallel passes on its own. After the final sync (Fix orchestration file generation when /app is root-owned (V2) #1626, Update roadmap status: Phase 7's 7a design, and container-only results and self-authored Run as #1633), the 20 affected suites pass: 930 tests.
    • A code review found six issues, all fixed. The most notable: Word merges whose later documents had lists came out different on every run, so a retried workflow Word merge could attach a second copy. List IDs are now derived.
    • paullizer-react-v2-ui now authorizes a retained result from its conversation and run without rereading its documents. A merged document is the deliberate exception: it is assembled from the originals when it renders, so they are read with the user's current access. Tests and docs cover this.

    Differences from the acceptance criteria

    Phase 3 is left unticked because four items were changed or not built:

    • Per-file checkpoints: none. Merges are deterministic and attach their file idempotently, so a resumed run repeats the merge and reuses the file it already attached.
    • Create & run once: not built. Chat proposes the workflow; it runs when created with Create & start, or through Run workflows from chat.
    • Workflow assistant: the AI workflow assistant doesn't author merge tasks yet.
    • Workspace publication: merged files are attached to the run's conversation. They can be saved to a workspace through the existing promotion of chat files, but nothing is published automatically.

    The Phase 4 assembly foundation (document_merge, document_assembly_v1, lineage-pinned Render profiles) shipped with Phase 7, together with chat document merges.

    Remaining

    • Push the branch and open the PR against paullizer-react-v2-ui. If more PRs land there first, this branch is synced and renumbered again.
    • Validate live with a real planner model and Azure storage, since automated tests use scripted models: chat PDF, Word and PowerPoint merges, the one-sheet-or-sheet-per-file question, and a 100-file workflow merge.
    • Follow-ups if wanted: the Phase 3 items above, and page and slide ranges (the engine supports them; no UI offers them yet). Key-based joins stay out of scope.

    Docs: docs/explanation/features/V2_FILE_MERGE.md (umbrella) and phase docs 1–7, docs/guides/merge-files.md, and release notes v0.261.234–v0.261.242.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P1Priority 1: urgent, address before P2 workenhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions