[Generated exports] Support universal CSV, JSON, and XML file generation

Summary

Provide one universal CSV-generation capability that works across Chat Search, ordinary Chat, streaming responses, selected agents, Analyze, Compare, workflows, and source-free prompts. CSV generation must not depend on whether the input is a PDF, Office document, text, image/media-derived evidence, CSV/XLSX, mixed selected files, a single file, many files within configured limits, or no workspace files.

User Value

Users need a reliable downloadable CSV whenever they ask for structured CSV output. The current behavior varies by route and source type, so equivalent requests can return inline text, a partial relevance-bounded result, or no artifact at all.

Current Failure

Select multiple workspace documents, for example 16 PDFs.
Leave the chat document action set to Search.
Ask: turn these into a single csv.
The response can say it contains one row per retrieved document and render inline CSV, but it does not reliably cover every selected file or attach a downloadable CSV artifact.

The shared CSV-intent predicate currently accepts turn these into a csv but rejects turn these into a single csv, so the exact request above does not invoke the artifact publisher.

Proposed Behavior

Treat CSV as an output capability shared by all response producers, not as a tabular-file-only or Analyze-only feature.
Recognize natural CSV requests including single CSV, combined CSV, one CSV, download/export/save CSV, and equivalent phrasing.
For explicit selected sources, preserve the path's evidence semantics while making the requested CSV coverage explicit. When the request requires all selected sources, do not silently export only relevance hits.
When a requested row shape or schema is ambiguous, ask one durable clarification rather than guessing whether rows represent files, documents, or extracted records.
For larger requests, use a durable background job and publish the authorized completed artifact to the originating chat. Small results may publish immediately.
For source-free requests, convert validated model-produced structured rows into the same downloadable CSV artifact contract.
Produce a valid downloadable CSV or an explicit failure/partial-coverage result; never fabricate unprocessed rows.

Acceptance Criteria

A request such as turn these into a single csv creates a downloadable CSV when the response contains valid structured rows.
CSV intent works for Chat Search, ordinary and streaming Chat, selected agents, Analyze, Compare, workflows, and source-free prompts.
The capability works with every supported queryable input format through its native evidence adapter, including PDF, Office/text, image/media-derived evidence, CSV/XLSX, and mixed selections.
Explicit selected-source requests disclose and enforce the requested coverage instead of silently reducing to relevance hits.
Ambiguous row granularity/schema produces one resumable clarification; explicit row/schema instructions bypass it.
Large work is durable, cancellable, reauthorizes source access and revision before publication, and exposes the completed download without duplicating artifacts.
CSV output preserves current formula-injection protection, safe headers, quoted/multiline values, and authorized download controls.
Partial, failed, unsupported, unresolved, canceled, and source-version-changed inputs are surfaced with per-source coverage and do not produce misleading complete results.
Functional coverage validates no-source, one source, multi-source, mixed formats, all response paths, exact single CSV phrasing, clarification, background completion, cancellation, authorization loss, and downloads.

Notes

This is a distinct output-capability issue, not a subtask of mixed-source orchestration issue [Mixed-source orchestration] Phase 6: Harden and roll out native mixed-source processing #1061.
Existing artifacts and downloads are already generic enough to reuse; the main remediation is shared intent/finalization and source-coverage coordination.
Focused existing CSV serializer tests passed 23/23. The current lossless-analysis artifact test needs harness maintenance because its AST loader does not inject the new cancellation helper and it has an outdated version pin.

Follow-up Architecture Decision

The durable execution and artifact-publication portion of this issue is intended to establish reusable, format-neutral primitives for generated deliverables:

persisted job lifecycle: queued, running, completed, failed, and canceled
ownership, conversation linkage, authorization and source-revision revalidation
leases, heartbeats, checkpoints, retry/resume, cancellation, and idempotent artifact publication
bounded progress/status metadata and authorized artifact download/promotion

CSV remains the first format-specific capability on this issue. The implementation should not turn the existing tabular CSV/JSON executor into a generic Office renderer. Instead, it should extract or define the neutral job/artifact contract so format adapters can own their own planning and rendering.

Follow-on consumers of this foundation:

[Generated documents] Add durable long-form Word document generation #1072: durable long-form Word document generation, with a DOCX outline/section/render adapter
[Generated presentations] Add durable long-form PowerPoint generation #1073: durable long-form PowerPoint generation, with a deck/slide/layout/render adapter

Template population and editing remain separate concerns under #1045, #968, #969, and #999.

Performance And Operability Requirements

Observed failure mode

A background CSV export can remain visibly Running at the same percentage/batch for a long period with no reliable indication whether it is actively making progress, waiting for capacity, retrying, stalled, or permanently hung. A progress card that continues to show an old ETA is not sufficient evidence of healthy execution.

Treat this as a first-class product requirement for universal generated artifacts, not only a CSV concern. The same job contract should serve the long-form DOCX and PPTX work tracked by #1072 and #1073.

Required performance model

Define workload cohorts and versioned performance budgets for small, medium, and large generation requests. Budgets must cover queue delay, time to first durable progress checkpoint, end-to-end completion latency, and artifact publication latency; use measured p50/p95 values rather than one unqualified global timeout.
Record wall-clock elapsed time separately from active processing time and queue/wait time.
Record checkpoint throughput: completed units per minute, rows/batches for CSV, and the equivalent planned/rendered units for future artifact adapters.
Record per-phase timing: authorization/manifest resolution, input staging, planning, each model/tool request, checkpoint persistence, validation, final rendering/serialization, blob upload, and artifact publication.
Record retry count, retry reason/category, scheduler delay, lease claim/reclaim count, queue depth/concurrency pressure where available, and ETA accuracy (estimated versus actual completion).
Emit only bounded aggregate telemetry and opaque correlation/run identifiers. Never log source contents, prompts, raw output rows, document text, filenames beyond approved aggregate labels, secrets, or user-visible data solely for performance analysis.

Forward-progress and recovery requirements

A heartbeat alone must not keep a run healthy. Detect stalling from the last durably committed forward-progress checkpoint such as completed batch/row/unit, not only last_heartbeat_at or a Running status.
Persist both last_heartbeat_at and last_forward_progress_at, plus the last completed unit and phase. Do not advance the displayed percentage until the corresponding checkpoint is durable.
When no forward progress occurs within a configurable workload-aware threshold, transition the public status to stalled or an equivalent actionable state, retain the completed checkpoint, and record the diagnostic reason.
Attempt bounded automatic recovery/reclaim when safe. After the configured retry/reclaim budget is exhausted, surface an explicit recoverable failure or blocked state rather than leaving the run indefinitely Running.
Reauthorization, revision checks, cancellation, lease ownership, and idempotent final artifact publication must still hold through automatic recovery and manual resume.

User-visible progress requirements

Show the current phase, completed versus total durable units, queue time, active elapsed time, last forward-progress timestamp, retry/recovery state, and ETA confidence rather than presenting an obsolete ETA as current.
Mark stale estimates as unavailable or stale. A run that has not crossed a durable checkpoint inside its threshold must not continue to present a normal Running badge without an explanation.
Provide accessible, actionable controls appropriate to state: Cancel while cancelable; Resume/Retry when recovery is safe; and a clear terminal error when it is not.
Keep status polling resilient to refresh, reconnect, worker restart, and authorization loss; polling must not make a stale job appear healthy.

Validation and release gates

Add deterministic tests for a slow-but-progressing run, a hung model/tool call with no checkpoint, stale heartbeat, stale forward progress despite a misleading worker state, lease loss, scheduler recovery, cancellation during a batch, retry/reclaim exhaustion, and idempotent final publication.
Add a representative large-export regression scenario, including the current style of multi-batch workload, to verify that progress, ETA, recovery, and terminal status remain correct under delay and restart conditions.
Add benchmark/soak coverage for each workload cohort. Track p50/p95 queue delay, time to first checkpoint, throughput, completion latency, retry rate, stale-run rate, recovery success rate, and artifact-publication latency as release/rollout metrics.
Do not default-enable a new generation path until its benchmark baseline and stall/recovery behavior meet the documented performance budgets.

Architecture boundary

Extract these metrics, lifecycle, checkpoint, recovery, and publication contracts into the format-neutral generated-artifact job foundation described in #1071. CSV, DOCX (#1072), and PowerPoint (#1073) should report comparable performance metrics while retaining format-specific planning and rendering adapters.

Implementation Update: Background CSV Export Throughput and Timeout Hardening

Implemented in application version 0.250.070.

What changed

Increased default post-schema background batch concurrency from 2 to 3 while preserving the existing maximum of 5.
Kept the first schema-discovery batch serial so later parallel batches still use one validated output schema.
Added a configurable per-batch model-call timeout with a default of 300 seconds.
Caps that timeout below the stale-worker threshold, so a genuinely stuck model call becomes a retryable timeout before stale-worker recovery can create competing execution.
Existing durable checkpoints, ordered output, cancellation, authorization/revision checks, retries, and idempotent artifact publication remain in place.

Expected impact

A six-batch export now completes its post-schema model work in two three-batch windows instead of up to three two-batch windows. A batch that hangs no longer occupies the background worker indefinitely; it is retried from its last durable checkpoint.

Validation

functional_tests/test_tabular_background_generated_exports.py: 7/7 passed.
Added a deterministic asynchronous stalled-model-call test that verifies timeout behavior.
py_compile passed for functions_tabular_generated_exports.py.
Editor diagnostics and scoped whitespace checks passed.

Remaining issue scope

This is a focused throughput and bounded-wait improvement. The broader work already defined in this issue remains: durable forward-progress detection distinct from heartbeat, workload performance budgets, benchmark/soak gates, ETA confidence, queue-pressure telemetry, and extraction of a format-neutral generated-artifact job foundation for CSV, DOCX, and PPTX.

Release notes and fix documentation were updated in the same version.

Related background-export resilience work is included in PR #1074: #1074

The PR increases safe post-schema batch concurrency from two to three and adds bounded model-call timeout recovery using the existing durable checkpoint path. It preserves serial schema discovery, authorization, ordered output, and idempotent artifact publication.

This is a related export reliability increment only; it does not claim to complete the broader universal CSV-generation scope in this issue.

Implementation Update: Universal CSV Generation

Implemented in application version 0.250.071.

PR: #1075

What changed

Added shared CSV intent recognition for natural requests such as turn these into a single CSV, one CSV, and combined CSV.
Finalized valid structured rows through the same downloadable CSV artifact contract across standard and streaming Chat, Chat Search, selected agents, Analyze, Compare, workflows, and source-free prompts.
Added one conversation-persisted row/schema clarification for ambiguous CSV requests. Explicit row or column instructions bypass it.
Reused the existing owner-authorized immediate upload and durable background queue, including cancellation, reauthorization, source-version checks, checkpointing, and idempotent publication.
Preserved formula-injection protection, safe headers, quoted/multiline values, and duplicate-artifact suppression.
Rebased the work onto PR Harden mixed-source orchestration and CSV exports #1074's timeout and throughput hardening so universal CSV exports retain bounded model-call recovery.

Validation

functional_tests/test_assistant_table_csv_artifact.py: 27/27 passed
functional_tests/test_mixed_source_analyze_workflow.py: passed
functional_tests/test_document_action_token_usage_aggregation.py: 5/5 passed
functional_tests/test_document_action_conversation_scope_metadata.py: 4/4 passed
functional_tests/test_mixed_source_conversation_continuity.py: 3/3 passed
functional_tests/test_tabular_row_orchestration_scale.py: passed
functional_tests/test_tabular_background_generated_exports.py: 7/7 passed
Analyze/Compare artifact plumbing tests: 2/2 each passed
Python compilation, editor diagnostics, and whitespace checks passed.

Remaining Scope

This completes the shared CSV intent/finalization and current durable-export integration increment. The broader format-neutral generated-artifact job foundation, forward-progress stall detection, workload performance budgets, and benchmark/soak gates described in this issue remain follow-up work, so the issue remains open.

Add universal CSV generation and mixed-source Analyze

Add generic generated file export framework

Follow-up Implementation Update: Generated File Export Framework

Implemented in application version 0.250.073.

PR: #1076

What changed

Extracted a format-neutral generated-file framework that owns output intent, structured function-result normalization, renderer dispatch, and artifact metadata.
CSV, DOCX, and PDF are now separate renderer capabilities on the same authorized chat-artifact publication path.
Standard/streaming Chat, agents/actions, Analyze, Compare, workflows, and source-free responses now supply their current-turn function citations to the framework.
Successful structured function results can produce a requested artifact even when the assistant summarizes rather than reproduces every row.
Function-result values are defensively filtered for sensitive fields; combined action rows retain source-action provenance.
Tabular-plugin rows remain on the existing coverage-aware, revision-aware CSV/XLSX export path and are not bypassed by the generic framework.
CSV retains the checkpointed durable queue for large output; DOCX/PDF render immediately for response-sized content.
Increased bounded authorized retrieval capacity for search, metadata, directory, assigned knowledge, workflows, and mixed-source tabular candidates.

Validation

functional_tests/test_assistant_table_csv_artifact.py: 31/31 passed, including CSV/DOCX/PDF rendering from structured function results and workflow DOCX publication.
functional_tests/test_mixed_source_manifest_contracts.py: 13/13 passed.
functional_tests/test_document_action_token_usage_aggregation.py: 5/5 passed.
Python compilation, editor diagnostics, and whitespace checks passed.

Scope Boundary

This establishes the shared source/planning/publication contract requested for generated artifacts. Durable long-form DOCX and PPTX planning, checkpoints, retries, and performance budgets remain separately tracked by #1072 and #1073.

Expanding this issue to cover the JSON/XML export failures observed in chat and Analyze flows.

Additional scope:

JSON and XML should use the same generated file/export artifact framework as CSV instead of being returned as large inline chat text.
JSON/XML requests from Chat Search, ordinary Chat, Analyze, Compare, workflows, selected agents, and source-free prompts should create downloadable chat artifacts when the user asks for a file-shaped output.
Large JSON/XML outputs should use durable background chunking/checkpointing where applicable, with status/progress metadata and final artifact publication only after validation.
XML template-population workflows should produce a complete downloadable .xml file, not an MD analysis artifact or a partial explanatory response.
XML document ingestion and analysis should be reliable enough for uploaded XML templates to participate in analysis/conversion workflows.

Additional acceptance criteria:

convert this entire XML into JSON creates a downloadable .json artifact and avoids dumping the full JSON into the assistant bubble.
extract content from this PDF and populate this XML template creates a downloadable .xml artifact or clearly reports why generation could not complete.
JSON/XML output requests share the existing generated artifact card/download route and large-output progress behavior used by CSV.
Partial evidence, truncation, unsupported source coverage, malformed model output, and source-version changes are explicit and do not produce misleading complete files.
Functional tests cover JSON artifacts, XML artifacts, XML upload processing, background metadata, and no-inline-output handoff behavior.

Implementation update from paullizer-export-file-framework-json-xml:

Added shared generated-file helpers for JSON extraction, XML extraction, XML serialization, and output-format normalization.
Wired JSON/XML file artifact capture into document-action, non-streaming chat, and streaming chat save paths.
Replaced persisted inline JSON/XML assistant content with a concise downloadable-file handoff when a valid artifact is saved.
Added XML artifact support to document-analysis generated artifacts and prompt guidance for JSON/XML-only final output.
Extended durable tabular generated exports to serialize checkpointed output as XML as well as CSV/JSON.
Consolidated XML document processing through one token-aware implementation.
Bumped app version to 0.250.110 and added fix documentation.

Validation:

python -m py_compile application\single_app\functions_generated_file_exports.py application\single_app\functions_tabular_generated_exports.py application\single_app\functions_workflow_runner.py application\single_app\functions_document_analysis.py application\single_app\functions_documents.py application\single_app\route_backend_chats.py functional_tests\test_generated_json_xml_exports.py
python functional_tests\test_generated_json_xml_exports.py
python functional_tests\test_tabular_background_generated_exports.py
git --no-pager diff --check

Update release notes for JSON XML exports

Harden generated XML export parsing

Harden mixed-source orchestration and CSV exports

Support generated JSON and XML exports

Add consolidated release note for generated file output work (Refs mi…

Status update: v0.260.004 through v0.260.011

Recent work delivered a meaningful part of this capability for the selected agents path in ordinary and streaming Chat, where the artifact is built from authorized action results. Posting a map of what is now covered and what this issue still needs, so the remaining scope stays clear.

Delivered

The artifact no longer fails or silently degrades

Multi-action turns returning different column sets stopped with Generated output schema mismatch at row 2. The export now pins a union of every column before the run starts and pads the missing cells.
A pasted sample of rows in the assistant reply used to outrank the retrieved dataset, producing a 3-row file from a 900-row query. Pasted rows are now used only when they are not an excerpt of the real result set.
Discovery calls in the same turn (list_instances, list_parameters) were blended into the dataset. Rows are grouped by originating action, and the action holding the substantive dataset wins.
Overlapping result pages are now de-duplicated. Agents frequently re-request a range from the same start time rather than paging forward, which produced a 1,000-row file for a window holding roughly 500 distinct records.

Ambiguous row shape produces one resumable clarification (acceptance criterion 5)

A turn that asks "should each row represent a telemetry sample, and which columns?" no longer publishes a placeholder file built from the question text.
Answering it with "yes, all columns" now publishes the artifact that was originally requested. The pending format is carried forward for both publication and model guidance.
Explicit row/schema instructions still bypass the clarification.

Follow-up turns reuse evidence already gathered

"Now make that a CSV" after the data was retrieved in an earlier turn no longer returns empty. The export reaches back through stored conversation citations, bounded by the configured conversation history limit, instead of re-querying the source.

Partial coverage is disclosed (partial credit toward acceptance criterion 8)

When an action reports that it truncated its own results, the artifact carries a Partial badge and a note stating it covers only the rows the action returned.
Models are instructed to request the remainder starting after the last row they hold, rather than repeating the original range.

The model no longer denies the capability

Every format now states the publication contract, including on the turn that only answers a clarification. This removes replies like "I cannot create or attach a CSV file in this interface" that were then followed by the file appearing anyway.

Format parity

CSV, DOCX, PDF, JSON, and XML resolve rows identically, reach back to earlier turns, decline to publish on a clarification turn, and report truncation.
The assistant's written answer is preserved for CSV, DOCX, and PDF. Only JSON and XML withhold their payload from the response, which is what the artifact contract requires.

Verified against the original repro

The shared intent predicate now accepts the phrasing called out in the description:

'turn these into a csv'           -> csv
'turn these into a single csv'    -> csv
'turn these into a combined csv'  -> csv
'make one csv'                    -> csv

That closes the intent half of acceptance criterion 1. The end-to-end multi-document coverage half of that criterion is not verified here, because the work above exercised action-result sources rather than a 16-PDF selection.

Still open on this issue

CSV intent across Analyze, Compare, and workflows. The work above covers ordinary and streaming Chat with selected agents.
Native evidence adapters for every queryable input format, including image/media-derived evidence and mixed selections.
Explicit selected-source requests disclosing and enforcing requested coverage instead of silently reducing to relevance hits.
Per-source coverage reporting for partial, failed, unsupported, unresolved, canceled, and source-version-changed inputs. Only action-reported truncation is surfaced today, and it is whole-artifact rather than per-source.
Functional coverage for the no-source, multi-source, and mixed-format matrix described in the last acceptance criterion.

Related PRs

PR	Scope
#1302	Union passthrough CSV schema, keep assistant text visible
#1308	Build CSV artifacts from authorized action rows
#1312	Give every generated file format the same row resolution
#1317	Paging de-duplication, truncation disclosure, guidance carry-forward

Fix documentation: docs/explanation/fixes/GENERATED_ARTIFACT_PAGING_AND_GUIDANCE_FIX.md
Release notes: consolidated entry under v0.260.011.

[Generated exports] Support universal CSV, JSON, and XML file generation #1071

Description

Summary

User Value

Current Failure

Proposed Behavior

Acceptance Criteria

Notes

Activity

paullizer commented on Jul 23, 2026

Follow-up Architecture Decision

paullizer commented on Jul 23, 2026

Performance And Operability Requirements

Observed failure mode

Required performance model

Forward-progress and recovery requirements

User-visible progress requirements

Validation and release gates

Architecture boundary

paullizer commented on Jul 23, 2026

Implementation Update: Background CSV Export Throughput and Timeout Hardening

What changed

Expected impact

Validation

Remaining issue scope

paullizer commented on Jul 23, 2026

paullizer commented on Jul 23, 2026

Implementation Update: Universal CSV Generation

What changed

Validation

Remaining Scope

paullizer commented on Jul 23, 2026

Follow-up Implementation Update: Generated File Export Framework

What changed

Validation

Scope Boundary

paullizer commented on Aug 4, 2026

paullizer commented on Aug 4, 2026

paullizer commented on Aug 19, 2026

Status update: v0.260.004 through v0.260.011

Delivered

Verified against the original repro

Still open on this issue

Related PRs

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions