From d2998cd060788f10cc83d57c5fc0d7fe52d37419 Mon Sep 17 00:00:00 2001 From: David Dizon Date: Thu, 9 Jul 2026 13:51:37 -0700 Subject: [PATCH 1/6] Add Unstructured Transform MCP integration Co-Authored-By: Claude Opus 4.8 (1M context) --- integrations/transform-mcp.md | 115 ++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 integrations/transform-mcp.md diff --git a/integrations/transform-mcp.md b/integrations/transform-mcp.md new file mode 100644 index 00000000..74517ddb --- /dev/null +++ b/integrations/transform-mcp.md @@ -0,0 +1,115 @@ +--- +layout: integration +name: Unstructured Transform MCP +description: "Call Unstructured Transform's document-processing pipeline (partition, enrich, chunk, embed) as MCP tools from a Haystack agent: parse PDFs, spreadsheets, and dozens of file types with tables and layout intact" +authors: + - name: Unstructured + socials: + github: Unstructured-IO +pypi: https://pypi.org/project/mcp-haystack/ +repo: https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mcp +type: Tool Integration +report_issue: https://github.com/deepset-ai/haystack-core-integrations/issues +logo: /logos/unstructured.svg +version: Haystack 2.0 +toc: true +--- +### **Table of Contents** +- [Overview](#overview) +- [Installation](#installation) +- [Usage](#usage) +- [Examples](#examples) +- [License](#license) + +## Overview + +**[Unstructured Transform](https://docs.unstructured.io/transform/overview)** turns any file into agent-ready data, called directly from your agent with no separate pipeline to wire up. It is Unstructured's document-processing pipeline, exposed as a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server at `https://mcp.transform.unstructured.io`. Drop in a PDF, spreadsheet, scan, or email and get back partitioned, enriched, chunked, and embedded output ready for RAG, vector stores, or agent memory, with tables and layout intact. It exposes four tools: + +- `transform_files`: submits one or more files (by URL or a previously returned reference) for processing and returns a `job_id` right away; the job runs asynchronously through configurable stages (`partition` -> `enrich` -> `chunk` -> `embed`) +- `check_transform_status`: polls a job's status until it reaches `COMPLETED` +- `get_transform_results`: fetches a completed job's rendered output as markdown, JSON, HTML, or plain text +- `request_file_upload_url`: returns a presigned upload URL and a durable reference for a local file that isn't already reachable over HTTPS + +This integration doesn't ship its own package. Instead, it uses `mcp-haystack`'s `MCPToolset` to connect any Haystack agent to the Transform MCP server over Streamable HTTP. The free tier includes 15,000 pages a month. + +## Installation + +```bash +pip install mcp-haystack +``` + +## Usage + +Transform MCP supports two ways to authenticate: interactive browser-based OAuth/OIDC (for clients that speak remote MCP natively), and a static API key passed as an `Authorization: Bearer` header (for headless frameworks like Haystack). Get your Unstructured API key from the [Transform get-started page](https://transform.unstructured.io/get-started) after signing in, and pass it through `StreamableHttpServerInfo`'s native `token` parameter: + +```python +from haystack_integrations.tools.mcp import MCPToolset, StreamableHttpServerInfo +from haystack.utils import Secret + +server_info = StreamableHttpServerInfo( + url="https://mcp.transform.unstructured.io", + token=Secret.from_env_var("UNSTRUCTURED_API_KEY"), +) +toolset = MCPToolset(server_info=server_info, eager_connect=True) + +for tool in toolset.tools: + print(f"{tool.name}: {tool.description}") +``` + +If your MCP client doesn't support native remote-MCP OAuth (or only supports local stdio servers), bridge to the hosted server with [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) instead: + +```bash +npm install -g mcp-remote +npx -y mcp-remote https://mcp.transform.unstructured.io +``` + +## Examples + +The snippet below connects the toolset to a Haystack `Agent` and asks it to parse and chunk a PDF end-to-end. Because `transform_files` is asynchronous, the agent's system prompt walks it through the `transform_files` -> `check_transform_status` -> `get_transform_results` polling loop: + +```python +from haystack.components.agents import Agent +from haystack.dataclasses import ChatMessage +from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator +from haystack_integrations.tools.mcp import MCPToolset, StreamableHttpServerInfo +from haystack.utils import Secret + +server_info = StreamableHttpServerInfo( + url="https://mcp.transform.unstructured.io", + token=Secret.from_env_var("UNSTRUCTURED_API_KEY"), +) +toolset = MCPToolset(server_info=server_info, eager_connect=True) + +agent = Agent( + chat_generator=AnthropicChatGenerator(model="claude-opus-4-6"), + tools=toolset, + system_prompt="""You are a document-processing assistant with access to Unstructured Transform MCP tools. + +Transform jobs are asynchronous. When asked to process a document: +1. Call `transform_files` with the file reference(s) and the requested processing stages. This returns a `job_id` immediately; the job itself runs in the background. +2. Call `check_transform_status` with that `job_id`, repeating until the status is COMPLETED. +3. Call `get_transform_results` with the `job_id` to fetch the rendered output, and summarize it for the user. +""", +) + +result = agent.run( + messages=[ + ChatMessage.from_user( + "Parse and chunk the PDF at https://arxiv.org/pdf/1706.03762 using the " + "'hi_res' partition strategy and chunk_by_title with max_characters=1000. " + "Once processing is complete, fetch the results as markdown and show me " + "the first two chunks." + ) + ] +) + +print(result["last_message"].text) +``` + +For a full walkthrough, see the [Document Processing with Unstructured Transform MCP](https://haystack.deepset.ai/cookbook/unstructured_transform_mcp) cookbook. + +## License + +`mcp-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license. + +Unstructured Transform MCP itself is a hosted service provided by Unstructured and is governed by [Unstructured's terms of service](https://unstructured.io/terms-and-conditions), separate from the license of the `mcp-haystack` client used to connect to it. From 63dcc37bea24b593d18ac38180ac0d4904e606db Mon Sep 17 00:00:00 2001 From: David Dizon Date: Tue, 28 Jul 2026 09:46:44 -0700 Subject: [PATCH 2/6] Update tool names for Transform MCP rename (transform_files -> start_transform_job, check_transform_status -> check_job_status, get_transform_results -> get_job_results) Co-Authored-By: Claude Sonnet 5 --- integrations/transform-mcp.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/integrations/transform-mcp.md b/integrations/transform-mcp.md index 74517ddb..71590dbe 100644 --- a/integrations/transform-mcp.md +++ b/integrations/transform-mcp.md @@ -25,9 +25,9 @@ toc: true **[Unstructured Transform](https://docs.unstructured.io/transform/overview)** turns any file into agent-ready data, called directly from your agent with no separate pipeline to wire up. It is Unstructured's document-processing pipeline, exposed as a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server at `https://mcp.transform.unstructured.io`. Drop in a PDF, spreadsheet, scan, or email and get back partitioned, enriched, chunked, and embedded output ready for RAG, vector stores, or agent memory, with tables and layout intact. It exposes four tools: -- `transform_files`: submits one or more files (by URL or a previously returned reference) for processing and returns a `job_id` right away; the job runs asynchronously through configurable stages (`partition` -> `enrich` -> `chunk` -> `embed`) -- `check_transform_status`: polls a job's status until it reaches `COMPLETED` -- `get_transform_results`: fetches a completed job's rendered output as markdown, JSON, HTML, or plain text +- `start_transform_job`: submits one or more files (by URL or a previously returned reference) for processing and returns a `job_id` right away; the job runs asynchronously through configurable stages (`partition` -> `enrich` -> `chunk` -> `embed`) +- `check_job_status`: polls a job's status until it reaches `COMPLETED` +- `get_job_results`: fetches a completed job's rendered output as markdown, JSON, HTML, or plain text - `request_file_upload_url`: returns a presigned upload URL and a durable reference for a local file that isn't already reachable over HTTPS This integration doesn't ship its own package. Instead, it uses `mcp-haystack`'s `MCPToolset` to connect any Haystack agent to the Transform MCP server over Streamable HTTP. The free tier includes 15,000 pages a month. @@ -65,7 +65,7 @@ npx -y mcp-remote https://mcp.transform.unstructured.io ## Examples -The snippet below connects the toolset to a Haystack `Agent` and asks it to parse and chunk a PDF end-to-end. Because `transform_files` is asynchronous, the agent's system prompt walks it through the `transform_files` -> `check_transform_status` -> `get_transform_results` polling loop: +The snippet below connects the toolset to a Haystack `Agent` and asks it to parse and chunk a PDF end-to-end. Because `start_transform_job` is asynchronous, the agent's system prompt walks it through the `start_transform_job` -> `check_job_status` -> `get_job_results` polling loop: ```python from haystack.components.agents import Agent @@ -86,9 +86,9 @@ agent = Agent( system_prompt="""You are a document-processing assistant with access to Unstructured Transform MCP tools. Transform jobs are asynchronous. When asked to process a document: -1. Call `transform_files` with the file reference(s) and the requested processing stages. This returns a `job_id` immediately; the job itself runs in the background. -2. Call `check_transform_status` with that `job_id`, repeating until the status is COMPLETED. -3. Call `get_transform_results` with the `job_id` to fetch the rendered output, and summarize it for the user. +1. Call `start_transform_job` with the file reference(s) and the requested processing stages. This returns a `job_id` immediately; the job itself runs in the background. +2. Call `check_job_status` with that `job_id`, repeating until the status is COMPLETED. +3. Call `get_job_results` with the `job_id` to fetch the rendered output, and summarize it for the user. """, ) From e8136a92d9559a6a855b35cbc6f8f4c01fc4b1dd Mon Sep 17 00:00:00 2001 From: David Dizon Date: Tue, 28 Jul 2026 09:56:35 -0700 Subject: [PATCH 3/6] Show how to set UNSTRUCTURED_API_KEY per Copilot review feedback Co-Authored-By: Claude Sonnet 5 --- integrations/transform-mcp.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/integrations/transform-mcp.md b/integrations/transform-mcp.md index 71590dbe..9ac57f36 100644 --- a/integrations/transform-mcp.md +++ b/integrations/transform-mcp.md @@ -45,6 +45,9 @@ Transform MCP supports two ways to authenticate: interactive browser-based OAuth ```python from haystack_integrations.tools.mcp import MCPToolset, StreamableHttpServerInfo from haystack.utils import Secret +import os + +os.environ["UNSTRUCTURED_API_KEY"] = "YOUR_UNSTRUCTURED_API_KEY" server_info = StreamableHttpServerInfo( url="https://mcp.transform.unstructured.io", @@ -73,6 +76,9 @@ from haystack.dataclasses import ChatMessage from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator from haystack_integrations.tools.mcp import MCPToolset, StreamableHttpServerInfo from haystack.utils import Secret +import os + +os.environ["UNSTRUCTURED_API_KEY"] = "YOUR_UNSTRUCTURED_API_KEY" server_info = StreamableHttpServerInfo( url="https://mcp.transform.unstructured.io", From f7425540a5bf3039ba90314b8d611f6a9775680c Mon Sep 17 00:00:00 2001 From: David Dizon Date: Mon, 10 Aug 2026 09:29:02 -0700 Subject: [PATCH 4/6] Address review feedback: drop unnecessary mcp-remote mention, add wait between status polls Co-Authored-By: Claude Sonnet 5 --- integrations/transform-mcp.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/integrations/transform-mcp.md b/integrations/transform-mcp.md index 9ac57f36..33fd17dd 100644 --- a/integrations/transform-mcp.md +++ b/integrations/transform-mcp.md @@ -59,13 +59,6 @@ for tool in toolset.tools: print(f"{tool.name}: {tool.description}") ``` -If your MCP client doesn't support native remote-MCP OAuth (or only supports local stdio servers), bridge to the hosted server with [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) instead: - -```bash -npm install -g mcp-remote -npx -y mcp-remote https://mcp.transform.unstructured.io -``` - ## Examples The snippet below connects the toolset to a Haystack `Agent` and asks it to parse and chunk a PDF end-to-end. Because `start_transform_job` is asynchronous, the agent's system prompt walks it through the `start_transform_job` -> `check_job_status` -> `get_job_results` polling loop: @@ -93,7 +86,7 @@ agent = Agent( Transform jobs are asynchronous. When asked to process a document: 1. Call `start_transform_job` with the file reference(s) and the requested processing stages. This returns a `job_id` immediately; the job itself runs in the background. -2. Call `check_job_status` with that `job_id`, repeating until the status is COMPLETED. +2. Call `check_job_status` with that `job_id`, waiting a few seconds between calls, repeating until the status is COMPLETED. 3. Call `get_job_results` with the `job_id` to fetch the rendered output, and summarize it for the user. """, ) From d4dc75e96864d2ba0667a27299fed04d57bbc50f Mon Sep 17 00:00:00 2001 From: David Dizon Date: Mon, 10 Aug 2026 10:51:12 -0700 Subject: [PATCH 5/6] Correct tool count: server exposes 7 tools total, doc covers the 4 core ones Co-Authored-By: Claude Sonnet 5 --- integrations/transform-mcp.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/integrations/transform-mcp.md b/integrations/transform-mcp.md index 33fd17dd..99ae73b1 100644 --- a/integrations/transform-mcp.md +++ b/integrations/transform-mcp.md @@ -23,13 +23,15 @@ toc: true ## Overview -**[Unstructured Transform](https://docs.unstructured.io/transform/overview)** turns any file into agent-ready data, called directly from your agent with no separate pipeline to wire up. It is Unstructured's document-processing pipeline, exposed as a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server at `https://mcp.transform.unstructured.io`. Drop in a PDF, spreadsheet, scan, or email and get back partitioned, enriched, chunked, and embedded output ready for RAG, vector stores, or agent memory, with tables and layout intact. It exposes four tools: +**[Unstructured Transform](https://docs.unstructured.io/transform/overview)** turns any file into agent-ready data, called directly from your agent with no separate pipeline to wire up. It is Unstructured's document-processing pipeline, exposed as a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server at `https://mcp.transform.unstructured.io`. Drop in a PDF, spreadsheet, scan, or email and get back partitioned, enriched, chunked, and embedded output ready for RAG, vector stores, or agent memory, with tables and layout intact. The server exposes seven tools; the four below cover the document-processing pipeline this integration focuses on: - `start_transform_job`: submits one or more files (by URL or a previously returned reference) for processing and returns a `job_id` right away; the job runs asynchronously through configurable stages (`partition` -> `enrich` -> `chunk` -> `embed`) - `check_job_status`: polls a job's status until it reaches `COMPLETED` - `get_job_results`: fetches a completed job's rendered output as markdown, JSON, HTML, or plain text - `request_file_upload_url`: returns a presigned upload URL and a durable reference for a local file that isn't already reachable over HTTPS +The remaining three (`start_extraction_job`, `suggest_extraction_schema_for_file`, `get_instructions`) support schema-based structured data extraction and on-demand server guidance, outside the scope of this integration page. + This integration doesn't ship its own package. Instead, it uses `mcp-haystack`'s `MCPToolset` to connect any Haystack agent to the Transform MCP server over Streamable HTTP. The free tier includes 15,000 pages a month. ## Installation From 9604f799cd505c5967d880dd5258c9803b80388f Mon Sep 17 00:00:00 2001 From: David Dizon Date: Tue, 11 Aug 2026 17:36:18 -0700 Subject: [PATCH 6/6] Stop hardcoding tool names: describe the pipeline by behavior, discover tools live Co-Authored-By: Claude Sonnet 5 --- integrations/transform-mcp.md | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/integrations/transform-mcp.md b/integrations/transform-mcp.md index 99ae73b1..24bc7bb6 100644 --- a/integrations/transform-mcp.md +++ b/integrations/transform-mcp.md @@ -23,14 +23,9 @@ toc: true ## Overview -**[Unstructured Transform](https://docs.unstructured.io/transform/overview)** turns any file into agent-ready data, called directly from your agent with no separate pipeline to wire up. It is Unstructured's document-processing pipeline, exposed as a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server at `https://mcp.transform.unstructured.io`. Drop in a PDF, spreadsheet, scan, or email and get back partitioned, enriched, chunked, and embedded output ready for RAG, vector stores, or agent memory, with tables and layout intact. The server exposes seven tools; the four below cover the document-processing pipeline this integration focuses on: +**[Unstructured Transform](https://docs.unstructured.io/transform/overview)** turns any file into agent-ready data, called directly from your agent with no separate pipeline to wire up. It is Unstructured's document-processing pipeline, exposed as a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server at `https://mcp.transform.unstructured.io`. Drop in a PDF, spreadsheet, scan, or email and get back partitioned, enriched, chunked, and embedded output ready for RAG, vector stores, or agent memory, with tables and layout intact. -- `start_transform_job`: submits one or more files (by URL or a previously returned reference) for processing and returns a `job_id` right away; the job runs asynchronously through configurable stages (`partition` -> `enrich` -> `chunk` -> `embed`) -- `check_job_status`: polls a job's status until it reaches `COMPLETED` -- `get_job_results`: fetches a completed job's rendered output as markdown, JSON, HTML, or plain text -- `request_file_upload_url`: returns a presigned upload URL and a durable reference for a local file that isn't already reachable over HTTPS - -The remaining three (`start_extraction_job`, `suggest_extraction_schema_for_file`, `get_instructions`) support schema-based structured data extraction and on-demand server guidance, outside the scope of this integration page. +The pipeline itself runs asynchronously as a job: submit a file for processing, poll until it's done, then fetch the rendered result; a separate helper mints an upload URL for files that aren't already reachable over HTTPS. Unstructured adds tools and capabilities to this server as they ship new features, so rather than list exact tool names and a fixed count here (which would go stale the next time they do), the snippets below discover the live toolset at connect time and let the agent match tools to the task by their description. This integration doesn't ship its own package. Instead, it uses `mcp-haystack`'s `MCPToolset` to connect any Haystack agent to the Transform MCP server over Streamable HTTP. The free tier includes 15,000 pages a month. @@ -63,7 +58,7 @@ for tool in toolset.tools: ## Examples -The snippet below connects the toolset to a Haystack `Agent` and asks it to parse and chunk a PDF end-to-end. Because `start_transform_job` is asynchronous, the agent's system prompt walks it through the `start_transform_job` -> `check_job_status` -> `get_job_results` polling loop: +The snippet below connects the toolset to a Haystack `Agent` and asks it to parse and chunk a PDF end-to-end. Because job submission is asynchronous, the agent's system prompt walks it through a submit -> poll -> fetch flow, described by behavior rather than by hardcoded tool name so it keeps working as Unstructured renames or adds tools: ```python from haystack.components.agents import Agent @@ -84,12 +79,12 @@ toolset = MCPToolset(server_info=server_info, eager_connect=True) agent = Agent( chat_generator=AnthropicChatGenerator(model="claude-opus-4-6"), tools=toolset, - system_prompt="""You are a document-processing assistant with access to Unstructured Transform MCP tools. + system_prompt="""You are a document-processing assistant with access to Unstructured Transform MCP tools. Check the tools available to you and use whichever ones match the steps below by description, since exact tool names may change over time. Transform jobs are asynchronous. When asked to process a document: -1. Call `start_transform_job` with the file reference(s) and the requested processing stages. This returns a `job_id` immediately; the job itself runs in the background. -2. Call `check_job_status` with that `job_id`, waiting a few seconds between calls, repeating until the status is COMPLETED. -3. Call `get_job_results` with the `job_id` to fetch the rendered output, and summarize it for the user. +1. Submit the file reference(s) and the requested processing stages to start a processing job. This returns a job ID immediately; the job itself runs in the background. +2. Check the job's status, waiting a few seconds between checks, until it reports as complete. +3. Fetch the job's rendered output using its job ID, and summarize it for the user. """, )