From 24b3ba3a59ee4c0a26a1d4f45dd9fd76cabecf76 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Fri, 14 Aug 2026 20:11:26 +0200 Subject: [PATCH 1/4] docs: extend AGENTS.md with conventions mined from PR reviews Our AGENTS.md documents the mechanics -- hatch per integration, tests, mypy, versioning -- but none of the judgement reviewers actually apply across 100+ integration packages. That knowledge lived only in review comments. This adds 130 rules mined with pydantic/braindump from 2,772 review comments written by the deepset team between 2025-07-01 and today, clustered and deduplicated across PRs so that only repeatedly-enforced conventions survive. Rules that recur across integrations (serialisation symmetry, warm_up lifecycle, Secret handling, filter policy, async parity) generalise; per-integration one-offs do not and were dropped. Because the corpus mostly predates Haystack 3.0, rules were filtered against the current tree first. This repo removed almost nothing of its own -- it gained the ~45 component classes that moved out of core -- so the staleness check here is inherited from core: guidance about APIs that left haystack but landed here stays valid, while guidance about APIs that disappeared entirely does not. Each marker traces back to its source review comments. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 316 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 316 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 67b954b90f..ce08941a11 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,3 +77,319 @@ Changelogs are auto-generated per integration and not meant to be edited manuall ## Creating a New Integration Follow the instructions in the "Create a new integration" section of `CONTRIBUTING.md`. + + + +The rules below were mined from 2,772 PR review comments written by the deepset +team between 2025-07-01 and 2026-08-14, then filtered against the current source tree so that +guidance referring to APIs removed or moved in Haystack 3.0 does not survive. Each +`` marker traces back to the review comments it came from. + +They describe what reviewers actually enforce. Follow them the way you would follow a +reviewer's note: they encode reasons, not ceremony, so when a rule genuinely does not +fit the change at hand, say why rather than contorting the code to satisfy it. + +## API Design + + +- Target Haystack 3.x APIs: use `ChatGenerator`s, create clients in `warm_up()`, and support the module allowlist — Keeps integrations compatible with Haystack 3.x lifecycle, chat APIs, and safe deserialization behavior. + +- Keep `to_dict()`/`from_dict()` symmetric with `__init__` — preserve all runtime config, including Watsonx `max_retries`, `meta_fields_to_embed`, and `embedding_separator`. + +- Keep Haystack `__init__` light; route reusable setup through `warm_up()` — prevents slow construction and duplicated lifecycle logic, especially in `integrations/mcp/src/haystack_integrations/tools/mcp/` + +- Preserve protocol parameter order; append store-specific args and pass ambiguous optionals by keyword — Maintains protocol compatibility while allowing implementations to add store-specific options without breaking callers. + +- Align public API names across signatures, docs, code, and returns — document intentional mismatches + +- Make required `run(...)` inputs explicit keyword-only args — prevents no-op component runs + +- Use `StreamingChunk.reasoning`/`ReasoningContent` for reasoning — keep `meta` incidental + +- Resolve `Secret` values in `warm_up`, not `__init__` — avoids init-time side effects + +- Use Haystack serialization shape with runtime `type` — e.g. `{"type": generate_qualified_class_name(type(self)), "data": ...}` + +- Align `Document` embedder constructors in `haystack/components/embedders` and `integrations/*/src/haystack_integrations/components/embedders/*` — include provider-relevant options like `meta_fields_to_embed`, `embedding_separator`, `prefix`, `suffix`, and `batch_size` when comparable embedders expose them + +- Make `warm_up()` idempotent with an `__init__` flag set only after setup succeeds — This prevents repeated expensive setup and avoids partially marking failed initialization as ready. + +- Type chat generator `tools` as `ToolsType`; don’t narrow provider-native params — preserves Haystack API compatibility and tool pass-through + +## Documentation + + +- Keep comments concise and substantive — explain non-obvious intent, limits, edge cases, workarounds, or real compat needs + +- Update `pydoc/config_docusaurus.yml` for public API changes — include modules, retrievers, and errors + +- Keep docstrings authoritative and preserve reference links — prevents stale or duplicated docs + +- Add the standard `SPDX-FileCopyrightText` and `Apache-2.0` header before imports — Ensures repository-wide license compliance and keeps source and test files consistent across core and integration packages. + +- Keep public examples current with supported APIs — update docstrings, cookbooks, integrations, Google GenAI model names, and `RagasEvaluator` `ragas.metrics.collections` usage + +## Config + + +- Source workflow creds from matching `${{ secrets. }}` env vars — keeps CI secure and reliable + +- Align workflow `python-version` matrices to min/max supported Python — catches boundary regressions + +- Default secrets from provider env vars; use `WATSONX_API_KEY` for Watsonx components — Predictable env-var defaults make credentials easy to configure across integrations while avoiding hardcoded secrets or setup surprises. + +- Use SDK env var names consistently across CI, tests, secrets, and skips — prevents config drift + +- Store credential fields as Haystack `Secret`, not `str` — avoids leaking sensitive config + +## Testing + + +- Gate `integrations//tests/` explicitly — wire CI coverage, inject `Secret`/env values, use provider-specific `pytest.mark.skipif(...)` reasons, precise `sys.version_info`/`sys.platform` checks, and validate external-API tests locally with a personal key. + +- Test each advertised format variant, including optional params like `embedding_types` — catches non-default response and content bugs + +- Include all supported OSes in `.github/workflows/` test matrices — catches OS-specific bugs + +## Code Style + + +- Update Haystack `Document`s with `dataclasses.replace(...)` — avoids in-place mutation bugs + +- Exclude generated artifacts from feature PRs — let release/merge workflows create them + +- Inline tiny private helpers used once — keeps control flow clear and avoids indirection + +## Dependencies + + +- Run `hatch` inside `integrations//` — each integration owns its envs + +- Pin `.github/workflows/` Action `uses:` deps to full commit SHAs — avoid mutable refs + +## Imports + + +- Import moved 3.0 integrations from `haystack_integrations`, not `haystack` — avoids broken imports + +- Import `from haystack import logging` for project logging; keep `logging.getLogger(__name__)` — This uses Haystack’s logging behavior consistently across tests, components, and integrations while preserving standard logger naming. + +## General + + +- Type inputs/returns precisely; replace `Any` when supported values or shapes are known — Precise structures and element types improve type checking, document API contracts, and prevent shape-related bugs. + +- Use `metadata_field`/`metadata_fields` in public APIs — clarifies document metadata args + +- Raise `TypeError` for wrong input types/shapes; `ValueError` for invalid values/config — This preserves consistent API misuse semantics and helps callers distinguish type errors from invalid values. + +## File-Specific Rules + +### `README.md` + +- Sort the `README.md` integrations table alphabetically — keeps entries findable and diffs clean + +## API Docstring Style + +_When to check: When writing or updating public API, component, constructor, method, or usage-example docstrings_ + + +- Use single backticks for inline code in prose; reserve double backticks for Haystack release notes — Single-backtick inline code renders consistently across docstrings and docs, while preserving the Haystack release-note exception. + +- Document all public callable params; put explicit `__init__` params in the `__init__` docstring — Keeps public API docs accurate and ensures generated documentation shows constructor args in the expected place. + +- Document performance caveats in API docstrings — warn about slow queries or large scans + +- Write concise public docstrings and sync async variants — document real contracts, not internals + +- Use single-line Markdown links in docstrings/comments — keeps docs readable and links rendering + +- Document public returns with Sphinx `:returns:` — include mapping keys/types and match `@component.output_types(...)` names + +- Describe `__init__` params by purpose/constraints; omit defaults already in signatures — Function signatures already expose defaults, so repeating them in docstrings creates stale documentation when defaults change. + +- Document public method exceptions with concrete `:raises:` conditions — no `If ...` placeholders + +- Write Haystack-style docstrings — one-line summary, blank line, then unindented sections/examples + +- Use `### Usage example` and fenced ` ```python ` blocks for Python examples — avoids renderer issues + +## Integration and Component Documentation + +_When to check: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts_ + + +- Keep `integrations/*/README.md` minimal and template-aligned — link to canonical docs/examples instead of duplicating long content + +- Leave generated docs under `integrations/` untouched — pipelines own `CHANGELOG.md` and API docs + +- Document auth by credential presence and `Secret` inputs — prevents unclear setup paths + +- Document only needed local test prerequisites in `integrations/{integration}/README.md` — avoids misleading setup + +- Document option precedence in `integrations` docs — list named options overridden by maps/headers + +- Document model-dependent APIs with canonical links — clarify supported kwargs, models, defaults, modes, outputs, and whether lists are exhaustive + +- Fill `integrations/*/LICENSE.txt` copyright fields — use real years and holders, not placeholders + +- Document non-obvious public API requirements — include ranges, external limits, syntax, and examples + +- Sync constructor docs, defaults, and `None` fallbacks — state env/base/upstream defaults + +- Document integrations with context-complete examples — show required `Document` setup, workflow execution, and output production + +- Align integration retriever docstrings with actual retrieval/scoring support — avoids misleading users + +## Directory-specific conventions + +### `integrations/` + + +- Put async document-store tests in `test_document_store_async.py` — keeps sync/async coverage clear + +- Parametrize duplicate `pytest` tests — merge same behavior into one test with `@pytest.mark.parametrize` + +- Base document store tests on `haystack.testing.document_store` classes — avoids duplicate, incomplete contract tests + +- Test `to_dict()`/`from_dict()` round trips with non-default init params — preserve serializable config + +- Test both sync and async integration paths — mirror gates, fixtures, inputs, and assertions + +- Store fixtures in `integrations/{integration}/tests/test_files/` — keeps tests local, stable, and isolated + +- Consolidate `integrations//tests/` by component — add document-store coverage to `test_document_store.py`, not one-off files + +- Test only changed integration chat behavior in `integrations/*/tests/test_*chat_generator*.py` — avoids redundant live/core coverage + +- Avoid routine `warm_up()` in `integrations/*/tests` init tests — call it only when asserting warm-up behavior + +- Assert persisted outcomes after mutating document-store tests — catches cleanup/index regressions + +- Delete unused test fixtures/helpers/setup — keeps integration tests focused and maintainable + +- Test chat helper conversions directly — cover provider-specific reasoning/thinking content + +- Align pytest markers in `integrations/*/pyproject.toml` — declare only used markers and set `--strict-markers` + +- Test converter skip/failure paths and warnings in `integrations/*/tests/` — preserves graceful-failure behavior + +- Test same-turn multi-tool calls in chat generator integrations — model them in one assistant message + +- Keep test comments specific and current — preserves intent and prevents stale guidance + +- Test mixed chat tools across init/runtime paths — assert merged `config.tools` and mirror sync/async coverage + +- Pass explicit init args in `integrations` tests — include `model`/backend IDs to validate custom paths + +- Test `close()`/reopen changes in `integrations/*/tests/test_document_store.py` — keeps lifecycle coverage consistent + +- Use local `pytest` fixtures only for shared non-trivial setup — keeps tests clear and uncoupled + +- Test provider streaming end-to-end in `integrations/*/tests/test_chat_generator.py` — assert every `StreamingChunk`, metadata/usage/finish field, tool-call/reasoning output, and final `ChatMessage` from realistic provider chunk sequences. + +- Update `integrations/amazon_bedrock/tests/` with generator changes — cover full requests, feature paths, and config deserialization + +- Test only real legacy serialization formats — avoid fake shims for missing current fields + +- Assert secret-backed credentials restore and resolve — prevents hidden credential bugs in tests + +- Use keyword-only args for optional public API params — preserves backward compatibility + +- Use case-insensitive literal substrings for metadata/search filters; in IBM DB use `LOCATE(UPPER(?), UPPER(column)) > 0`, not `LIKE` — Case-normalized literal matching keeps search behavior consistent across integrations and prevents `%`/`_` wildcard bugs in IBM DB. + +- Use `Secret` for sensitive integration config/API values — prevents leaked credentials + +- Prefix non-public helpers in `integrations` with `_` — clarifies API boundaries + +- Add async APIs only for native async I/O; mirror sync contracts and tests — Keeps async APIs non-blocking and consistent with sync behavior, preventing event-loop stalls and contract drift. + +- Keep public Document Store APIs consistent across backends — preserves portability + +- Expose only wired, supported public params; reject or document unsupported filters/flags — Avoids misleading no-op APIs, runtime surprises, and inconsistent integration behavior. + +- Use Haystack default serialization when `init_parameters` can rebuild the component — avoids brittle custom `to_dict`/`from_dict`; require valid `init_parameters` for custom deserialization. + +- Set `ToolCallDelta.index` from provider-stable call IDs — preserves chunk correlation + +- Accept `meta` as `dict | list[dict] | None` for multi-`sources` converters — keeps integration metadata semantics consistent + +- Merge `ByteStream.meta` into converter `Document.meta` and document it — preserves metadata + +- Declare `SUPPORTED_MODELS` beside limited integration components — documents model limits + +- Use backend-native bulk APIs in integration document stores — improves throughput and avoids race-prone per-document logic + +- Preserve provider-native stream indices in `StreamingChunk` — don’t hardcode or reshape for helpers + +- Apply filters before iterating, aggregating, or paginating docs — reuse `filter_documents(filters=filters)` to prevent query bugs + +- Use `streaming_callback` for `StreamingChunk`s — don't return chunks in outputs; preserve metadata, handle unsupported chunk shapes explicitly, and test streaming/non-streaming paths + +- Validate concrete backend deps in `__init__` — fail fast when objects are incompatible + +- Preserve converter source provenance — use original paths or `ByteStream.meta['file_path']`, not synthetic temp filenames + +- Expose reusable integrations as named importable APIs — e.g., `@component` `GitHubFileEditor` plus `GitHubFileEditorTool` + +- Use `filter_policy` and `apply_filter_policy(...)` for retriever filters — avoids inconsistent merge bugs + +- Align `integrations/*/pyproject.toml` Hatch docs config — use `haystack-pydoc pydoc/config_docusaurus.yml` and only docs/lint deps like `haystack-pydoc-tools` and `ruff` + +- Populate `project.keywords` in each `integrations/*/pyproject.toml` — improves package discoverability + +- Align `integrations/*/pyproject.toml` Python metadata with tested support — prevents invalid installs and stale compatibility claims + +- Align `integrations//pyproject.toml` Ruff config with the shared template; put test-only ignores under `[tool.ruff.lint.per-file-ignores]` for `"tests/**/*"` — keeps integration linting consistent without weakening global rules + +- Align `integrations/*/pyproject.toml` with the canonical template — ensures consistent packaging + +- Set real `[project].authors` in `integrations/*/pyproject.toml` — captures true maintainer and partner ownership + +- Set `description` in `integrations/*/pyproject.toml` to approved integration wording — avoids vague package metadata + +- Prefer type-correct code over `# type: ignore`; if needed, use exact-line `# type: ignore[code]` with a safety comment — Type-correct code avoids hiding real bugs, while targeted suppressions keep unavoidable checker limitations auditable and safe. + +- Place integration `py.typed` at the exposed package boundary, e.g. `haystack_integrations/tools/py.typed` — enables correct type discovery + +- Align Haystack `run()` return types with `@component.output_types(...)` and returned `dict[...]` shape — Keeps Haystack component APIs type-safe and consistent with `@component.output_types(...)`, preventing misleading nullable or overly broad contracts. + +- Centralize untyped import suppressions in `integrations/*/pyproject.toml` — avoids scattered `# type: ignore[import-untyped]` + +- Keep each integration’s tooling in `pyproject.toml`; include every importable package in `types` — ensures all shipped integration code is type-checked + +- Use direct annotations for available types — avoid unnecessary quoted strings + +- Mark state-free helper methods `@staticmethod` — clarifies no `self`/`cls` coupling + +- Use structured `logger.*` placeholders in `integrations/` — pass dynamic values as kwargs + +- Delete obsolete `integrations/` artifacts — stale docs, examples, deps, and configs mislead users + +- Keep `integrations/**/haystack_integrations` roots/intermediates namespace-only — omit `__init__.py` unless a concrete integration package needs exports/init + +- Keep sync/async `document_store.py` methods symmetrical; share filter/count/arg/error helpers — Symmetric sync/async APIs and shared helpers reduce drift, duplicated bugs, and inconsistent behavior across integration document stores. + +- Initialize external clients and `None`→instance attrs in `warm_up()` — avoids lazy runtime failures + +- Set explicit minimum deps in `integrations/*/pyproject.toml`; avoid pins/upper bounds unless required — Accurate lower bounds keep integrations installable on the oldest compatible stack while avoiding unnecessary resolver conflicts and premature incompatibility with newer Haystack releases. + +- Use official provider Python SDKs in `integrations` when they cover the workflow — reduces provider API bugs + +- Keep `integrations/*/pyproject.toml` test deps minimal — rely on inherited deps and add only imports/tests need + +- Declare only directly used runtime deps in `integrations/*/pyproject.toml` — avoids bloated installs + +- Use `request_with_retry`/`async_request_with_retry` for HTTP retries — avoid custom loops and expose `timeout`/`max_retries` + +- Use canonical integration names everywhere — match package names in `integrations/*`, READMEs, URLs, and tables + +- Name async document-store methods `_async` — keeps sync/async APIs and logs unambiguous + +- Import required deps at module top; reserve lazy/`try` imports for optional deps or cycles — Failing fast exposes missing required packages during import instead of hiding broken integrations until runtime. + +- Re-export only intentional public API in `__init__.py` — preserves stable imports + +- Wrap document-store backend failures as `DocumentStoreError` — keep sync/async handling consistent and preserve Elasticsearch bulk write/delete `try`/`except` behavior unless intentionally documented + + From 8a94655a14d06384d69e08944de90455affd3387 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sat, 15 Aug 2026 09:41:28 +0200 Subject: [PATCH 2/4] docs: split AGENTS.md into nested files and slim the always-loaded set Review feedback on the single-file version: - Nested layout, as braindump generates it. The cross-integration rules move to integrations/AGENTS.md and the two topic guides stay as linked agent_docs/ pages, so the root file carries only what applies repo-wide. Root drops 5,755 -> 2,116 tokens (-64%). - CLAUDE.md now imports AGENTS.md via '@AGENTS.md'. Claude Code reads CLAUDE.md, not AGENTS.md, and the previous prose ('read the AGENTS.md file...') only worked if the model chose to act on it. The import inlines the content at session start. integrations/ gets the same one-line CLAUDE.md, since nested memory files are picked up lazily. - Dropped the inline markers, ~10% of every file for traceability nothing reads at runtime. Note that most work in this repo happens under integrations/, so the split saves less here than in core -- the win is mainly that repo-level work (CI, tooling, docs) no longer loads 73 integration-specific rules. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 253 +----------------- CLAUDE.md | 2 +- agent_docs/api-docstring-style.md | 20 ++ integrations/AGENTS.md | 106 ++++++++ integrations/CLAUDE.md | 3 + ...integration-and-component-documentation.md | 21 ++ 6 files changed, 163 insertions(+), 242 deletions(-) create mode 100644 agent_docs/api-docstring-style.md create mode 100644 integrations/AGENTS.md create mode 100644 integrations/CLAUDE.md create mode 100644 integrations/agent_docs/integration-and-component-documentation.md diff --git a/AGENTS.md b/AGENTS.md index ce08941a11..e68276952a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,314 +82,85 @@ Follow the instructions in the "Create a new integration" section of `CONTRIBUTI The rules below were mined from 2,772 PR review comments written by the deepset team between 2025-07-01 and 2026-08-14, then filtered against the current source tree so that -guidance referring to APIs removed or moved in Haystack 3.0 does not survive. Each -`` marker traces back to the review comments it came from. +guidance referring to APIs removed or moved in Haystack 3.0 does not survive. They describe what reviewers actually enforce. Follow them the way you would follow a reviewer's note: they encode reasons, not ceremony, so when a rule genuinely does not fit the change at hand, say why rather than contorting the code to satisfy it. +Also see directory-specific guidelines: + +- [integrations/AGENTS.md](integrations/AGENTS.md) + ## API Design - - Target Haystack 3.x APIs: use `ChatGenerator`s, create clients in `warm_up()`, and support the module allowlist — Keeps integrations compatible with Haystack 3.x lifecycle, chat APIs, and safe deserialization behavior. - - Keep `to_dict()`/`from_dict()` symmetric with `__init__` — preserve all runtime config, including Watsonx `max_retries`, `meta_fields_to_embed`, and `embedding_separator`. - - Keep Haystack `__init__` light; route reusable setup through `warm_up()` — prevents slow construction and duplicated lifecycle logic, especially in `integrations/mcp/src/haystack_integrations/tools/mcp/` - - Preserve protocol parameter order; append store-specific args and pass ambiguous optionals by keyword — Maintains protocol compatibility while allowing implementations to add store-specific options without breaking callers. - - Align public API names across signatures, docs, code, and returns — document intentional mismatches - - Make required `run(...)` inputs explicit keyword-only args — prevents no-op component runs - - Use `StreamingChunk.reasoning`/`ReasoningContent` for reasoning — keep `meta` incidental - - Resolve `Secret` values in `warm_up`, not `__init__` — avoids init-time side effects - - Use Haystack serialization shape with runtime `type` — e.g. `{"type": generate_qualified_class_name(type(self)), "data": ...}` - - Align `Document` embedder constructors in `haystack/components/embedders` and `integrations/*/src/haystack_integrations/components/embedders/*` — include provider-relevant options like `meta_fields_to_embed`, `embedding_separator`, `prefix`, `suffix`, and `batch_size` when comparable embedders expose them - - Make `warm_up()` idempotent with an `__init__` flag set only after setup succeeds — This prevents repeated expensive setup and avoids partially marking failed initialization as ready. - - Type chat generator `tools` as `ToolsType`; don’t narrow provider-native params — preserves Haystack API compatibility and tool pass-through ## Documentation - - Keep comments concise and substantive — explain non-obvious intent, limits, edge cases, workarounds, or real compat needs - - Update `pydoc/config_docusaurus.yml` for public API changes — include modules, retrievers, and errors - - Keep docstrings authoritative and preserve reference links — prevents stale or duplicated docs - - Add the standard `SPDX-FileCopyrightText` and `Apache-2.0` header before imports — Ensures repository-wide license compliance and keeps source and test files consistent across core and integration packages. - - Keep public examples current with supported APIs — update docstrings, cookbooks, integrations, Google GenAI model names, and `RagasEvaluator` `ragas.metrics.collections` usage ## Config - - Source workflow creds from matching `${{ secrets. }}` env vars — keeps CI secure and reliable - - Align workflow `python-version` matrices to min/max supported Python — catches boundary regressions - - Default secrets from provider env vars; use `WATSONX_API_KEY` for Watsonx components — Predictable env-var defaults make credentials easy to configure across integrations while avoiding hardcoded secrets or setup surprises. - - Use SDK env var names consistently across CI, tests, secrets, and skips — prevents config drift - - Store credential fields as Haystack `Secret`, not `str` — avoids leaking sensitive config ## Testing - - Gate `integrations//tests/` explicitly — wire CI coverage, inject `Secret`/env values, use provider-specific `pytest.mark.skipif(...)` reasons, precise `sys.version_info`/`sys.platform` checks, and validate external-API tests locally with a personal key. - - Test each advertised format variant, including optional params like `embedding_types` — catches non-default response and content bugs - - Include all supported OSes in `.github/workflows/` test matrices — catches OS-specific bugs ## Code Style - - Update Haystack `Document`s with `dataclasses.replace(...)` — avoids in-place mutation bugs - - Exclude generated artifacts from feature PRs — let release/merge workflows create them - - Inline tiny private helpers used once — keeps control flow clear and avoids indirection ## Dependencies - - Run `hatch` inside `integrations//` — each integration owns its envs - - Pin `.github/workflows/` Action `uses:` deps to full commit SHAs — avoid mutable refs ## Imports - - Import moved 3.0 integrations from `haystack_integrations`, not `haystack` — avoids broken imports - - Import `from haystack import logging` for project logging; keep `logging.getLogger(__name__)` — This uses Haystack’s logging behavior consistently across tests, components, and integrations while preserving standard logger naming. ## General - - Type inputs/returns precisely; replace `Any` when supported values or shapes are known — Precise structures and element types improve type checking, document API contracts, and prevent shape-related bugs. - - Use `metadata_field`/`metadata_fields` in public APIs — clarifies document metadata args - - Raise `TypeError` for wrong input types/shapes; `ValueError` for invalid values/config — This preserves consistent API misuse semantics and helps callers distinguish type errors from invalid values. +## Topic Guides + +Check these when working in specific areas: + +- **[API Docstring Style](agent_docs/api-docstring-style.md)**: When writing or updating public API, component, constructor, method, or usage-example docstrings + ## File-Specific Rules ### `README.md` - -- Sort the `README.md` integrations table alphabetically — keeps entries findable and diffs clean -## API Docstring Style - -_When to check: When writing or updating public API, component, constructor, method, or usage-example docstrings_ - - -- Use single backticks for inline code in prose; reserve double backticks for Haystack release notes — Single-backtick inline code renders consistently across docstrings and docs, while preserving the Haystack release-note exception. - -- Document all public callable params; put explicit `__init__` params in the `__init__` docstring — Keeps public API docs accurate and ensures generated documentation shows constructor args in the expected place. - -- Document performance caveats in API docstrings — warn about slow queries or large scans - -- Write concise public docstrings and sync async variants — document real contracts, not internals - -- Use single-line Markdown links in docstrings/comments — keeps docs readable and links rendering - -- Document public returns with Sphinx `:returns:` — include mapping keys/types and match `@component.output_types(...)` names - -- Describe `__init__` params by purpose/constraints; omit defaults already in signatures — Function signatures already expose defaults, so repeating them in docstrings creates stale documentation when defaults change. - -- Document public method exceptions with concrete `:raises:` conditions — no `If ...` placeholders - -- Write Haystack-style docstrings — one-line summary, blank line, then unindented sections/examples - -- Use `### Usage example` and fenced ` ```python ` blocks for Python examples — avoids renderer issues - -## Integration and Component Documentation - -_When to check: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts_ - - -- Keep `integrations/*/README.md` minimal and template-aligned — link to canonical docs/examples instead of duplicating long content - -- Leave generated docs under `integrations/` untouched — pipelines own `CHANGELOG.md` and API docs - -- Document auth by credential presence and `Secret` inputs — prevents unclear setup paths - -- Document only needed local test prerequisites in `integrations/{integration}/README.md` — avoids misleading setup - -- Document option precedence in `integrations` docs — list named options overridden by maps/headers - -- Document model-dependent APIs with canonical links — clarify supported kwargs, models, defaults, modes, outputs, and whether lists are exhaustive - -- Fill `integrations/*/LICENSE.txt` copyright fields — use real years and holders, not placeholders - -- Document non-obvious public API requirements — include ranges, external limits, syntax, and examples - -- Sync constructor docs, defaults, and `None` fallbacks — state env/base/upstream defaults - -- Document integrations with context-complete examples — show required `Document` setup, workflow execution, and output production - -- Align integration retriever docstrings with actual retrieval/scoring support — avoids misleading users - -## Directory-specific conventions - -### `integrations/` - - -- Put async document-store tests in `test_document_store_async.py` — keeps sync/async coverage clear - -- Parametrize duplicate `pytest` tests — merge same behavior into one test with `@pytest.mark.parametrize` - -- Base document store tests on `haystack.testing.document_store` classes — avoids duplicate, incomplete contract tests - -- Test `to_dict()`/`from_dict()` round trips with non-default init params — preserve serializable config - -- Test both sync and async integration paths — mirror gates, fixtures, inputs, and assertions - -- Store fixtures in `integrations/{integration}/tests/test_files/` — keeps tests local, stable, and isolated - -- Consolidate `integrations//tests/` by component — add document-store coverage to `test_document_store.py`, not one-off files - -- Test only changed integration chat behavior in `integrations/*/tests/test_*chat_generator*.py` — avoids redundant live/core coverage - -- Avoid routine `warm_up()` in `integrations/*/tests` init tests — call it only when asserting warm-up behavior - -- Assert persisted outcomes after mutating document-store tests — catches cleanup/index regressions - -- Delete unused test fixtures/helpers/setup — keeps integration tests focused and maintainable - -- Test chat helper conversions directly — cover provider-specific reasoning/thinking content - -- Align pytest markers in `integrations/*/pyproject.toml` — declare only used markers and set `--strict-markers` - -- Test converter skip/failure paths and warnings in `integrations/*/tests/` — preserves graceful-failure behavior - -- Test same-turn multi-tool calls in chat generator integrations — model them in one assistant message - -- Keep test comments specific and current — preserves intent and prevents stale guidance - -- Test mixed chat tools across init/runtime paths — assert merged `config.tools` and mirror sync/async coverage - -- Pass explicit init args in `integrations` tests — include `model`/backend IDs to validate custom paths - -- Test `close()`/reopen changes in `integrations/*/tests/test_document_store.py` — keeps lifecycle coverage consistent - -- Use local `pytest` fixtures only for shared non-trivial setup — keeps tests clear and uncoupled - -- Test provider streaming end-to-end in `integrations/*/tests/test_chat_generator.py` — assert every `StreamingChunk`, metadata/usage/finish field, tool-call/reasoning output, and final `ChatMessage` from realistic provider chunk sequences. - -- Update `integrations/amazon_bedrock/tests/` with generator changes — cover full requests, feature paths, and config deserialization - -- Test only real legacy serialization formats — avoid fake shims for missing current fields - -- Assert secret-backed credentials restore and resolve — prevents hidden credential bugs in tests - -- Use keyword-only args for optional public API params — preserves backward compatibility - -- Use case-insensitive literal substrings for metadata/search filters; in IBM DB use `LOCATE(UPPER(?), UPPER(column)) > 0`, not `LIKE` — Case-normalized literal matching keeps search behavior consistent across integrations and prevents `%`/`_` wildcard bugs in IBM DB. - -- Use `Secret` for sensitive integration config/API values — prevents leaked credentials - -- Prefix non-public helpers in `integrations` with `_` — clarifies API boundaries - -- Add async APIs only for native async I/O; mirror sync contracts and tests — Keeps async APIs non-blocking and consistent with sync behavior, preventing event-loop stalls and contract drift. - -- Keep public Document Store APIs consistent across backends — preserves portability - -- Expose only wired, supported public params; reject or document unsupported filters/flags — Avoids misleading no-op APIs, runtime surprises, and inconsistent integration behavior. - -- Use Haystack default serialization when `init_parameters` can rebuild the component — avoids brittle custom `to_dict`/`from_dict`; require valid `init_parameters` for custom deserialization. - -- Set `ToolCallDelta.index` from provider-stable call IDs — preserves chunk correlation - -- Accept `meta` as `dict | list[dict] | None` for multi-`sources` converters — keeps integration metadata semantics consistent - -- Merge `ByteStream.meta` into converter `Document.meta` and document it — preserves metadata - -- Declare `SUPPORTED_MODELS` beside limited integration components — documents model limits - -- Use backend-native bulk APIs in integration document stores — improves throughput and avoids race-prone per-document logic - -- Preserve provider-native stream indices in `StreamingChunk` — don’t hardcode or reshape for helpers - -- Apply filters before iterating, aggregating, or paginating docs — reuse `filter_documents(filters=filters)` to prevent query bugs - -- Use `streaming_callback` for `StreamingChunk`s — don't return chunks in outputs; preserve metadata, handle unsupported chunk shapes explicitly, and test streaming/non-streaming paths - -- Validate concrete backend deps in `__init__` — fail fast when objects are incompatible - -- Preserve converter source provenance — use original paths or `ByteStream.meta['file_path']`, not synthetic temp filenames - -- Expose reusable integrations as named importable APIs — e.g., `@component` `GitHubFileEditor` plus `GitHubFileEditorTool` - -- Use `filter_policy` and `apply_filter_policy(...)` for retriever filters — avoids inconsistent merge bugs - -- Align `integrations/*/pyproject.toml` Hatch docs config — use `haystack-pydoc pydoc/config_docusaurus.yml` and only docs/lint deps like `haystack-pydoc-tools` and `ruff` - -- Populate `project.keywords` in each `integrations/*/pyproject.toml` — improves package discoverability - -- Align `integrations/*/pyproject.toml` Python metadata with tested support — prevents invalid installs and stale compatibility claims - -- Align `integrations//pyproject.toml` Ruff config with the shared template; put test-only ignores under `[tool.ruff.lint.per-file-ignores]` for `"tests/**/*"` — keeps integration linting consistent without weakening global rules - -- Align `integrations/*/pyproject.toml` with the canonical template — ensures consistent packaging - -- Set real `[project].authors` in `integrations/*/pyproject.toml` — captures true maintainer and partner ownership - -- Set `description` in `integrations/*/pyproject.toml` to approved integration wording — avoids vague package metadata - -- Prefer type-correct code over `# type: ignore`; if needed, use exact-line `# type: ignore[code]` with a safety comment — Type-correct code avoids hiding real bugs, while targeted suppressions keep unavoidable checker limitations auditable and safe. - -- Place integration `py.typed` at the exposed package boundary, e.g. `haystack_integrations/tools/py.typed` — enables correct type discovery - -- Align Haystack `run()` return types with `@component.output_types(...)` and returned `dict[...]` shape — Keeps Haystack component APIs type-safe and consistent with `@component.output_types(...)`, preventing misleading nullable or overly broad contracts. - -- Centralize untyped import suppressions in `integrations/*/pyproject.toml` — avoids scattered `# type: ignore[import-untyped]` - -- Keep each integration’s tooling in `pyproject.toml`; include every importable package in `types` — ensures all shipped integration code is type-checked - -- Use direct annotations for available types — avoid unnecessary quoted strings - -- Mark state-free helper methods `@staticmethod` — clarifies no `self`/`cls` coupling - -- Use structured `logger.*` placeholders in `integrations/` — pass dynamic values as kwargs - -- Delete obsolete `integrations/` artifacts — stale docs, examples, deps, and configs mislead users - -- Keep `integrations/**/haystack_integrations` roots/intermediates namespace-only — omit `__init__.py` unless a concrete integration package needs exports/init - -- Keep sync/async `document_store.py` methods symmetrical; share filter/count/arg/error helpers — Symmetric sync/async APIs and shared helpers reduce drift, duplicated bugs, and inconsistent behavior across integration document stores. - -- Initialize external clients and `None`→instance attrs in `warm_up()` — avoids lazy runtime failures - -- Set explicit minimum deps in `integrations/*/pyproject.toml`; avoid pins/upper bounds unless required — Accurate lower bounds keep integrations installable on the oldest compatible stack while avoiding unnecessary resolver conflicts and premature incompatibility with newer Haystack releases. - -- Use official provider Python SDKs in `integrations` when they cover the workflow — reduces provider API bugs - -- Keep `integrations/*/pyproject.toml` test deps minimal — rely on inherited deps and add only imports/tests need - -- Declare only directly used runtime deps in `integrations/*/pyproject.toml` — avoids bloated installs - -- Use `request_with_retry`/`async_request_with_retry` for HTTP retries — avoid custom loops and expose `timeout`/`max_retries` - -- Use canonical integration names everywhere — match package names in `integrations/*`, READMEs, URLs, and tables - -- Name async document-store methods `_async` — keeps sync/async APIs and logs unambiguous - -- Import required deps at module top; reserve lazy/`try` imports for optional deps or cycles — Failing fast exposes missing required packages during import instead of hiding broken integrations until runtime. - -- Re-export only intentional public API in `__init__.py` — preserves stable imports - -- Wrap document-store backend failures as `DocumentStoreError` — keep sync/async handling consistent and preserve Elasticsearch bulk write/delete `try`/`except` behavior unless intentionally documented +- Sort the `README.md` integrations table alphabetically — keeps entries findable and diffs clean diff --git a/CLAUDE.md b/CLAUDE.md index 8f8a6efba4..f6aa6c0262 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,3 +1,3 @@ # CLAUDE.md -Before you start working on this repository, read the AGENTS.md file and follow all the instructions. +@AGENTS.md diff --git a/agent_docs/api-docstring-style.md b/agent_docs/api-docstring-style.md new file mode 100644 index 0000000000..aeb365f9fa --- /dev/null +++ b/agent_docs/api-docstring-style.md @@ -0,0 +1,20 @@ + + +# API Docstring Style + +> How to write public API and component docstrings: concise Haystack-style summaries, complete parameter/return/exception documentation, synchronized sync/async docs, performance caveats, default-value wording, inline-code markup, Markdown links, and fenced Python usage examples. + +**When to check**: When writing or updating public API, component, constructor, method, or usage-example docstrings + +## Rules + +- Use single backticks for inline code in prose; reserve double backticks for Haystack release notes — Single-backtick inline code renders consistently across docstrings and docs, while preserving the Haystack release-note exception. +- Document all public callable params; put explicit `__init__` params in the `__init__` docstring — Keeps public API docs accurate and ensures generated documentation shows constructor args in the expected place. +- Document performance caveats in API docstrings — warn about slow queries or large scans +- Write concise public docstrings and sync async variants — document real contracts, not internals +- Use single-line Markdown links in docstrings/comments — keeps docs readable and links rendering +- Document public returns with Sphinx `:returns:` — include mapping keys/types and match `@component.output_types(...)` names +- Describe `__init__` params by purpose/constraints; omit defaults already in signatures — Function signatures already expose defaults, so repeating them in docstrings creates stale documentation when defaults change. +- Document public method exceptions with concrete `:raises:` conditions — no `If ...` placeholders +- Write Haystack-style docstrings — one-line summary, blank line, then unindented sections/examples +- Use `### Usage example` and fenced ` ```python ` blocks for Python examples — avoids renderer issues diff --git a/integrations/AGENTS.md b/integrations/AGENTS.md new file mode 100644 index 0000000000..fbd99d6f4c --- /dev/null +++ b/integrations/AGENTS.md @@ -0,0 +1,106 @@ + + +# integrations/ Guidelines + +## Testing + +- Put async document-store tests in `test_document_store_async.py` — keeps sync/async coverage clear +- Parametrize duplicate `pytest` tests — merge same behavior into one test with `@pytest.mark.parametrize` +- Base document store tests on `haystack.testing.document_store` classes — avoids duplicate, incomplete contract tests +- Test `to_dict()`/`from_dict()` round trips with non-default init params — preserve serializable config +- Test both sync and async integration paths — mirror gates, fixtures, inputs, and assertions +- Store fixtures in `integrations/{integration}/tests/test_files/` — keeps tests local, stable, and isolated +- Consolidate `integrations//tests/` by component — add document-store coverage to `test_document_store.py`, not one-off files +- Test only changed integration chat behavior in `integrations/*/tests/test_*chat_generator*.py` — avoids redundant live/core coverage +- Avoid routine `warm_up()` in `integrations/*/tests` init tests — call it only when asserting warm-up behavior +- Assert persisted outcomes after mutating document-store tests — catches cleanup/index regressions +- Delete unused test fixtures/helpers/setup — keeps integration tests focused and maintainable +- Test chat helper conversions directly — cover provider-specific reasoning/thinking content +- Align pytest markers in `integrations/*/pyproject.toml` — declare only used markers and set `--strict-markers` +- Test converter skip/failure paths and warnings in `integrations/*/tests/` — preserves graceful-failure behavior +- Test same-turn multi-tool calls in chat generator integrations — model them in one assistant message +- Keep test comments specific and current — preserves intent and prevents stale guidance +- Test mixed chat tools across init/runtime paths — assert merged `config.tools` and mirror sync/async coverage +- Pass explicit init args in `integrations` tests — include `model`/backend IDs to validate custom paths +- Test `close()`/reopen changes in `integrations/*/tests/test_document_store.py` — keeps lifecycle coverage consistent +- Use local `pytest` fixtures only for shared non-trivial setup — keeps tests clear and uncoupled +- Test provider streaming end-to-end in `integrations/*/tests/test_chat_generator.py` — assert every `StreamingChunk`, metadata/usage/finish field, tool-call/reasoning output, and final `ChatMessage` from realistic provider chunk sequences. +- Update `integrations/amazon_bedrock/tests/` with generator changes — cover full requests, feature paths, and config deserialization +- Test only real legacy serialization formats — avoid fake shims for missing current fields +- Assert secret-backed credentials restore and resolve — prevents hidden credential bugs in tests + +## API Design + +- Use keyword-only args for optional public API params — preserves backward compatibility +- Use case-insensitive literal substrings for metadata/search filters; in IBM DB use `LOCATE(UPPER(?), UPPER(column)) > 0`, not `LIKE` — Case-normalized literal matching keeps search behavior consistent across integrations and prevents `%`/`_` wildcard bugs in IBM DB. +- Use `Secret` for sensitive integration config/API values — prevents leaked credentials +- Prefix non-public helpers in `integrations` with `_` — clarifies API boundaries +- Add async APIs only for native async I/O; mirror sync contracts and tests — Keeps async APIs non-blocking and consistent with sync behavior, preventing event-loop stalls and contract drift. +- Keep public Document Store APIs consistent across backends — preserves portability +- Expose only wired, supported public params; reject or document unsupported filters/flags — Avoids misleading no-op APIs, runtime surprises, and inconsistent integration behavior. +- Use Haystack default serialization when `init_parameters` can rebuild the component — avoids brittle custom `to_dict`/`from_dict`; require valid `init_parameters` for custom deserialization. +- Set `ToolCallDelta.index` from provider-stable call IDs — preserves chunk correlation +- Accept `meta` as `dict | list[dict] | None` for multi-`sources` converters — keeps integration metadata semantics consistent +- Merge `ByteStream.meta` into converter `Document.meta` and document it — preserves metadata +- Declare `SUPPORTED_MODELS` beside limited integration components — documents model limits +- Use backend-native bulk APIs in integration document stores — improves throughput and avoids race-prone per-document logic +- Preserve provider-native stream indices in `StreamingChunk` — don’t hardcode or reshape for helpers +- Apply filters before iterating, aggregating, or paginating docs — reuse `filter_documents(filters=filters)` to prevent query bugs +- Use `streaming_callback` for `StreamingChunk`s — don't return chunks in outputs; preserve metadata, handle unsupported chunk shapes explicitly, and test streaming/non-streaming paths +- Validate concrete backend deps in `__init__` — fail fast when objects are incompatible +- Preserve converter source provenance — use original paths or `ByteStream.meta['file_path']`, not synthetic temp filenames +- Expose reusable integrations as named importable APIs — e.g., `@component` `GitHubFileEditor` plus `GitHubFileEditorTool` +- Use `filter_policy` and `apply_filter_policy(...)` for retriever filters — avoids inconsistent merge bugs + +## Config + +- Align `integrations/*/pyproject.toml` Hatch docs config — use `haystack-pydoc pydoc/config_docusaurus.yml` and only docs/lint deps like `haystack-pydoc-tools` and `ruff` +- Populate `project.keywords` in each `integrations/*/pyproject.toml` — improves package discoverability +- Align `integrations/*/pyproject.toml` Python metadata with tested support — prevents invalid installs and stale compatibility claims +- Align `integrations//pyproject.toml` Ruff config with the shared template; put test-only ignores under `[tool.ruff.lint.per-file-ignores]` for `"tests/**/*"` — keeps integration linting consistent without weakening global rules +- Align `integrations/*/pyproject.toml` with the canonical template — ensures consistent packaging +- Set real `[project].authors` in `integrations/*/pyproject.toml` — captures true maintainer and partner ownership +- Set `description` in `integrations/*/pyproject.toml` to approved integration wording — avoids vague package metadata + +## Type System + +- Prefer type-correct code over `# type: ignore`; if needed, use exact-line `# type: ignore[code]` with a safety comment — Type-correct code avoids hiding real bugs, while targeted suppressions keep unavoidable checker limitations auditable and safe. +- Place integration `py.typed` at the exposed package boundary, e.g. `haystack_integrations/tools/py.typed` — enables correct type discovery +- Align Haystack `run()` return types with `@component.output_types(...)` and returned `dict[...]` shape — Keeps Haystack component APIs type-safe and consistent with `@component.output_types(...)`, preventing misleading nullable or overly broad contracts. +- Centralize untyped import suppressions in `integrations/*/pyproject.toml` — avoids scattered `# type: ignore[import-untyped]` +- Keep each integration’s tooling in `pyproject.toml`; include every importable package in `types` — ensures all shipped integration code is type-checked +- Use direct annotations for available types — avoid unnecessary quoted strings + +## Code Style + +- Mark state-free helper methods `@staticmethod` — clarifies no `self`/`cls` coupling +- Use structured `logger.*` placeholders in `integrations/` — pass dynamic values as kwargs +- Delete obsolete `integrations/` artifacts — stale docs, examples, deps, and configs mislead users +- Keep `integrations/**/haystack_integrations` roots/intermediates namespace-only — omit `__init__.py` unless a concrete integration package needs exports/init +- Keep sync/async `document_store.py` methods symmetrical; share filter/count/arg/error helpers — Symmetric sync/async APIs and shared helpers reduce drift, duplicated bugs, and inconsistent behavior across integration document stores. +- Initialize external clients and `None`→instance attrs in `warm_up()` — avoids lazy runtime failures + +## Dependencies + +- Set explicit minimum deps in `integrations/*/pyproject.toml`; avoid pins/upper bounds unless required — Accurate lower bounds keep integrations installable on the oldest compatible stack while avoiding unnecessary resolver conflicts and premature incompatibility with newer Haystack releases. +- Use official provider Python SDKs in `integrations` when they cover the workflow — reduces provider API bugs +- Keep `integrations/*/pyproject.toml` test deps minimal — rely on inherited deps and add only imports/tests need +- Declare only directly used runtime deps in `integrations/*/pyproject.toml` — avoids bloated installs +- Use `request_with_retry`/`async_request_with_retry` for HTTP retries — avoid custom loops and expose `timeout`/`max_retries` + +## Naming + +- Use canonical integration names everywhere — match package names in `integrations/*`, READMEs, URLs, and tables +- Name async document-store methods `_async` — keeps sync/async APIs and logs unambiguous + +## General + +- Import required deps at module top; reserve lazy/`try` imports for optional deps or cycles — Failing fast exposes missing required packages during import instead of hiding broken integrations until runtime. +- Re-export only intentional public API in `__init__.py` — preserves stable imports +- Wrap document-store backend failures as `DocumentStoreError` — keep sync/async handling consistent and preserve Elasticsearch bulk write/delete `try`/`except` behavior unless intentionally documented + +## Topic Guides + +Check these when working in specific areas: + +- **[Integration and Component Documentation](agent_docs/integration-and-component-documentation.md)**: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts diff --git a/integrations/CLAUDE.md b/integrations/CLAUDE.md new file mode 100644 index 0000000000..f6aa6c0262 --- /dev/null +++ b/integrations/CLAUDE.md @@ -0,0 +1,3 @@ +# CLAUDE.md + +@AGENTS.md diff --git a/integrations/agent_docs/integration-and-component-documentation.md b/integrations/agent_docs/integration-and-component-documentation.md new file mode 100644 index 0000000000..684ba5c467 --- /dev/null +++ b/integrations/agent_docs/integration-and-component-documentation.md @@ -0,0 +1,21 @@ + + +# Integration and Component Documentation + +> Documentation expectations for integrations and components, including minimal template-aligned integration READMEs, avoiding generated docs artifacts, documenting authentication modes, local test prerequisites, precedence rules, externally defined behavior, license placeholders, parameter requirements/defaults, realistic examples, and alignment with actual retrieval/scoring behavior. + +**When to check**: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts + +## Rules + +- Keep `integrations/*/README.md` minimal and template-aligned — link to canonical docs/examples instead of duplicating long content +- Leave generated docs under `integrations/` untouched — pipelines own `CHANGELOG.md` and API docs +- Document auth by credential presence and `Secret` inputs — prevents unclear setup paths +- Document only needed local test prerequisites in `integrations/{integration}/README.md` — avoids misleading setup +- Document option precedence in `integrations` docs — list named options overridden by maps/headers +- Document model-dependent APIs with canonical links — clarify supported kwargs, models, defaults, modes, outputs, and whether lists are exhaustive +- Fill `integrations/*/LICENSE.txt` copyright fields — use real years and holders, not placeholders +- Document non-obvious public API requirements — include ranges, external limits, syntax, and examples +- Sync constructor docs, defaults, and `None` fallbacks — state env/base/upstream defaults +- Document integrations with context-complete examples — show required `Document` setup, workflow execution, and output production +- Align integration retriever docstrings with actual retrieval/scoring support — avoids misleading users From 7c3d98de50147aec3159448ca42298d093ad8be6 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sat, 15 Aug 2026 11:47:43 +0200 Subject: [PATCH 3/4] docs: drop the braindump marker comments and generalise instance-bound rules The comments existed so the mined block could be replaced in place on a re-run. That anchor now lives in the generator instead, so the shipped files carry no scaffolding. Clustering preserves whichever example the source review comments happened to discuss, which leaves some rules reading as if they only apply to one integration: - 'preserve all runtime config, including Watsonx max_retries, ...' -> every constructor argument that affects runtime behaviour must round-trip. - 'use WATSONX_API_KEY for Watsonx components' -> default each Secret from the provider's conventional env var. WATSONX_API_KEY is one of ~10 such variables here (COHERE_API_KEY, NVIDIA_API_KEY, JINA_API_KEY, HF_API_TOKEN, ...). - 'preserve Elasticsearch bulk write/delete try/except behavior' -> preserve documented bulk write/delete error behaviour. DocumentStoreError is used by 17 document stores, not just Elasticsearch. - 'update ... Google GenAI model names and RagasEvaluator ragas.metrics.collections usage' -> refresh docstrings, cookbooks and integration docs when model names or provider APIs change. - 'especially in integrations/mcp/src/haystack_integrations/tools/mcp/' -> dropped the path; the warm_up() rule is repo-wide. Removed 'Update integrations/amazon_bedrock/tests/ with generator changes': it is entirely about one integration, and the general form is trivial. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 12 ++++-------- integrations/AGENTS.md | 3 +-- 2 files changed, 5 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e68276952a..88ed7b4093 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,8 +78,6 @@ Changelogs are auto-generated per integration and not meant to be edited manuall Follow the instructions in the "Create a new integration" section of `CONTRIBUTING.md`. - - The rules below were mined from 2,772 PR review comments written by the deepset team between 2025-07-01 and 2026-08-14, then filtered against the current source tree so that guidance referring to APIs removed or moved in Haystack 3.0 does not survive. @@ -95,8 +93,8 @@ Also see directory-specific guidelines: ## API Design - Target Haystack 3.x APIs: use `ChatGenerator`s, create clients in `warm_up()`, and support the module allowlist — Keeps integrations compatible with Haystack 3.x lifecycle, chat APIs, and safe deserialization behavior. -- Keep `to_dict()`/`from_dict()` symmetric with `__init__` — preserve all runtime config, including Watsonx `max_retries`, `meta_fields_to_embed`, and `embedding_separator`. -- Keep Haystack `__init__` light; route reusable setup through `warm_up()` — prevents slow construction and duplicated lifecycle logic, especially in `integrations/mcp/src/haystack_integrations/tools/mcp/` +- Keep `to_dict()`/`from_dict()` symmetric with `__init__` — every constructor argument that affects runtime behaviour must round-trip, including retry, batching, and metadata options +- Keep `__init__` light; route reusable setup through `warm_up()` — avoids slow construction and duplicated lifecycle logic - Preserve protocol parameter order; append store-specific args and pass ambiguous optionals by keyword — Maintains protocol compatibility while allowing implementations to add store-specific options without breaking callers. - Align public API names across signatures, docs, code, and returns — document intentional mismatches - Make required `run(...)` inputs explicit keyword-only args — prevents no-op component runs @@ -113,13 +111,13 @@ Also see directory-specific guidelines: - Update `pydoc/config_docusaurus.yml` for public API changes — include modules, retrievers, and errors - Keep docstrings authoritative and preserve reference links — prevents stale or duplicated docs - Add the standard `SPDX-FileCopyrightText` and `Apache-2.0` header before imports — Ensures repository-wide license compliance and keeps source and test files consistent across core and integration packages. -- Keep public examples current with supported APIs — update docstrings, cookbooks, integrations, Google GenAI model names, and `RagasEvaluator` `ragas.metrics.collections` usage +- Keep public examples current with supported APIs — refresh docstrings, cookbooks, and integration docs when model names or provider APIs change ## Config - Source workflow creds from matching `${{ secrets. }}` env vars — keeps CI secure and reliable - Align workflow `python-version` matrices to min/max supported Python — catches boundary regressions -- Default secrets from provider env vars; use `WATSONX_API_KEY` for Watsonx components — Predictable env-var defaults make credentials easy to configure across integrations while avoiding hardcoded secrets or setup surprises. +- Default each `Secret` from the provider's conventional env var (`COHERE_API_KEY`, `NVIDIA_API_KEY`, `WATSONX_API_KEY`, ...) — predictable defaults keep credential setup consistent and avoid hardcoded secrets - Use SDK env var names consistently across CI, tests, secrets, and skips — prevents config drift - Store credential fields as Haystack `Secret`, not `str` — avoids leaking sensitive config @@ -162,5 +160,3 @@ Check these when working in specific areas: ### `README.md` - Sort the `README.md` integrations table alphabetically — keeps entries findable and diffs clean - - diff --git a/integrations/AGENTS.md b/integrations/AGENTS.md index fbd99d6f4c..187fec09b1 100644 --- a/integrations/AGENTS.md +++ b/integrations/AGENTS.md @@ -25,7 +25,6 @@ - Test `close()`/reopen changes in `integrations/*/tests/test_document_store.py` — keeps lifecycle coverage consistent - Use local `pytest` fixtures only for shared non-trivial setup — keeps tests clear and uncoupled - Test provider streaming end-to-end in `integrations/*/tests/test_chat_generator.py` — assert every `StreamingChunk`, metadata/usage/finish field, tool-call/reasoning output, and final `ChatMessage` from realistic provider chunk sequences. -- Update `integrations/amazon_bedrock/tests/` with generator changes — cover full requests, feature paths, and config deserialization - Test only real legacy serialization formats — avoid fake shims for missing current fields - Assert secret-backed credentials restore and resolve — prevents hidden credential bugs in tests @@ -97,7 +96,7 @@ - Import required deps at module top; reserve lazy/`try` imports for optional deps or cycles — Failing fast exposes missing required packages during import instead of hiding broken integrations until runtime. - Re-export only intentional public API in `__init__.py` — preserves stable imports -- Wrap document-store backend failures as `DocumentStoreError` — keep sync/async handling consistent and preserve Elasticsearch bulk write/delete `try`/`except` behavior unless intentionally documented +- Wrap backend failures as `DocumentStoreError` — keep sync and async handling consistent, and preserve documented bulk write/delete error behaviour ## Topic Guides From a5644338d694a30e317e5d199996c1efb88998e1 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sat, 15 Aug 2026 12:17:27 +0200 Subject: [PATCH 4/4] docs: consolidate AGENTS.md into a single root file The split layout left 21 rules in agent_docs/*.md topic files that the root AGENTS.md only linked to. Those are not memory files, so nothing loads them automatically -- an agent has to choose to follow the link. Measured with canary fixtures: with file tools blocked the topic content is absent 3/3 while the root file is present, confirming it is a discretionary read rather than context. With tools available the agent did follow the link 3/3, but that test named docstrings explicitly and so matched the link's "when to check" text almost word for word; a vaguer real task matches less well. A rule that loads every time beats one that usually loads. Folding integrations/AGENTS.md in too, rather than only the topic files: nearly all work in this repo happens under integrations/, so the nested split saved little. Core keeps its nested layout, where the rules divide cleanly by directory (test/, releasenotes/notes/, docs-website/) and group produced no topic files. Cost: ~5.1k tokens loaded every session instead of ~2.1k. The tradeoff is that nothing can be missed. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 139 ++++++++++++++++-- agent_docs/api-docstring-style.md | 20 --- integrations/AGENTS.md | 105 ------------- integrations/CLAUDE.md | 3 - ...integration-and-component-documentation.md | 21 --- 5 files changed, 129 insertions(+), 159 deletions(-) delete mode 100644 agent_docs/api-docstring-style.md delete mode 100644 integrations/AGENTS.md delete mode 100644 integrations/CLAUDE.md delete mode 100644 integrations/agent_docs/integration-and-component-documentation.md diff --git a/AGENTS.md b/AGENTS.md index 88ed7b4093..5609de9249 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,10 +86,6 @@ They describe what reviewers actually enforce. Follow them the way you would fol reviewer's note: they encode reasons, not ceremony, so when a rule genuinely does not fit the change at hand, say why rather than contorting the code to satisfy it. -Also see directory-specific guidelines: - -- [integrations/AGENTS.md](integrations/AGENTS.md) - ## API Design - Target Haystack 3.x APIs: use `ChatGenerator`s, create clients in `warm_up()`, and support the module allowlist — Keeps integrations compatible with Haystack 3.x lifecycle, chat APIs, and safe deserialization behavior. @@ -149,14 +145,137 @@ Also see directory-specific guidelines: - Use `metadata_field`/`metadata_fields` in public APIs — clarifies document metadata args - Raise `TypeError` for wrong input types/shapes; `ValueError` for invalid values/config — This preserves consistent API misuse semantics and helps callers distinguish type errors from invalid values. -## Topic Guides - -Check these when working in specific areas: - -- **[API Docstring Style](agent_docs/api-docstring-style.md)**: When writing or updating public API, component, constructor, method, or usage-example docstrings - ## File-Specific Rules ### `README.md` - Sort the `README.md` integrations table alphabetically — keeps entries findable and diffs clean + +## API Docstring Style + +_When to check: When writing or updating public API, component, constructor, method, or usage-example docstrings_ + +- Use single backticks for inline code in prose; reserve double backticks for Haystack release notes — Single-backtick inline code renders consistently across docstrings and docs, while preserving the Haystack release-note exception. +- Document all public callable params; put explicit `__init__` params in the `__init__` docstring — Keeps public API docs accurate and ensures generated documentation shows constructor args in the expected place. +- Document performance caveats in API docstrings — warn about slow queries or large scans +- Write concise public docstrings and sync async variants — document real contracts, not internals +- Use single-line Markdown links in docstrings/comments — keeps docs readable and links rendering +- Document public returns with Sphinx `:returns:` — include mapping keys/types and match `@component.output_types(...)` names +- Describe `__init__` params by purpose/constraints; omit defaults already in signatures — Function signatures already expose defaults, so repeating them in docstrings creates stale documentation when defaults change. +- Document public method exceptions with concrete `:raises:` conditions — no `If ...` placeholders +- Write Haystack-style docstrings — one-line summary, blank line, then unindented sections/examples +- Use `### Usage example` and fenced ` ```python ` blocks for Python examples — avoids renderer issues + +## Integration and Component Documentation + +_When to check: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts_ + +- Keep `integrations/*/README.md` minimal and template-aligned — link to canonical docs/examples instead of duplicating long content +- Leave generated docs under `integrations/` untouched — pipelines own `CHANGELOG.md` and API docs +- Document auth by credential presence and `Secret` inputs — prevents unclear setup paths +- Document only needed local test prerequisites in `integrations/{integration}/README.md` — avoids misleading setup +- Document option precedence in `integrations` docs — list named options overridden by maps/headers +- Document model-dependent APIs with canonical links — clarify supported kwargs, models, defaults, modes, outputs, and whether lists are exhaustive +- Fill `integrations/*/LICENSE.txt` copyright fields — use real years and holders, not placeholders +- Document non-obvious public API requirements — include ranges, external limits, syntax, and examples +- Sync constructor docs, defaults, and `None` fallbacks — state env/base/upstream defaults +- Document integrations with context-complete examples — show required `Document` setup, workflow execution, and output production +- Align integration retriever docstrings with actual retrieval/scoring support — avoids misleading users + +## Conventions for `integrations/` + +### Testing + +- Put async document-store tests in `test_document_store_async.py` — keeps sync/async coverage clear +- Parametrize duplicate `pytest` tests — merge same behavior into one test with `@pytest.mark.parametrize` +- Base document store tests on `haystack.testing.document_store` classes — avoids duplicate, incomplete contract tests +- Test `to_dict()`/`from_dict()` round trips with non-default init params — preserve serializable config +- Test both sync and async integration paths — mirror gates, fixtures, inputs, and assertions +- Store fixtures in `integrations/{integration}/tests/test_files/` — keeps tests local, stable, and isolated +- Consolidate `integrations//tests/` by component — add document-store coverage to `test_document_store.py`, not one-off files +- Test only changed integration chat behavior in `integrations/*/tests/test_*chat_generator*.py` — avoids redundant live/core coverage +- Avoid routine `warm_up()` in `integrations/*/tests` init tests — call it only when asserting warm-up behavior +- Assert persisted outcomes after mutating document-store tests — catches cleanup/index regressions +- Delete unused test fixtures/helpers/setup — keeps integration tests focused and maintainable +- Test chat helper conversions directly — cover provider-specific reasoning/thinking content +- Align pytest markers in `integrations/*/pyproject.toml` — declare only used markers and set `--strict-markers` +- Test converter skip/failure paths and warnings in `integrations/*/tests/` — preserves graceful-failure behavior +- Test same-turn multi-tool calls in chat generator integrations — model them in one assistant message +- Keep test comments specific and current — preserves intent and prevents stale guidance +- Test mixed chat tools across init/runtime paths — assert merged `config.tools` and mirror sync/async coverage +- Pass explicit init args in `integrations` tests — include `model`/backend IDs to validate custom paths +- Test `close()`/reopen changes in `integrations/*/tests/test_document_store.py` — keeps lifecycle coverage consistent +- Use local `pytest` fixtures only for shared non-trivial setup — keeps tests clear and uncoupled +- Test provider streaming end-to-end in `integrations/*/tests/test_chat_generator.py` — assert every `StreamingChunk`, metadata/usage/finish field, tool-call/reasoning output, and final `ChatMessage` from realistic provider chunk sequences. +- Test only real legacy serialization formats — avoid fake shims for missing current fields +- Assert secret-backed credentials restore and resolve — prevents hidden credential bugs in tests + +### API Design + +- Use keyword-only args for optional public API params — preserves backward compatibility +- Use case-insensitive literal substrings for metadata/search filters; in IBM DB use `LOCATE(UPPER(?), UPPER(column)) > 0`, not `LIKE` — Case-normalized literal matching keeps search behavior consistent across integrations and prevents `%`/`_` wildcard bugs in IBM DB. +- Use `Secret` for sensitive integration config/API values — prevents leaked credentials +- Prefix non-public helpers in `integrations` with `_` — clarifies API boundaries +- Add async APIs only for native async I/O; mirror sync contracts and tests — Keeps async APIs non-blocking and consistent with sync behavior, preventing event-loop stalls and contract drift. +- Keep public Document Store APIs consistent across backends — preserves portability +- Expose only wired, supported public params; reject or document unsupported filters/flags — Avoids misleading no-op APIs, runtime surprises, and inconsistent integration behavior. +- Use Haystack default serialization when `init_parameters` can rebuild the component — avoids brittle custom `to_dict`/`from_dict`; require valid `init_parameters` for custom deserialization. +- Set `ToolCallDelta.index` from provider-stable call IDs — preserves chunk correlation +- Accept `meta` as `dict | list[dict] | None` for multi-`sources` converters — keeps integration metadata semantics consistent +- Merge `ByteStream.meta` into converter `Document.meta` and document it — preserves metadata +- Declare `SUPPORTED_MODELS` beside limited integration components — documents model limits +- Use backend-native bulk APIs in integration document stores — improves throughput and avoids race-prone per-document logic +- Preserve provider-native stream indices in `StreamingChunk` — don’t hardcode or reshape for helpers +- Apply filters before iterating, aggregating, or paginating docs — reuse `filter_documents(filters=filters)` to prevent query bugs +- Use `streaming_callback` for `StreamingChunk`s — don't return chunks in outputs; preserve metadata, handle unsupported chunk shapes explicitly, and test streaming/non-streaming paths +- Validate concrete backend deps in `__init__` — fail fast when objects are incompatible +- Preserve converter source provenance — use original paths or `ByteStream.meta['file_path']`, not synthetic temp filenames +- Expose reusable integrations as named importable APIs — e.g., `@component` `GitHubFileEditor` plus `GitHubFileEditorTool` +- Use `filter_policy` and `apply_filter_policy(...)` for retriever filters — avoids inconsistent merge bugs + +### Config + +- Align `integrations/*/pyproject.toml` Hatch docs config — use `haystack-pydoc pydoc/config_docusaurus.yml` and only docs/lint deps like `haystack-pydoc-tools` and `ruff` +- Populate `project.keywords` in each `integrations/*/pyproject.toml` — improves package discoverability +- Align `integrations/*/pyproject.toml` Python metadata with tested support — prevents invalid installs and stale compatibility claims +- Align `integrations//pyproject.toml` Ruff config with the shared template; put test-only ignores under `[tool.ruff.lint.per-file-ignores]` for `"tests/**/*"` — keeps integration linting consistent without weakening global rules +- Align `integrations/*/pyproject.toml` with the canonical template — ensures consistent packaging +- Set real `[project].authors` in `integrations/*/pyproject.toml` — captures true maintainer and partner ownership +- Set `description` in `integrations/*/pyproject.toml` to approved integration wording — avoids vague package metadata + +### Type System + +- Prefer type-correct code over `# type: ignore`; if needed, use exact-line `# type: ignore[code]` with a safety comment — Type-correct code avoids hiding real bugs, while targeted suppressions keep unavoidable checker limitations auditable and safe. +- Place integration `py.typed` at the exposed package boundary, e.g. `haystack_integrations/tools/py.typed` — enables correct type discovery +- Align Haystack `run()` return types with `@component.output_types(...)` and returned `dict[...]` shape — Keeps Haystack component APIs type-safe and consistent with `@component.output_types(...)`, preventing misleading nullable or overly broad contracts. +- Centralize untyped import suppressions in `integrations/*/pyproject.toml` — avoids scattered `# type: ignore[import-untyped]` +- Keep each integration’s tooling in `pyproject.toml`; include every importable package in `types` — ensures all shipped integration code is type-checked +- Use direct annotations for available types — avoid unnecessary quoted strings + +### Code Style + +- Mark state-free helper methods `@staticmethod` — clarifies no `self`/`cls` coupling +- Use structured `logger.*` placeholders in `integrations/` — pass dynamic values as kwargs +- Delete obsolete `integrations/` artifacts — stale docs, examples, deps, and configs mislead users +- Keep `integrations/**/haystack_integrations` roots/intermediates namespace-only — omit `__init__.py` unless a concrete integration package needs exports/init +- Keep sync/async `document_store.py` methods symmetrical; share filter/count/arg/error helpers — Symmetric sync/async APIs and shared helpers reduce drift, duplicated bugs, and inconsistent behavior across integration document stores. +- Initialize external clients and `None`→instance attrs in `warm_up()` — avoids lazy runtime failures + +### Dependencies + +- Set explicit minimum deps in `integrations/*/pyproject.toml`; avoid pins/upper bounds unless required — Accurate lower bounds keep integrations installable on the oldest compatible stack while avoiding unnecessary resolver conflicts and premature incompatibility with newer Haystack releases. +- Use official provider Python SDKs in `integrations` when they cover the workflow — reduces provider API bugs +- Keep `integrations/*/pyproject.toml` test deps minimal — rely on inherited deps and add only imports/tests need +- Declare only directly used runtime deps in `integrations/*/pyproject.toml` — avoids bloated installs +- Use `request_with_retry`/`async_request_with_retry` for HTTP retries — avoid custom loops and expose `timeout`/`max_retries` + +### Naming + +- Use canonical integration names everywhere — match package names in `integrations/*`, READMEs, URLs, and tables +- Name async document-store methods `_async` — keeps sync/async APIs and logs unambiguous + +### General + +- Import required deps at module top; reserve lazy/`try` imports for optional deps or cycles — Failing fast exposes missing required packages during import instead of hiding broken integrations until runtime. +- Re-export only intentional public API in `__init__.py` — preserves stable imports +- Wrap backend failures as `DocumentStoreError` — keep sync and async handling consistent, and preserve documented bulk write/delete error behaviour diff --git a/agent_docs/api-docstring-style.md b/agent_docs/api-docstring-style.md deleted file mode 100644 index aeb365f9fa..0000000000 --- a/agent_docs/api-docstring-style.md +++ /dev/null @@ -1,20 +0,0 @@ - - -# API Docstring Style - -> How to write public API and component docstrings: concise Haystack-style summaries, complete parameter/return/exception documentation, synchronized sync/async docs, performance caveats, default-value wording, inline-code markup, Markdown links, and fenced Python usage examples. - -**When to check**: When writing or updating public API, component, constructor, method, or usage-example docstrings - -## Rules - -- Use single backticks for inline code in prose; reserve double backticks for Haystack release notes — Single-backtick inline code renders consistently across docstrings and docs, while preserving the Haystack release-note exception. -- Document all public callable params; put explicit `__init__` params in the `__init__` docstring — Keeps public API docs accurate and ensures generated documentation shows constructor args in the expected place. -- Document performance caveats in API docstrings — warn about slow queries or large scans -- Write concise public docstrings and sync async variants — document real contracts, not internals -- Use single-line Markdown links in docstrings/comments — keeps docs readable and links rendering -- Document public returns with Sphinx `:returns:` — include mapping keys/types and match `@component.output_types(...)` names -- Describe `__init__` params by purpose/constraints; omit defaults already in signatures — Function signatures already expose defaults, so repeating them in docstrings creates stale documentation when defaults change. -- Document public method exceptions with concrete `:raises:` conditions — no `If ...` placeholders -- Write Haystack-style docstrings — one-line summary, blank line, then unindented sections/examples -- Use `### Usage example` and fenced ` ```python ` blocks for Python examples — avoids renderer issues diff --git a/integrations/AGENTS.md b/integrations/AGENTS.md deleted file mode 100644 index 187fec09b1..0000000000 --- a/integrations/AGENTS.md +++ /dev/null @@ -1,105 +0,0 @@ - - -# integrations/ Guidelines - -## Testing - -- Put async document-store tests in `test_document_store_async.py` — keeps sync/async coverage clear -- Parametrize duplicate `pytest` tests — merge same behavior into one test with `@pytest.mark.parametrize` -- Base document store tests on `haystack.testing.document_store` classes — avoids duplicate, incomplete contract tests -- Test `to_dict()`/`from_dict()` round trips with non-default init params — preserve serializable config -- Test both sync and async integration paths — mirror gates, fixtures, inputs, and assertions -- Store fixtures in `integrations/{integration}/tests/test_files/` — keeps tests local, stable, and isolated -- Consolidate `integrations//tests/` by component — add document-store coverage to `test_document_store.py`, not one-off files -- Test only changed integration chat behavior in `integrations/*/tests/test_*chat_generator*.py` — avoids redundant live/core coverage -- Avoid routine `warm_up()` in `integrations/*/tests` init tests — call it only when asserting warm-up behavior -- Assert persisted outcomes after mutating document-store tests — catches cleanup/index regressions -- Delete unused test fixtures/helpers/setup — keeps integration tests focused and maintainable -- Test chat helper conversions directly — cover provider-specific reasoning/thinking content -- Align pytest markers in `integrations/*/pyproject.toml` — declare only used markers and set `--strict-markers` -- Test converter skip/failure paths and warnings in `integrations/*/tests/` — preserves graceful-failure behavior -- Test same-turn multi-tool calls in chat generator integrations — model them in one assistant message -- Keep test comments specific and current — preserves intent and prevents stale guidance -- Test mixed chat tools across init/runtime paths — assert merged `config.tools` and mirror sync/async coverage -- Pass explicit init args in `integrations` tests — include `model`/backend IDs to validate custom paths -- Test `close()`/reopen changes in `integrations/*/tests/test_document_store.py` — keeps lifecycle coverage consistent -- Use local `pytest` fixtures only for shared non-trivial setup — keeps tests clear and uncoupled -- Test provider streaming end-to-end in `integrations/*/tests/test_chat_generator.py` — assert every `StreamingChunk`, metadata/usage/finish field, tool-call/reasoning output, and final `ChatMessage` from realistic provider chunk sequences. -- Test only real legacy serialization formats — avoid fake shims for missing current fields -- Assert secret-backed credentials restore and resolve — prevents hidden credential bugs in tests - -## API Design - -- Use keyword-only args for optional public API params — preserves backward compatibility -- Use case-insensitive literal substrings for metadata/search filters; in IBM DB use `LOCATE(UPPER(?), UPPER(column)) > 0`, not `LIKE` — Case-normalized literal matching keeps search behavior consistent across integrations and prevents `%`/`_` wildcard bugs in IBM DB. -- Use `Secret` for sensitive integration config/API values — prevents leaked credentials -- Prefix non-public helpers in `integrations` with `_` — clarifies API boundaries -- Add async APIs only for native async I/O; mirror sync contracts and tests — Keeps async APIs non-blocking and consistent with sync behavior, preventing event-loop stalls and contract drift. -- Keep public Document Store APIs consistent across backends — preserves portability -- Expose only wired, supported public params; reject or document unsupported filters/flags — Avoids misleading no-op APIs, runtime surprises, and inconsistent integration behavior. -- Use Haystack default serialization when `init_parameters` can rebuild the component — avoids brittle custom `to_dict`/`from_dict`; require valid `init_parameters` for custom deserialization. -- Set `ToolCallDelta.index` from provider-stable call IDs — preserves chunk correlation -- Accept `meta` as `dict | list[dict] | None` for multi-`sources` converters — keeps integration metadata semantics consistent -- Merge `ByteStream.meta` into converter `Document.meta` and document it — preserves metadata -- Declare `SUPPORTED_MODELS` beside limited integration components — documents model limits -- Use backend-native bulk APIs in integration document stores — improves throughput and avoids race-prone per-document logic -- Preserve provider-native stream indices in `StreamingChunk` — don’t hardcode or reshape for helpers -- Apply filters before iterating, aggregating, or paginating docs — reuse `filter_documents(filters=filters)` to prevent query bugs -- Use `streaming_callback` for `StreamingChunk`s — don't return chunks in outputs; preserve metadata, handle unsupported chunk shapes explicitly, and test streaming/non-streaming paths -- Validate concrete backend deps in `__init__` — fail fast when objects are incompatible -- Preserve converter source provenance — use original paths or `ByteStream.meta['file_path']`, not synthetic temp filenames -- Expose reusable integrations as named importable APIs — e.g., `@component` `GitHubFileEditor` plus `GitHubFileEditorTool` -- Use `filter_policy` and `apply_filter_policy(...)` for retriever filters — avoids inconsistent merge bugs - -## Config - -- Align `integrations/*/pyproject.toml` Hatch docs config — use `haystack-pydoc pydoc/config_docusaurus.yml` and only docs/lint deps like `haystack-pydoc-tools` and `ruff` -- Populate `project.keywords` in each `integrations/*/pyproject.toml` — improves package discoverability -- Align `integrations/*/pyproject.toml` Python metadata with tested support — prevents invalid installs and stale compatibility claims -- Align `integrations//pyproject.toml` Ruff config with the shared template; put test-only ignores under `[tool.ruff.lint.per-file-ignores]` for `"tests/**/*"` — keeps integration linting consistent without weakening global rules -- Align `integrations/*/pyproject.toml` with the canonical template — ensures consistent packaging -- Set real `[project].authors` in `integrations/*/pyproject.toml` — captures true maintainer and partner ownership -- Set `description` in `integrations/*/pyproject.toml` to approved integration wording — avoids vague package metadata - -## Type System - -- Prefer type-correct code over `# type: ignore`; if needed, use exact-line `# type: ignore[code]` with a safety comment — Type-correct code avoids hiding real bugs, while targeted suppressions keep unavoidable checker limitations auditable and safe. -- Place integration `py.typed` at the exposed package boundary, e.g. `haystack_integrations/tools/py.typed` — enables correct type discovery -- Align Haystack `run()` return types with `@component.output_types(...)` and returned `dict[...]` shape — Keeps Haystack component APIs type-safe and consistent with `@component.output_types(...)`, preventing misleading nullable or overly broad contracts. -- Centralize untyped import suppressions in `integrations/*/pyproject.toml` — avoids scattered `# type: ignore[import-untyped]` -- Keep each integration’s tooling in `pyproject.toml`; include every importable package in `types` — ensures all shipped integration code is type-checked -- Use direct annotations for available types — avoid unnecessary quoted strings - -## Code Style - -- Mark state-free helper methods `@staticmethod` — clarifies no `self`/`cls` coupling -- Use structured `logger.*` placeholders in `integrations/` — pass dynamic values as kwargs -- Delete obsolete `integrations/` artifacts — stale docs, examples, deps, and configs mislead users -- Keep `integrations/**/haystack_integrations` roots/intermediates namespace-only — omit `__init__.py` unless a concrete integration package needs exports/init -- Keep sync/async `document_store.py` methods symmetrical; share filter/count/arg/error helpers — Symmetric sync/async APIs and shared helpers reduce drift, duplicated bugs, and inconsistent behavior across integration document stores. -- Initialize external clients and `None`→instance attrs in `warm_up()` — avoids lazy runtime failures - -## Dependencies - -- Set explicit minimum deps in `integrations/*/pyproject.toml`; avoid pins/upper bounds unless required — Accurate lower bounds keep integrations installable on the oldest compatible stack while avoiding unnecessary resolver conflicts and premature incompatibility with newer Haystack releases. -- Use official provider Python SDKs in `integrations` when they cover the workflow — reduces provider API bugs -- Keep `integrations/*/pyproject.toml` test deps minimal — rely on inherited deps and add only imports/tests need -- Declare only directly used runtime deps in `integrations/*/pyproject.toml` — avoids bloated installs -- Use `request_with_retry`/`async_request_with_retry` for HTTP retries — avoid custom loops and expose `timeout`/`max_retries` - -## Naming - -- Use canonical integration names everywhere — match package names in `integrations/*`, READMEs, URLs, and tables -- Name async document-store methods `_async` — keeps sync/async APIs and logs unambiguous - -## General - -- Import required deps at module top; reserve lazy/`try` imports for optional deps or cycles — Failing fast exposes missing required packages during import instead of hiding broken integrations until runtime. -- Re-export only intentional public API in `__init__.py` — preserves stable imports -- Wrap backend failures as `DocumentStoreError` — keep sync and async handling consistent, and preserve documented bulk write/delete error behaviour - -## Topic Guides - -Check these when working in specific areas: - -- **[Integration and Component Documentation](agent_docs/integration-and-component-documentation.md)**: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts diff --git a/integrations/CLAUDE.md b/integrations/CLAUDE.md deleted file mode 100644 index f6aa6c0262..0000000000 --- a/integrations/CLAUDE.md +++ /dev/null @@ -1,3 +0,0 @@ -# CLAUDE.md - -@AGENTS.md diff --git a/integrations/agent_docs/integration-and-component-documentation.md b/integrations/agent_docs/integration-and-component-documentation.md deleted file mode 100644 index 684ba5c467..0000000000 --- a/integrations/agent_docs/integration-and-component-documentation.md +++ /dev/null @@ -1,21 +0,0 @@ - - -# Integration and Component Documentation - -> Documentation expectations for integrations and components, including minimal template-aligned integration READMEs, avoiding generated docs artifacts, documenting authentication modes, local test prerequisites, precedence rules, externally defined behavior, license placeholders, parameter requirements/defaults, realistic examples, and alignment with actual retrieval/scoring behavior. - -**When to check**: When writing or updating READMEs, docstrings, examples, authentication docs, parameter documentation, LICENSE files, or generated documentation artifacts - -## Rules - -- Keep `integrations/*/README.md` minimal and template-aligned — link to canonical docs/examples instead of duplicating long content -- Leave generated docs under `integrations/` untouched — pipelines own `CHANGELOG.md` and API docs -- Document auth by credential presence and `Secret` inputs — prevents unclear setup paths -- Document only needed local test prerequisites in `integrations/{integration}/README.md` — avoids misleading setup -- Document option precedence in `integrations` docs — list named options overridden by maps/headers -- Document model-dependent APIs with canonical links — clarify supported kwargs, models, defaults, modes, outputs, and whether lists are exhaustive -- Fill `integrations/*/LICENSE.txt` copyright fields — use real years and holders, not placeholders -- Document non-obvious public API requirements — include ranges, external limits, syntax, and examples -- Sync constructor docs, defaults, and `None` fallbacks — state env/base/upstream defaults -- Document integrations with context-complete examples — show required `Document` setup, workflow execution, and output production -- Align integration retriever docstrings with actual retrieval/scoring support — avoids misleading users