Merge mindsdb/engine into fork (catch-up, 26.0.1 → 26.2.0)

[gabrielbressan-tfy]

Why this PR

Our fork was tracking mindsdb/mindsdb, but that repo has been repurposed into the "Minds Platform" product (a different codebase: backend/ + a frontend submodule). The actual Python query engine we forked now lives at mindsdb/engine. The two still share full git history, so this is an ordinary catch-up merge — not a manual port.

This PR re-points upstream to mindsdb/engine and merges ~330 engine commits into our fork, reconciling them with all of our custom work (handlers, planner/select fixes).

• Version: 26.0.1 → 26.2.0
• Merged commits: ~330 from mindsdb/engine
• Merge base: aeca4da7e (shared ancestor — clean 3-way merge)

Going forward, syncing is just git fetch upstream && git merge upstream/main (upstream = engine now). The hard reconciliation is baked into history with this PR.

---

What we kept (our work — fully preserved)

Custom handlers (engine does not ship these; they stay in-tree):
google_analytics · github · gmail · google_calendar · google_search · hubspot · bigquery · s3 · s3vectors · ms_one_drive · litellm · langsmith · xero · linkedin_ads · meta_ad_library · microsoft_ads · multi_format_api · sentry

Custom planner / select behavior (documented in CLAUDE.md):

• JOIN column collection includes WHERE-clause columns
• item.conditions filter exclusion + IS NULL guard
• CTE clearing after plan_cte
• order_by not forwarded to API handlers
• applied-WHERE-column stripping in SubSelectStep
• _collect_identifiers recursive column extraction

Framework stack: kept langchain (our RAG/reranker depend on it via the MindsDB abstraction). Engine has migrated away from langchain — see dependency notes below.

---

What comes with the upgrade (engine 26.0.1 → 26.2.0)

Architecture change — community-handler model

Engine moved ~140 standard handlers (most DB connectors + niche APIs: slack, twitter, youtube, mongodb, elasticsearch, clickhouse, etc.) out of the core tree into an on-demand fetch model:

• New module mindsdb/integrations/utilities/community_handler_fetcher.py
• Off by default — gated by MINDSDB_COMMUNITY_HANDLERS=true
• When enabled: fetches an index from a separate repo (mindsdb/mindsdb-community-handlers) and lazily downloads each handler on first use into a local storage dir
• ⚠️ That repo appears to be private currently (uses GITHUB_TOKEN), so those handlers may not be fetchable without access. If we need one of the dropped handlers, restore it from git history rather than relying on the fetcher.
• In-tree handler count: 302 (old main) → 41 (engine core) → 68 (engine core + our 28 custom/kept)

Security fixes

• safe_extract path-traversal hardening (security(utilities): harden path traversal validation in safe_extract mindsdb/minds#12347, Bugfix/files safe extract mindsdb/minds#12010)
• Dockerfile base-image security updates (Applies security updates to Dockerfile base image mindsdb/minds#12381)
• Dependency security fixes (Fix security issues in deps mindsdb/minds#12334); protobuf 4.24.4 → 6.33.5 (Snyk)
• MCP OAuth fix in Docker (fix mcp oauth in docker mindsdb/minds#12397)

Features / improvements

• KB: litellm replaced in knowledge base (Litellm replace in knowledge base mindsdb/minds#12354, #4e9d9746d); KB creation works without explicit provider configs for non-OpenAI (Fix: Create Knowledge base without explicitly providing provider configs is successful for non-openAI scenarios mindsdb/minds#11975); added KB unit tests (KB unit tests mindsdb/minds#12321)
• Agents: AG2 multi-agent handler (feat: add AG2 multi-agent handler mindsdb/minds#12352); MCP bumped 1.10 → 1.26
• Planner: optimize_inner_join — batches LIMITed inner-join fetches into partitioned steps (restored on top of our JOIN fixes)
• New / updated handlers in engine core: Jira updates ([Feature]Jira Handler updates mindsdb/minds#11951), Freshdesk (Feature - Freshdesk Integration mindsdb/minds#11683), Ollama temperature param (feat(ollama): add support for temperature parameter  mindsdb/minds#12114)
• DB tree view shows all schemas by default (feat: Show all schemas by default in database tree view mindsdb/minds#11234)

Dependency bumps (came with engine; a few adjusted for our stack)

Package	Old	New
flask / werkzeug	3.0.3 / 3.0.6	3.1.3 / 3.1.6
pandas	2.2.3	2.3.1
cryptography	>=35.0	>=46.0.5
redis	5.x	6.4.0
protobuf	4.24.4	6.33.5
mind-castle	>=0.4.9	0.5.0
pyjwt / orjson / python-multipart	—	bumped
added	—	xlrd, aiobotocore, google-genai, faiss-cpu (KB default)

---

Decisions made during the merge

Area	Decision
pydantic-ai	Engine bumped to ==1.77.0 (needs openai>=2.29), which conflicts with our langchain-openai==0.3.6 (openai<2.0). Engine dropped langchain; we depend on it. Pinned pydantic-ai>=0.0.14 → resolves to 1.30.1 (openai-1.x compatible). Kept our duckdb==1.4.0, aipdf==0.0.6.3.
Response types	Our DataHubResponse (inherited from old-upstream PRs mindsdb#10716/mindsdb#10632, never our own work) was removed — engine never had it. Switched to engine's OkResponse/TableResponse/DataHandlerResponse.
plan_join.py	Kept our JOIN fixes and restored engine's optimize_inner_join (they only collided textually). Adopted engine's check_use_limit (ours had regressed it — dropped the feature and had an or/and precedence bug).
bigquery_handler	Merged both sides: our usage-metering + include/exclude tables and engine's dataset_project, service_account_json, LEFT-JOIN fix.
integrations.py	Adopted engine's community-handler loading (our in-tree handlers register via _register_handler_dir); preserved our HandlersCache cross-thread fix.
vectordatabase_handler	Kept our update()/hybrid_search() methods; switched to engine's response types.
hubspot	Kept ours wholesale (our OAuth2 rewrite; engine's changes were cosmetic).

Bugs fixed during integration

• insert_step.py read response.data_frame on an OkResponse → AttributeError on KB insert. Aligned to engine's behavior (return ResultSet(affected_rows=...)). Behavioral note: KB insert now returns affected-row count only, not the inserted rows.
• Handler-restore gotcha: git checkout --ours only restored modified files in the deleted handler dirs; unmodified __init__.py/__about__.py/icon/tests/ were silently dropped. Restored full dirs from history for github, gmail, google_analytics, google_calendar, google_search, litellm, ms_one_drive, s3.

Testing

• ✅ python -m compileall mindsdb — clean (no syntax errors)
• ✅ Dependency resolution validated with uv pip compile
• ✅ Image rebuilt; container boots clean (HTTP API on 47334, no import/startup errors)
• ✅ KB insert, custom handlers, and RAG/reranker verified working in the dev container

Notes / follow-ups

• The dependabot alerts on main are pre-existing, not introduced here (this PR actually pulls engine's security bumps).
• Longer term we've diverged from engine at the framework level (langchain vs pydantic-ai); not blocking, but worth tracking.

[gabrielbressan-tfy]

Eu testei localmente aqui

• Insert e query em Knowledge Base via file upload no mktplace
• Query nos datasources: BigQuery, S3, Google Ads, Shopify e Xero