feat(mason): add code-first tool integrations

[junchoi-db]

What

• Retire agent.toml as Mason's agent/tool source of truth and replace it with an ordinary, non-executed DATABRICKS_TOOLS Python registry.
• Keep mason tools add sandbox|mcp|uc-function and tools list for Databricks-managed integrations, with exact definition/attachment line output and --framework support for BYO projects.
• Keep custom Python tools framework-native and code-first: users write them directly; Mason does not scaffold or register them through the CLI.
• Add explicit native adapters for both supported harnesses: request-aware LangGraph tool loading and lifecycle-safe OpenAI Agents SDK binding.
• Give Mason-owned templates an active empty-selection seam. Empty registries perform no credential or network work; BYO agent behavior is changed only by the explicit attachment line.

Runtime contract

# LangGraph
tools = [
    *local_tools,
    *await load_tools(DATABRICKS_TOOLS, existing_tools=local_tools),
]

# OpenAI Agents SDK
agent = await bind_tools(agent, DATABRICKS_TOOLS, stack=stack)

The adapters preserve customer-authored tools and MCP servers, enforce Sandbox downscoping, propagate account-host workspace routing, and fail closed on deterministic server/tool-name collisions. The legacy mcp_tools() API now raises migration guidance instead of silently dropping manifest selections.

For a custom Python tool, the CUJ remains ordinary framework-native code:

# agent/tools/lookup_ticket.py
from langchain_core.tools import tool


@tool
def lookup_ticket(ticket_id: str) -> str:
    """Look up one support ticket."""
    return f"Ticket {ticket_id}"

The template's all_tools() auto-discovers that file; there is no second registry entry or mason tools add python step.

Verification

Latest-main merge and review-fix verification at 5fdb7d4:

• Full Mason suite: 260 passed.
• Exact CI commands with both highest and lowest-direct dependency resolution: 260 passed each.
• Focused integration registry, tools CLI, initialization, and OpenAI binding suites: 67 passed.
• ruff check, ruff format --check, and ty check: passed.
• The OpenAI template now uses agent/databricks_tools.py and binds once in stream_handler; synchronous invocations delegate through that same path.
• TDD command-tree regression proves mason tools add exposes exactly sandbox, mcp, and uc-function.

Original implementation verification:

• Python 3.10 lowest-direct CI command: 258 passed.
• Python 3.10 highest CI command: 258 passed.
• Wheel build succeeded and contains the new integration registry plus LangGraph/OpenAI adapter modules; retired manifest modules are absent.
• LangGraph and OpenAI template construction smokes passed.

Manual dual-framework E2E (2026-09-02)

Open the HTML CUJ report
Published SHA-256: 96c81ffb1ea7d6658bc524a8aee99b31abe036abdf72fad7ad99868b7d8fcbff (dogfood round trip byte-identical).

• Ran the customer-shaped CLI journey for LangGraph and OpenAI Agents SDK: init → tools add sandbox → source verification → tools list → mason dev → real curl → mason deploy → OAuth curl.
• tools add changed only each Python Databricks integration registry. Required agent-behavior edits: 0 for both Mason-owned templates; their agent construction files remained byte-identical.
• All 6 local/managed, streaming/non-streaming HTTP paths passed after the documented premerge workarounds: each returned 200, called Sandbox run_code, observed [[21932]] for samples.nyctaxi.trips, and answered 21,932. OpenAI streams terminated with [DONE].
• Both temporary Apps, synced workspace sources, local processes/monitors, and the temporary MLflow experiment were deleted after the run.
• The clean, workaround-free CUJ remains blocked by two release issues detailed in the report: the PR wheel collides with an already-published 0.1.1.dev0 artifact that lacks the new integration module, and Mason does not resolve/provision the OpenAI template's named experiment resource for dev/deploy.

Design

https://docs.google.com/document/d/10CMZsRDribahRvUoZKcyM5uvGOUW-dTapZrAZXgyNDs/edit?tab=t.0

This pull request and its description were written by Isaac.