diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 90bd3f743..eaf69c57b 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -1,8 +1,7 @@ # Connection parameters reference -This document lists **every public connection / session parameter that is -consumed by at least one currently-supported backend** the Python connector -(`databricks.sql.connect(...)`) accepts, and — because the driver +This document lists **every recognized connection / session parameter** the Python +connector (`databricks.sql.connect(...)`) accepts, and — because the driver ships more than one backend — whether each parameter is honored on the **Thrift** backend (the default) or the **Kernel** backend (opt-in via `use_kernel=True`). @@ -14,20 +13,23 @@ but ignored (or rejected) on the other is called out in the **Note** column. > backend — a native Rust core exposed via PyO3 — is selected with > `use_kernel=True`; it requires **Python ≥ 3.10** and the > `databricks-sql-connector[kernel]` extra, and is **early access** (its -> parameter surface is still landing and may change without notice). Crucially, -> the session layer forwards only a *curated, named subset* of parameters to +> parameter surface is still landing and may change without notice). The +> session layer forwards only a *curated, named subset* of backend options to > the kernel (`Session._create_backend`, `src/databricks/sql/session.py`) — it -> does **not** splat `**kwargs` — so anything outside that subset is silently -> ignored on the kernel path. That is why many rows below are ❌ for Kernel. +> does **not** splat `**kwargs`. Options implemented in the shared Python driver +> layer can still apply to both backends; unforwarded backend-specific options +> are ignored or rejected on the kernel path. > > A separate pure-Python **SEA** backend (`use_sea=True`) also exists but is -> being deprecated, so it is intentionally omitted from this reference. +> being deprecated. Its selector and SEA-only hybrid-disposition option are +> listed for completeness, but this reference does not provide a third SEA +> support column. ## Legend | Symbol | Meaning | | ------ | ------------------------------------------------------------------- | -| ✅ | Honored — the option is read and forwarded to the backend. | +| ✅ | Honored by the backend or by the shared driver layer. | | ❌ | Ignored or rejected — see the Note column. | | ⚠️ | Partially supported or behaves differently from the other backend. | | — | Not applicable / no public equivalent / no default on this backend. | @@ -54,6 +56,17 @@ to change without notice. --- +## Backend selection + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ------------------------ | ------ | :----: | :----: | ------------- | -------------------------------------------------------------------------------------------------- | +| `use_kernel` | `bool` | — | — | `False` | Routing selector: `False` selects Thrift; `True` selects Kernel. Mutually exclusive with `use_sea`. | +| `use_sea` | `bool` | — | — | `False` | Select the deprecated pure-Python SEA backend. Mutually exclusive with `use_kernel`. | +| `use_hybrid_disposition` | `bool` | ❌ | ❌ | `False` | SEA-only result-disposition option; ignored by Thrift and Kernel. | +| `max_connections` | `int` | ❌ | ❌ | `10` | Internal SEA-only HTTP-pool limit; ignored by Thrift and Kernel. | + +--- + ## Connection identity | Option | Type | Thrift | Kernel | Default Value | Note | @@ -62,15 +75,16 @@ to change without notice. | `http_path` | `str` | ✅ | ✅ | — (required) | Thrift accepts a SQL-warehouse path **or** an all-purpose-cluster path; Kernel requires a warehouse/endpoint path. | | `_port` | `int` | ✅ | — | `443` | TCP port (advanced). Not threaded to the kernel; it derives host/port from `server_hostname` + `http_path`. | | `_connection_uri` | `str` | ✅ | — | `None` | Thrift-only internal override of `server_hostname`/`http_path`. No kernel equivalent. | +| `http_headers` | `List[Tuple[str, str]]` | ✅ | ⚠️ | `None` | Extra headers sent on requests. Kernel filters `Authorization` and `x-databricks-org-id` because it manages them itself; other headers are forwarded. | | `user_agent_entry`| `str` | ✅ | ✅ | `None` | Custom tag folded into the composed `User-Agent` on both. (`_user_agent_entry` is a deprecated alias that warns.) | ## Authentication | Option | Type | Thrift | Kernel | Default Value | Note | | --------------------------------------------------- | -------------------- | :----: | :----: | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `access_token` (PAT) | `str` | ✅ | ✅ | `None` | Personal Access Token / bearer token. The default auth mode when set; otherwise auth falls back to OAuth. | -| `auth_type` | `str` | ✅ | ✅ | `None` ⇒ Databricks OAuth | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three are supported on the kernel path (#919): `azure-oauth` routes to the kernel's OAuth U2M flow identically to `databricks-oauth` (the in-house workspace-federated browser flow, which Azure workspaces support), so it uses the `databricks-sql-python` app rather than Thrift's direct-Entra Azure app. All three work on the Thrift path. | -| `oauth_client_id` (U2M) | `str` | ✅ | ✅ | built-in client id | Custom U2M client id. Forwarded on both; when absent, each path applies its own built-in default. | +| `access_token` (PAT) | `str` | ✅ | ✅ | `None` | Personal Access Token / bearer token. When no auth signal is supplied, Thrift falls back to Databricks OAuth U2M; Kernel requires an explicit U2M `auth_type` or another supported credential flow. | +| `auth_type` | `str` | ✅ | ✅ | `None` | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three work on both backends. Thrift treats an otherwise credential-less `None` as Databricks OAuth; Kernel does not implicitly select U2M. On Kernel, `azure-oauth` uses the same workspace-federated browser flow as `databricks-oauth`. | +| `oauth_client_id` (OAuth) | `str` | ✅ | ✅ | built-in id for U2M | Custom U2M client id on both backends. Kernel also uses it with `oauth_client_secret` or the JWT options for M2M; those M2M flows have no built-in client-id default. | | `oauth_redirect_port` (U2M) | `int` | ✅ | ✅ | `None` | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) | | `oauth_client_secret` (OAuth M2M) | `str` | ❌ | ✅ | `None` | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. | | `oauth_jwt_key_file` (OAuth M2M, JWT private key) | `str` | ❌ | ✅ | `None` | **Kernel-only.** Path to the PEM private key for JWT private-key M2M (RFC 7523 client assertion). Supplying it selects the JWT flow: the kernel signs a short-lived assertion with the key instead of sending a client secret. Requires `oauth_client_id` + `oauth_jwt_kid`; mutually exclusive with `oauth_client_secret` / `credentials_provider`. | @@ -78,7 +92,7 @@ to change without notice. | `oauth_jwt_passphrase` (OAuth M2M, JWT private key) | `str` | ❌ | ✅ | `None` | **Kernel-only.** Passphrase for an encrypted PKCS#8 private key; omit for an unencrypted key. | | `oauth_jwt_algorithm` (OAuth M2M, JWT private key) | `str` | ❌ | ✅ | `RS256` | **Kernel-only.** JWT signing algorithm (`RS256`/`384`/`512`, `PS256`/`384`/`512`, `ES256`, `ES384`). | | `token_url` (OAuth M2M) | `str` | ❌ | ✅ | `None` ⇒ OIDC discovery | **Kernel-only.** OAuth IdP token endpoint override. Required for JWT M2M against an external-IdP-backed workspace (e.g. Entra ID for Azure Databricks), since Databricks-native OIDC does not advertise the `private_key_jwt` method. Applies to shared-secret M2M too. | -| `oauth_scopes` | `List[str]` | ❌ | ✅ | `["sql","offline_access"]`| **Thrift ignores custom scopes** — it always uses the built-in scope set. Only the kernel honors a custom `oauth_scopes`. | +| `oauth_scopes` | `List[str]` | ❌ | ✅ | `["sql","offline_access"]` for U2M | **Thrift ignores custom scopes** — it always uses the built-in scope set. Kernel honors custom scopes for U2M and M2M; an omitted M2M value uses the kernel's own default. | | `credentials_provider` | `CredentialsProvider`| ✅ | ❌ | `None` | Custom external credentials provider. **Rejected on the kernel path** (`NotSupportedError`) — it is an opaque token source, so the kernel cannot own the token lifecycle; use `oauth_client_id` + `oauth_client_secret` for M2M, or the Thrift backend. | | `identity_federation_client_id` | `str` | ✅ | ✅ | `None` | Workload identity / token-federation client id (kernel support added in #910). | | `experimental_oauth_persistence` | `OAuthPersistence` | ✅ | ❌ | `None` | **Thrift-only.** The kernel owns its own token lifecycle and does not accept a persistence store. | @@ -91,7 +105,7 @@ to change without notice. ## HTTP client, proxy, retries > **Retry defaults are Thrift defaults.** The **Default Value** column lists the -> values the *Thrift* backend applies. For the ✅-Kernel retry rows +> values the *Thrift* backend applies. For the Kernel-supported retry rows > (`_retry_stop_after_attempts_count` / `_duration`, `_retry_delay_min` / > `_max`), `session.py` forwards each as `kwargs.get(...)` **with no fallback**, > so when a caller omits one, `None` is passed and the kernel's Rust retry @@ -101,13 +115,13 @@ to change without notice. | Option | Type | Thrift | Kernel | Default Value | Note | | ------------------------------------ | ----------- | :----: | :----: | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `_socket_timeout` | `float` (s) | ✅ | ✅ | `900` (Thrift); `120` (kernel) | Thrift: socket send/recv/connect timeout. Kernel: total HTTP request deadline from connect through response-body completion. A positive value is forwarded; unset or `0` selects the kernel's 120s default. On the kernel path, `0` is neither unlimited nor an immediate timeout. | -| `_pool_connections` | `int` | ✅ | ⚠️ | `10` | Number of urllib3 connection pools. Configures the connector's shared Python HTTP client; the kernel's query transport is its own Rust stack. | -| `_pool_maxsize` | `int` | ✅ | ⚠️ | `20` (Thrift); `100` (kernel when unset) | Max idle connections retained per host. A positive value always configures the shared Python HTTP client and also configures the kernel's Rust HTTP pool with kernel ≥ 1.1.0. Unset or `0` keeps each client's default. | -| `_proxy_auth_method` | `str` | ✅ | ⚠️ | `None` | `basic` or `negotiate` (Kerberos). Applies to the shared Python HTTP client; not threaded to the kernel query transport. See [`docs/proxy.md`](docs/proxy.md). | +| `_pool_connections` | `int` | ⚠️ | ❌ | `10` | Number of pools in the shared Python HTTP client. On Thrift this affects auxiliary auth/feature-flag/telemetry traffic, not the separate Thrift query transport. Kernel-owned network traffic does not use this client. | +| `_pool_maxsize` | `int` | ⚠️ | ⚠️ | `20` (Python); `100` (kernel when unset) | Max idle connections retained per host. It configures the shared Python client but not the Thrift query transport. With kernel ≥ 1.1.0, a positive value also configures the kernel's Rust HTTP pool. Unset or `0` keeps each client's default. | +| `_proxy_auth_method` | `str` | ✅ | ❌ | `None` | `basic` or `negotiate` (Kerberos). Applied to both Thrift query traffic and the shared Python client. It is not forwarded to the kernel's Rust transport. See [`docs/proxy.md`](docs/proxy.md). | | `_retry_stop_after_attempts_count` | `int` | ✅ | ✅ | `30` | Max attempts in a retry sequence. Bounded to `[1, 60]` on Thrift; forwarded to the kernel's retry policy. | -| `_retry_stop_after_attempts_duration`| `float` (s) | ✅ | ✅ | `900` | Max total wall-clock seconds spent retrying. Forwarded to the kernel. | -| `_retry_delay_min` | `float` (s) | ✅ | ✅ | `1` | Minimum backoff delay. Forwarded to the kernel. | -| `_retry_delay_max` | `float` (s) | ✅ | ✅ | `60` | Maximum backoff delay. Forwarded to the kernel. | +| `_retry_stop_after_attempts_duration`| `float` (s) | ✅ | ⚠️ | `900` | Max total wall-clock seconds spent retrying. Kernel rounds to whole seconds. | +| `_retry_delay_min` | `float` (s) | ✅ | ⚠️ | `1` | Minimum backoff delay. Kernel rounds to whole seconds and floors a positive sub-second value at one second. | +| `_retry_delay_max` | `float` (s) | ✅ | ⚠️ | `60` | Maximum backoff delay. Kernel rounds to whole seconds. | | `_retry_delay_default` | `float` (s) | ✅ | ❌ | `5` | Delay used when a poll fails due to a TCP/OS error. Not forwarded — the kernel's backoff has no flat-default equivalent. | | `_retry_dangerous_codes` | `List[int]` | ✅ | ❌ | `[]` | HTTP status codes for which even non-idempotent commands are retried. Thrift-only. | | `_respect_server_retry_after_header` | `bool` | ✅ | ❌ | `False` | Honor the server's `Retry-After` header. Thrift-only. | @@ -150,28 +164,30 @@ to change without notice. | `session_configuration` | `Dict[str, Any]` | ✅ | ✅ | `None` | Spark/SQL session parameters (e.g. `{"ansi_mode": "true"}`). Delivered via `open_session` on both backends. | | `catalog` | `str` | ✅ | ✅ | `None` | Initial catalog for the session (DBR 9.0+). | | `schema` | `str` | ✅ | ✅ | `None` | Initial schema for the session (DBR 9.0+). | -| `query_tags` | `Dict[str, Optional[str]]` | ✅ | ✅ | `None` | Key/value tags serialized into the reserved `QUERY_TAGS` conf. On the kernel path they are applied per statement via `set_query_tags`. | +| `query_tags` | `Dict[str, Optional[str]]` | ✅ | ✅ | `None` | Connection-level key/value tags serialized into the reserved `QUERY_TAGS` session configuration on both backends. Cursor-level `query_tags` are applied per statement separately. | | `enable_metric_view_metadata` | `bool` | ✅ | ✅ | `False` | Sets `spark.sql.thriftserver.metadata.metricview.enabled` via session config so metric-view metadata surfaces. | -| `use_inline_params` | `bool` \| `"silent"` | ✅ | ⚠️ | `False` | Render parameters inline (legacy) vs. native bound params (DBR 14.1+). The kernel uses native binding; inline may differ. | +| `use_inline_params` | `bool` \| `"silent"` | ✅ | ✅ | `False` | Render parameters inline (legacy) vs. native bound params (DBR 14.1+). Shared cursor code performs the rendering before dispatch, so both backends honor it identically. | | `ignore_transactions` | `bool` | ✅ | ✅ | `True` | When `True`: `commit()` is a no-op, `rollback()` raises `NotSupportedError`, and setting `autocommit` is a no-op. | | `fetch_autocommit_from_server`| `bool` | ✅ | ✅ | `False` | Query the server (`SET AUTOCOMMIT`) for autocommit state instead of returning the cached value. | | `staging_allowed_local_path` | `str` \| `List[str]` | ✅ | ❌ | `None` | Local path(s) permitted for Unity Catalog Volume `PUT`/`GET`. **Thrift-only** — the kernel has no Volume API yet. | ## Telemetry -All `*telemetry*` options live in the driver layer (the shared HTTP client and -`TelemetryClientFactory`), not inside either backend, so they are read -regardless of `use_kernel`. +Thrift telemetry is owned by the Python driver and normally gated by the +`enableTelemetryForPythonDriver` server feature flag. Kernel telemetry is owned +by the Rust kernel: explicit `enable_telemetry`, `telemetry_batch_size`, and +`_telemetry_circuit_breaker_enabled` values are forwarded when supported by the +installed kernel binding. `force_enable_telemetry` remains Python-driver-only. | Option | Type | Thrift | Kernel | Default Value | Note | | ---------------------------------- | ------ | :----: | :----: | ------------- | -------------------------------------------------------- | -| `enable_telemetry` | `bool` | ✅ | ✅ | `True` | Enable client telemetry (also gated by a server flag). | -| `force_enable_telemetry` | `bool` | ✅ | ✅ | `False` | Force telemetry on regardless of the server-side flag. | +| `enable_telemetry` | `bool` | ✅ | ✅ | `True` (Thrift); kernel-defined when unset | Thrift is also gated by the server flag. An explicit Kernel value is forwarded without that Python-side gate. | +| `force_enable_telemetry` | `bool` | ✅ | ❌ | `False` | Bypass the server-side feature flag on Thrift. Not forwarded to Kernel. | | `telemetry_batch_size` | `int` | ✅ | ✅ | `100` | Events buffered before a flush. | -| `_telemetry_circuit_breaker_enabled`| `bool`| ✅ | ✅ | `True` | Enable the telemetry circuit breaker. | +| `_telemetry_circuit_breaker_enabled`| `bool`| ✅ | ✅ | effective `False` (Thrift); kernel-defined when unset | The `ClientContext` signature defaults to `True`, but `connect()` passes `None` when this option is omitted and `bool(None)` makes the effective Thrift default `False`. An explicit value is forwarded to Kernel. | -> **Telemetry _events_ differ by backend.** The knobs above are backend-agnostic, -> but because the kernel owns result fetching internally it emits fewer +> **Telemetry _events_ differ by backend.** Because the kernel owns result +> fetching and telemetry internally, it emits fewer > per-statement / CloudFetch telemetry events than the Thrift path. --- @@ -182,34 +198,46 @@ regardless of `use_kernel`. 1. Custom `credentials_provider` (rejected on the kernel path). 2. `experimental_oauth_persistence` (custom OAuth token store). -3. Azure service-principal / Entra ID OAuth (`azure_client_id`, - `azure_client_secret`, `azure_tenant_id`, `azure_workspace_resource_id`). -4. TLS-client-cert *authentication* (`_use_cert_as_auth`) — note the TLS +3. TLS-client-cert *authentication* (`_use_cert_as_auth`) — note the TLS *transport* options (`_tls_*`) themselves **are** honored on both backends. -5. Result-transport tuning: `use_cloud_fetch`, `max_download_threads`, +4. Result-transport tuning: `use_cloud_fetch`, `max_download_threads`, `enable_query_result_lz4_compression`. -6. Arrow-native rendering for `_use_arrow_native_decimals` / +5. Arrow-native rendering for `_use_arrow_native_decimals` / `_use_arrow_native_timestamps` (complex types **are** forwarded). -7. `staging_allowed_local_path` (Volume `PUT`/`GET`). -8. Retry fine-tuning beyond the four forwarded knobs: `_retry_delay_default`, +6. `staging_allowed_local_path` (Volume `PUT`/`GET`). +7. Retry fine-tuning beyond the four forwarded knobs: `_retry_delay_default`, `_retry_dangerous_codes`, `_respect_server_retry_after_header`, `_retry_max_redirects`, `_enable_v3_retries`. -9. `_socket_timeout`, `_port`, `_connection_uri`. +8. `force_enable_telemetry`. +9. `_port` and `_connection_uri`. +10. `_pool_connections` and `_proxy_auth_method` do not configure the + kernel-owned network stack. -### Supported on Kernel, no Thrift public equivalent +### Supported on Kernel, missing / ignored on Thrift -None — the kernel's parameter surface is currently a subset of Thrift's. +1. Kernel-managed OAuth M2M: `oauth_client_secret`, `oauth_jwt_key_file`, + `oauth_jwt_kid`, `oauth_jwt_passphrase`, `oauth_jwt_algorithm`, and + `token_url`. +2. Custom `oauth_scopes`. +3. Kernel U2M encrypted token storage (`oauth_token_cache_enabled`). ### Behavioral divergences to watch -- **Connection pooling / proxy**: `_pool_connections` and `_proxy_auth_method` - configure only the shared Python HTTP client. `_pool_maxsize` also configures - the kernel's Rust HTTP pool when positive and using kernel ≥ 1.1.0. A value of - `0` is treated as unset: the shared client keeps 20 and the kernel keeps 100. -- **`use_inline_params`** renders parameters inline on Thrift; the kernel uses - native parameter binding. +- **Authentication default**: a credential-less Thrift connection defaults to + Databricks OAuth U2M. Kernel requires an explicit U2M `auth_type` or another + supported credential flow. +- **Connection pooling / proxy**: `_pool_connections` configures only the + shared Python client and `_proxy_auth_method` is not forwarded to Kernel. + `_pool_maxsize` also configures the kernel's Rust HTTP pool when positive and + using kernel ≥ 1.1.0. A value of `0` is treated as unset: the Python client + keeps 20 and the kernel keeps 100. +- **Custom headers**: Kernel drops `Authorization` and + `x-databricks-org-id`; it forwards other `http_headers`. +- **Telemetry**: Kernel owns its telemetry lifecycle. It does not honor + `force_enable_telemetry`, and an omitted `enable_telemetry` or circuit-breaker + setting uses the kernel's default. > All kernel-path behavior reflects the current **early-access** surface and is -> subject to change. Generated against connector version `4.4.0`; the source of +> subject to change. Generated against connector version `4.5.0`; the source of > truth is `Connection.__init__` (`src/databricks/sql/client.py`) and the > per-backend clients under `src/databricks/sql/backend/`.