chore(release): prepare v0.70.0 release, documentation overhaul, and adapter refinements

.github/workflows/ci.yml

@@ -65,7 +65,7 @@ jobs:
                   docs/*|zensical.toml|CONTRIBUTING.md|tools/docs/*|.github/workflows/docs.yml)
                     docs_required=true
                     ;;
-                  sqlspec/*|pyproject.toml|uv.lock|.github/workflows/ci.yml)
+                  sqlspec/*|pyproject.toml|uv.lock|Makefile|.github/workflows/ci.yml)
                     ci_required=true
                     docs_required=true
                     ;;
@@ -548,12 +548,13 @@ jobs:
         run: uv python install 3.12
 
       - name: Install dependencies
-        run: uv sync --group doc
+        run: uv sync --no-default-groups --group doc
 
       - name: Verify public docstrings and CLI reference
         run: |
           uv run --no-sync python -m tools.docs.scan_public_docstrings --check-leaks --check-overloads --check-typeddicts sqlspec
           uv run --no-sync python -m tools.docs.generate_cli_docs --check
+          uv run --no-sync python -m tools.docs.generate_data_dictionary_docs --check
 
       - name: Build docs
         run: make docs

.github/workflows/docs.yml

@@ -36,7 +36,7 @@ jobs:
         run: uv python install 3.12
 
       - name: Install dependencies
-        run: uv sync --group doc
+        run: uv sync --no-default-groups --group doc
 
       - name: Build documentation
         run: make docs

Makefile

@@ -326,23 +326,18 @@ docs: docs-clean                                   ## Build documentation
 	@echo "${INFO} Building documentation... 📝"
 	@uv run --no-sync python -m tools.docs.scan_public_docstrings --check-leaks --check-overloads --check-typeddicts sqlspec
 	@uv run --no-sync python -m tools.docs.generate_cli_docs --check
+	@uv run --no-sync python -m tools.docs.generate_data_dictionary_docs --check
 	@uv run --no-sync zensical build --clean
 	@uv run --no-sync python -m tools.docs.postbuild --config zensical.toml --docs-dir docs --site-dir site
 	@uv run --no-sync python -m tools.docs.verify_docs_artifacts --site-dir site
 	@echo "${OK} Documentation built successfully"
 
-.PHONY: docs-check
-docs-check: docs                                   ## Build and verify documentation artifacts
-
 .PHONY: docs-linkcheck
 docs-linkcheck:                                    ## Check documentation links and anchors
 	@echo "${INFO} Checking documentation links... 🔗"
 	@uv run --no-sync python -m tools.docs.verify_docs_artifacts --site-dir site
 	@echo "${OK} Link check complete"
 
-.PHONY: docs-linkcheck-full
-docs-linkcheck-full: docs-linkcheck                ## Run full documentation link check
-
 .PHONY: docs-demos
 docs-demos:                                        ## Generate CLI demo GIFs from VHS tapes
 	@if ! command -v vhs >/dev/null 2>&1; then \

README.md

@@ -82,7 +82,7 @@ with spec.provide_session(config) as session:
 | Area | Overview |
 | --- | --- |
 | **Unified Execution** | Consistent `execute`, `select`, `select_one`, `select_one_or_none`, `select_value`, `execute_many`, `select_stream`, and transaction APIs across sync and `asyncio` drivers. |
-| **Database Adapters** | PostgreSQL (`asyncpg`, `psycopg`, `psqlpy`), CockroachDB, SQLite (`sqlite3`, `aiosqlite`), DuckDB, MySQL & MariaDB, Oracle, SQL Server, Db2, BigQuery, Cloud Spanner, ADBC, and Arrow ODBC. |
+| **Database Adapters** | PostgreSQL (`asyncpg`, `psycopg`, `psqlpy`), CockroachDB (`asyncpg`, `psycopg`), SQLite (`sqlite3`, `aiosqlite`), MySQL & MariaDB (`asyncmy`, `aiomysql`, `pymysql`, `mysql-connector-python`), Oracle (`oracledb`), SQL Server (`mssql-python`, `pymssql`), Sybase ASE (`pymssql`), SAP SQL Anywhere (`sqlanydb`), IBM Db2 (`ibm_db`), DuckDB (`duckdb`), BigQuery (`google-cloud-bigquery`), Cloud Spanner (`google-cloud-spanner`), ADBC (`adbc-driver-manager`), and Arrow ODBC (`arrow-odbc`). |
 | **SQL Files & Builder** | Load named queries from `.sql` files via `SQLFileLoader` or build statements with `sql.select`, `sql.insert`, `sql.update`, `sql.delete`, and `sql.merge` with cached AST compilation. |
 | **Typed Rows & Arrow** | Project rows into `msgspec.Struct`, `Pydantic`, `dataclass`, `TypedDict`, or `attrs` models, or stream `pyarrow`, `polars`, and `pandas` datasets across local, `fsspec`, and `obstore` backends. |
 | **Integrations & Ops** | First-party plugins and stores for Litestar, Google ADK, FastAPI, Starlette, Flask, Sanic, Dishka, and Events, plus OpenTelemetry spans, Prometheus metrics, and CLI migrations. |

docs/changelog.md

@@ -7,6 +7,44 @@ important operational fixes.
 
 ## Recent Updates
 
+### v0.70.0 - Sybase ASE and SAP SQL Anywhere adapters, streaming Arrow ingestion, canonical DDL synthesis, sharded fixtures, and documentation overhaul
+
+**Added:**
+
+* Added Sybase Adaptive Server Enterprise (`sqlspec.adapters.sybase` with `SybaseSyncConfig` and `SybaseSyncDriver`) and SAP SQL Anywhere (`sqlspec.adapters.sqlanywhere` with `SqlAnywhereSyncConfig` and `SqlAnywhereSyncDriver`) database adapters, custom SQLGlot dialects (`Sybase` and `SQLAnywhere` in `sqlspec.dialects.sybase`), data dictionaries (`SybaseSyncDataDictionary` and `SqlAnywhereSyncDataDictionary` with `SYBASE_CONFIG` and `SQLANYWHERE_CONFIG`), type converters (`SybaseOutputConverter` and `SqlAnywhereOutputConverter`), and extension stores for Litestar sessions (`SybaseStore`, `SqlAnywhereStore`), Event queues (`SybaseEventQueueStore`, `SqlAnywhereEventQueueStore`), and Google ADK sessions and memory (`SybaseADKStore`, `SybaseADKMemoryStore`, `SqlAnywhereADKStore`, `SqlAnywhereADKMemoryStore`).
+  ([#870](https://github.com/litestar-org/sqlspec/pull/870))
+* Added `arrow_odbc` dialect auto-detection and `information_schema` / system catalog introspection for Sybase ASE (`Adaptive Server Enterprise` / `ASE`) and SAP SQL Anywhere (`SQL Anywhere` / `Sybase IQ`), routing detected connections to the `sybase` and `sqlanywhere` data dictionaries.
+  ([#870](https://github.com/litestar-org/sqlspec/pull/870))
+* Added streaming Apache Arrow batch ingestion (`iter_arrow_batches`), PostgreSQL binary `COPY` framing (`encode_arrow_batch_pg_binary`, `iter_pgcopy_binary_chunks`, `iter_pgcopy_binary_chunks_async`), staged `MERGE` / `ON CONFLICT` upserts (`execute_staged_load_sync`, `execute_staged_load_async`) across `asyncpg`, `psycopg`, `psqlpy`, `oracledb`, `duckdb`, `sqlite`, `aiosqlite`, `bigquery`, `spanner`, `mssql_python`, `pymssql`, `db2`, and MySQL adapters, and bulk-load secondary index drop/recreate lifecycle helpers (`disable_secondary_indexes`, `rebuild_secondary_indexes`) on `SyncDriverAdapterBase` and `AsyncDriverAdapterBase`.
+  ([#868](https://github.com/litestar-org/sqlspec/pull/868))
+* Added canonical DDL squashing (`MigrationSquasher.fold_canonical_ddl`, `sqlspec squash --canonical`), live database schema dumping (`MigrationCommands.dump_schema`, `sqlspec dump-schema` with `--dialect`, `--schema`, `--include-domains`, `--exclude-domains`, `--exclude-tables`, and `--create-migration`), and data-driven DDL synthesis (`CreateTable.from_metadata`, `CreateTable.from_arrow_schema`, `CreateTable.from_records`, `CreateIndex.from_metadata`, `compile_table_ddl_results`, `get_schema_ddl`, and `profile_table`).
+  ([#862](https://github.com/litestar-org/sqlspec/pull/862))
+* Added sharded fixture loading (`find_fixture_files`, `load_fixture`, `iter_fixture_records`, `write_fixture_shards`), vector embedding column serialization across PostgreSQL (`pgvector`), Oracle (`VECTOR`), DuckDB, BigQuery, Spanner, and SQLite, and cross-adapter primary key sequence resynchronization (`sync_table_sequences`, `sync_table_sequences_async`) after explicit-ID fixture loads.
+  ([#863](https://github.com/litestar-org/sqlspec/pull/863))
+* Added generated Data Dictionary dialect and SQL query reference documentation (`docs/reference/data-dictionary/dialects/` via `tools.docs.generate_data_dictionary_docs`) covering all 13 dialect packs (`bigquery`, `cockroach`, `db2`, `duckdb`, `mssql`, `mysql` / `mariadb`, `oracle`, `postgres`, `spanner`, `sqlanywhere`, `sqlite`, and `sybase`), 214 SQL query files, and 336 named catalog queries.
+
+**Changed:**
+
+* Replaced correlated per-column scalar subqueries in data dictionary `columns.sql` and `indexes.sql` catalog queries across `postgres`, `cockroach`, `mssql`, `mysql`, `oracle`, `db2`, and `duckdb` with pre-aggregated CTEs and joins, and reduced `ColumnMetadata` to its 18 core catalog fields.
+  ([#866](https://github.com/litestar-org/sqlspec/pull/866))
+* Redesigned the documentation site architecture around Zensical and `mkdocstrings-python` with Diataxis-aligned navigation (`Getting Started`, `Learn`, `Adapters`, `Extensions`, `Examples`, `Reference`), interactive WebAssembly SQL playground, runnable `docs/examples/` snippets, conceptual Mermaid diagrams, and tabbed package manager and `SQLFileLoader` examples.
+  ([#864](https://github.com/litestar-org/sqlspec/pull/864),
+  [#867](https://github.com/litestar-org/sqlspec/pull/867),
+  [#869](https://github.com/litestar-org/sqlspec/pull/869))
+* Migrated integration test infrastructure to clientless `pytest-databases` fixtures (`#859`).
+  ([#859](https://github.com/litestar-org/sqlspec/pull/859))
+* Upgraded CI GitHub Actions `astral-sh/setup-uv` from `10.3.1` to `10.3.3` and pinned `actions/checkout` (`6.0.2`), `actions/setup-python` (`7.1.0`), `actions/cache` (`6.0.0`), and `actions/upload-pages-artifact` / `actions/deploy-pages` in `.github/workflows/ci.yml` and `.github/workflows/docs.yml`.
+  ([#872](https://github.com/litestar-org/sqlspec/pull/872))
+
+**Fixed:**
+
+* Removed runtime `duckdb` imports from `sqlspec.adapters.adbc` (`AdbcDataDictionary` and `sqlspec/adapters/adbc/core.py`) and enforced strict adapter import isolation across all 20 database adapters in `tests/unit/test_import_time.py`.
+  ([#865](https://github.com/litestar-org/sqlspec/pull/865))
+* Eliminated extra row copies and `UserWarning` in `DuckDBDriver.select_to_arrow` on DuckDB 2.0+ by routing parameterless `SELECT` queries through `connection.sql(query).to_arrow_table()` and `to_arrow_reader(batch_size=...)` when no open DB-API transaction is active on the connection.
+* Fixed `compile_table_ddl_results` and `CreateIndex` cross-dialect DDL synthesis to emit `COMMENT ON` only on dialects that support it (`postgres`, `oracle`, `db2`, `duckdb`, `snowflake`), inline `COMMENT` clauses on `mysql` and `mariadb`, render `STORING (...)` on `spanner` and `spangres`, fold unquoted source identifiers to target case before quoting target-only reserved keywords, and preserve SQLite `NUMERIC` type affinity.
+* Fixed `MigrationSquasher.fold_canonical_ddl` to fall back cleanly to raw statement concatenation when a migration contains interleaved DML or procedural blocks (`INSERT`, `UPDATE`, `DELETE`, `MERGE`, `COPY`, `DO`, `CALL`), preserve table-level constraints across `ALTER TABLE ... RENAME COLUMN` and `DROP COLUMN`, and update `sqlspec_migrations` tracker history atomically inside a single transaction when applying squashed migrations.
+* Fixed `write_fixture_shards` to unlink stale shard and legacy single-file artifacts (`<table>.json`, `<table>.json.gz`, and higher-numbered shards) before writing new shards so subsequent `load_fixture` calls do not read orphaned records.
+
 ### v0.69.0 - Oracle PKCS#12 wallet conversion, TNS alias resolution, proximity RRF, and server-side DBMS_CLOUD ingestion
 
 **Added:**

docs/databases/adbc.md

@@ -6,9 +6,29 @@
 
 Install the core `adbc` extra alongside one or more PyPI-installable ADBC wire drivers:
 
-```bash
-uv add "sqlspec[adbc]" adbc-driver-postgresql
-```
+=== "uv"
+
+    ```bash
+    uv add "sqlspec[adbc]" adbc-driver-postgresql
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "sqlspec[adbc]" adbc-driver-postgresql
+    ```
+
+=== "pdm"
+
+    ```bash
+    pdm add "sqlspec[adbc]" adbc-driver-postgresql
+    ```
+
+=== "poetry"
+
+    ```bash
+    poetry add "sqlspec[adbc]" adbc-driver-postgresql
+    ```
 
 | Target Engine | PyPI Package | Supported URI Schemes | Short `driver` Aliases |
 | --- | --- | --- | --- |
@@ -104,6 +124,7 @@ with spec.provide_session(config) as session:
 | --- | --- | --- |
 | `driver` / `driver_name` / `uri` | `connection_config` | Short driver alias (`driver="duckdb"`, `driver="snowflake"`, `driver="bigquery"`), explicit module name (`driver_name="adbc_driver_duckdb"`), or connection URI with auto-detected scheme (`duckdb://`, `snowflake://`, `bigquery://`, `postgresql://`, `oracle://`, `db2://`, `quack://`, `grpc+tls://`). |
 | `username` / `password` / `token` | `connection_config` | Credentials lifted automatically into each driver's expected `db_kwargs` (`adbc.quack.token`, Oracle `db_kwargs`, etc.). |
+| `auth_token_provider` | `driver_features` | Callable invoked when opening ADBC connections to supply fresh IAM, OAuth, or Bearer tokens dynamically. |
 | `auth_type` / `tls_root_certs` / `headers` | `connection_config` | FlightSQL and GizmoSQL OAuth/SSO (`auth_type="external"`), mTLS, and gRPC options (`grpc_max_message_size`, `query_timeout`). |
 | `db_kwargs` / `conn_kwargs` | `connection_config` | Low-level ADBC database and connection option dictionaries. |
 | `enable_lowercase_column_names` | `driver_features` | Lowercases uppercase unquoted identifiers (`True` by default when `dialect` is `"oracle"` or `"db2"`). |

docs/databases/arrow-odbc.md

@@ -4,9 +4,29 @@
 
 ## Install and When to Choose
 
-```bash
-uv add "sqlspec[arrow-odbc]"
-```
+=== "uv"
+
+    ```bash
+    uv add "sqlspec[arrow-odbc]"
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "sqlspec[arrow-odbc]"
+    ```
+
+=== "pdm"
+
+    ```bash
+    pdm add "sqlspec[arrow-odbc]"
+    ```
+
+=== "poetry"
+
+    ```bash
+    poetry add "sqlspec[arrow-odbc]"
+    ```
 
 Choose `ArrowOdbcConfig` when your environment standardizes on enterprise unixODBC / Windows ODBC drivers (such as `ODBC Driver 18 for SQL Server`, `psqlodbc`, or IBM Db2 `libdb2o.so`) and needs fast, bounded-memory Arrow batch extraction without row-by-row `pyodbc` tuple overhead.
 

docs/databases/bigquery.md

@@ -4,9 +4,29 @@
 
 ## Install and When to Choose
 
-```bash
-uv add "sqlspec[bigquery]"
-```
+=== "uv"
+
+    ```bash
+    uv add "sqlspec[bigquery]"
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "sqlspec[bigquery]"
+    ```
+
+=== "pdm"
+
+    ```bash
+    pdm add "sqlspec[bigquery]"
+    ```
+
+=== "poetry"
+
+    ```bash
+    poetry add "sqlspec[bigquery]"
+    ```
 
 Choose `BigQueryConfig` for serverless warehouse queries, multi-statement GoogleSQL procedural scripts, cost-governed analytical APIs (`maximum_bytes_billed`, `reservation`), and streaming Arrow/Parquet lakehouse pipelines.
 

docs/databases/cockroachdb.md

@@ -4,9 +4,29 @@ SQLSpec provides two dedicated CockroachDB adapters—`sqlspec.adapters.cockroac
 
 ## Install and When to Choose
 
-```bash
-uv add "sqlspec[cockroachdb]"
-```
+=== "uv"
+
+    ```bash
+    uv add "sqlspec[cockroachdb]"
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "sqlspec[cockroachdb]"
+    ```
+
+=== "pdm"
+
+    ```bash
+    pdm add "sqlspec[cockroachdb]"
+    ```
+
+=== "poetry"
+
+    ```bash
+    poetry add "sqlspec[cockroachdb]"
+    ```
 
 - Choose **`CockroachAsyncpgConfig`** for pure-`asyncio` services prioritizing binary-protocol performance.
 - Choose **`CockroachPsycopgSyncConfig`** or **`CockroachPsycopgAsyncConfig`** when sharing sync/async pools or libpq TLS/GSSAPI certificate options.
@@ -72,6 +92,7 @@ uv add "sqlspec[cockroachdb]"
 | `default_staleness` | `str` | Follower-read staleness expression (such as `"follower_read_timestamp()"` or `"-10s"`). |
 | `enable_native_storage` | `bool` | Uses CockroachDB native CSV/nodelocal/cloud storage import/export paths where configured. |
 | `enable_pgvector` | `bool` | Enables CockroachDB `VECTOR` column codecs and similarity search in ADK stores (`cockroach_asyncpg`). |
+| `auth_token_provider` | `Callable` | Dynamic credential callback invoked per connection creation for short-lived cluster OAuth/IAM tokens. |
 
 See [Cockroach AsyncPG Reference](/reference/adapters/cockroach-asyncpg/) and [Cockroach Psycopg Reference](/reference/adapters/cockroach-psycopg/) for full configuration and retry TypedDict tables.
 

docs/databases/db2.md

@@ -4,9 +4,29 @@
 
 ## Install and When to Choose
 
-```bash
-uv add "sqlspec[db2]"
-```
+=== "uv"
+
+    ```bash
+    uv add "sqlspec[db2]"
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "sqlspec[db2]"
+    ```
+
+=== "pdm"
+
+    ```bash
+    pdm add "sqlspec[db2]"
+    ```
+
+=== "poetry"
+
+    ```bash
+    poetry add "sqlspec[db2]"
+    ```
 
 - Choose **`Db2SyncConfig`** or **`Db2AsyncConfig`** for transactional OLTP workloads over IBM's native `clidriver` (`ibm_db` / `ibm_db_dbi`).
 - Note on Linux `aarch64`: IBM publishes prebuilt `clidriver` wheels for Linux `x86_64`, macOS `arm64`, and Windows. On Linux ARM64 hosts, connect to Db2 via [`adbc`](adbc.md) (`adbc-driver-db2` pure-Go DRDA wheel) or guard `ibm-db` with environment markers.

docs/databases/duckdb.md

@@ -4,9 +4,29 @@
 
 ## Install and When to Choose
 
-```bash
-uv add "sqlspec[duckdb]"
-```
+=== "uv"
+
+    ```bash
+    uv add "sqlspec[duckdb]"
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "sqlspec[duckdb]"
+    ```
+
+=== "pdm"
+
+    ```bash
+    pdm add "sqlspec[duckdb]"
+    ```
+
+=== "poetry"
+
+    ```bash
+    poetry add "sqlspec[duckdb]"
+    ```
 
 Choose `DuckDBConfig` for embedded OLAP analytics, local lakehouse queries over Parquet/CSV/JSON on S3 or GCS, fast feature engineering pipelines, and columnar ETL jobs running inside a single Python worker.
 
@@ -40,9 +60,9 @@ For persistent analytical databases, set `"database": "var/analytics.duckdb"` (a
 | --- | --- | --- |
 | `database` / `read_only` | `connection_config` | File path or `":memory:"` (shared across pooled cursors derived from the primary connection) and read-only open flag. |
 | `pool_min_size` / `pool_max_size` | `connection_config` | Bounds for `DuckDBConnectionPool`. |
-| `threads` / `memory_limit` | `connection_config` | DuckDB engine worker threads and memory cap (such as `"4GB"`). |
+| `threads` / `memory_limit` | `connection_config` | DuckDB engine worker threads and memory cap (such as `"4GB"`). In Linux containers, `DuckDBConfig` automatically clamps default memory limits to cgroup v1/v2 container quotas (`resolve_cgroup_memory_limit`). |
 | `extensions` | `driver_features` | Sequence of `DuckDBExtensionConfig` dicts (`name`, `repository`, `force_install`) installed/loaded on startup. |
-| `secrets` | `driver_features` | Sequence of `DuckDBSecretConfig` dicts configuring S3, GCS, Azure, R2, or HuggingFace credentials via `CREATE SECRET`. |
+| `secrets` | `driver_features` | Sequence of `DuckDBSecretConfig` dicts configuring S3, GCS, Azure, R2, or HuggingFace credentials via `CREATE SECRET`, supporting dynamic credential rotation via `value_provider` callables. |
 | `enable_statement_profiling` | `driver_features` | Captures DuckDB JSON execution plans (`EXPLAIN (ANALYZE, FORMAT JSON)`) onto `StatementEvent.profile`. |
 | `enable_uuid_conversion` | `driver_features` | Normalizes UUID columns consistently across native and Arrow execution paths. |