Repository navigation
feat(db_engine_specs): complete metadata definitions and add missing unit test suites - #43695
FrancescoCastaldi wants to merge 67 commits into
Conversation
…base schema on catalog override Address follow-up review comments from @sadpandajoe on PR apache#43695: 1. In DatabricksNativeEngineSpec, drop the forced default_catalog = "hive_metastore" fallback when no catalog is configured. Leaving catalog unset allows the server's default catalog (e.g. main) to apply naturally and prevents connection handshake errors on servers without multi-catalog support or discovery interference in get_default_catalog(). 2. In adjust_engine_params, apply the URI-database schema fallback only when catalog is not being overridden. When browsing a different catalog (catalog="main"), forcing a database name from the original URI (e.g. /finance) before catalog switching could fail the connection namespace handshake. 3. Update unit tests in test_databricks.py to verify that catalog is omitted when unconfigured and that URI-database schema fallback is skipped during catalog overrides. Refs: apache#43695
|
Follow-up updates (@sadpandajoe) - addressed in commit 6209589
@sadpandajoe, ready for re-review when convenient. |
…erride Follow up on review comments from @sadpandajoe on PR apache#43695: - In DatabricksNativeEngineSpec.adjust_engine_params, mutate query parameters via dict(uri.query) and uri.set(query=query_dict) instead of uri.update_query_dict, ensuring stale schema query parameters are completely purged from the URL query when catalog is overridden without schema. - Shorten test docstring in test_databricks.py to fit within the 88-column Ruff limit. - Add unit test verifying that starting from a URI with ?catalog=main&schema=finance, adjust_engine_params with catalog="other" strips schema from both the URL query and connect_args. Refs: apache#43695
|
Follow-up update in commit
All review items on PR #43695 are addressed. |
…etadata stdout ascii test Address review feedback from @sadpandajoe on PR apache#43695: 1. In superset/db_engine_specs/databricks.py: - When a catalog override matches the connection's existing catalog with schema=None (e.g. creating a dataset with catalog="main", schema=None against a /default?catalog=main URI), preserve the existing schema from connect_args, uri.query, or uri.database instead of dropping it to None. - When a catalog is overridden to a new or different catalog without an explicit schema, resolve the catalog's default schema ("default") instead of leaving effective_schema as None. This ensures the dialect's self.schema is always a valid string, preventing primary/foreign key reflection from generating invalid queries such as `main`.`None`.`orders`. 2. In tests/unit_tests/db_engine_specs/test_databricks.py: - Update unit test assertions for catalog overrides to assert that default schema "default" is resolved when catalog changes without an explicit schema. - Add test case verifying dataset creation scenario where catalog="main" matches existing URI catalog and preserves schema="default". 3. In tests/unit_tests/db_engine_specs/test_lint_metadata.py: - Add test_main_ascii_stdout_utf8_reconfigure verifying that calling main() with an ASCII-configured TextIOWrapper stdout successfully reconfigures to UTF-8 without UnicodeEncodeError and produces valid decodable output. - Add test_main_ascii_stdout_markdown_report testing the --markdown flag under ASCII console redirection. Refs: apache#43695
PR Update: Databricks Default Schema Resolution and CLI ASCII Stdout Test CoverageAddressing review feedback from @sadpandajoe: Changes Applied (Commit
|
Address review feedback from @sadpandajoe on PR apache#43695: - databricks.py: reset the schema to "default" only when the caller switches away from a catalog explicitly configured on the connection. A connection without a catalog (e.g. /finance?http_path=...) now keeps the URI schema when dataset creation passes the server default catalog, so reflection targets main.finance.orders instead of main.default.orders. - test_databricks.py: pin the no-catalog /finance URI with catalog="main", the /finance?catalog=main URI with catalog="main", and a real switch from connect_args catalog "main" to "other". Apply ruff format. - test_lint_metadata.py: move `import io` into the stdlib group (I001).
|
@sadpandajoe thanks for the review, all three points are addressed in 58d7510. Databricks catalog/schema resolution
Lint and formatting
Verification
Ready for another look when you have a moment. |
…nt_metadata (apache#43695) Address review feedback from @sadpandajoe on PR apache#43695: 1. In tests/unit_tests/db_engine_specs/test_databricks.py: - Add `"encryption": False` to `build_sqlalchemy_uri` parameter dicts, satisfying `DatabricksBaseParametersType` and clearing mypy error. - Add unit test case verifying `uri.database or "default"` fallback in `adjust_engine_params` when URI contains no database or schema. 2. In tests/unit_tests/db_engine_specs/test_lint_metadata.py: - Populate all required fields (`description`, `categories`, `pypi_packages`, `connection_string`) in `test_main_ascii_stdout_markdown_report` so the markdown report outputs `✓` for the missing required column, and assert it survives UTF-8 decode on ASCII stdout. 3. In superset/db_engine_specs/databricks.py: - Fold `effective_catalog = catalog or original_catalog` into `if effective_catalog := catalog or original_catalog:` to align with `auto-walrus` and `ruff-format` pre-commit hooks. Refs: apache#43695
|
Addressed all 4 review follow-ups in commit
@sadpandajoe Ready for re-review. |
|
@sadpandajoe you approved but then left some review comments... did you want to block on those, or are they optional? Up to @FrancescoCastaldi if you want to assess/address them in the meantime :D Tempted by the big green merge button :D |
|
@rusackas @sadpandajoe Both follow-ups were addressed and verified:
Ready for the big green merge button whenever you like, thank you both for the great reviews! |
rusackas
left a comment
There was a problem hiding this comment.
Heya @FrancescoCastaldi, this has been a genuinely impressive back-and-forth, and almost all of @sadpandajoe's threads are buttoned up now.
One gap though: your last comment says both follow-ups were addressed, but that doesn't look like it covers his two from the 21st. The tracking_url still points at github.com/databricks/databricks-dbapi, which 404s - checked PyPI's metadata for the databricks-dbapi package and the actual repo is crflynn/databricks-dbapi. Left a suggestion for both spots.
The registry.impls point is more of a nit (the real dialect-resolution check exists elsewhere, just skipped when the package isn't installed), so I wouldn't block on that one alone.
@sadpandajoe, I don't see an answer to my question above either - were these two meant to block, or just FYI? Up to you whether this needs another look before the green button.
| "support is planned for the legacy driver. Use the Databricks " | ||
| "Python Connector (databricks-sql-connector) instead." | ||
| ), | ||
| "tracking_url": "https://github.com/databricks/databricks-dbapi", |
There was a problem hiding this comment.
Dead link, confirmed via PyPI's project metadata for databricks-dbapi - this is the real repo.
| "tracking_url": "https://github.com/databricks/databricks-dbapi", | |
| "tracking_url": "https://github.com/crflynn/databricks-dbapi", |
There was a problem hiding this comment.
note that this repo is ALSO archived... not sure the latest state entry point for this.
| "support is planned for the legacy driver. Use the Databricks " | ||
| "Python Connector (databricks-sql-connector) instead." | ||
| ), | ||
| "tracking_url": "https://github.com/databricks/databricks-dbapi", |
There was a problem hiding this comment.
Same dead link as the ODBC spec above.
| "tracking_url": "https://github.com/databricks/databricks-dbapi", | |
| "tracking_url": "https://github.com/crflynn/databricks-dbapi", |
There was a problem hiding this comment.
Updated in ec4dc71 to point at https://github.com/crflynn/databricks-dbapi.
… dialect resolution (apache#43695) - Update tracking_url for databricks-dbapi incompatibility to https://github.com/crflynn/databricks-dbapi in DatabricksODBCEngineSpec and DatabricksHiveEngineSpec. - Verify registry.load("databricks.connector") resolves to DatabricksDialect in test_databricks_connector_dialect_registration. Signed-off-by: Francesco Castaldi <info@francescocastaldi.it>
…mat databricks test to 88 columns Signed-off-by: Francesco Castaldi <francesco.castaldi@mapsgroup.it>
Signed-off-by: Francesco Castaldi <info@francescocastaldi.it>
| class ComplexValuesBaseEngineSpec(BaseEngineSpec): | ||
| engine: str = "complex_engine" | ||
| engine_name: str = "Complex Engine" | ||
| metadata: dict[str, Any] = { |
There was a problem hiding this comment.
The new ast.Tuple branch in _eval_ast_value has no coverage: the fixtures only use sets, negative numbers, and strings, so deleting the branch would silently turn tuple-valued metadata into None while every linter test stays green. Could a fixture add a tuple value such as "pair": ("value", -1) and assert the exact extracted tuple?
There was a problem hiding this comment.
Updated ComplexValuesBaseEngineSpec fixture in tests/unit_tests/db_engine_specs/test_lint_metadata.py with "pair": ("value", -1) to explicitly assert ast.Tuple evaluation in test_get_all_engine_specs_ast_set_and_unary_minus.
Summary of Review Feedback ResolutionsAll inline review threads and follow-ups have been resolved and verified across the test suite:
cc @sadpandajoe @rusackas @villebro for final review and merge. |
| "logo": "clickhouse.svg", | ||
| "logo_dark": "clickhouse-dark.svg", | ||
| "homepage_url": "https://clickhouse.com/", | ||
| "sqlalchemy_docs_url": "https://clickhouse.com/docs/en/integrations/sql-clients/superset", |
There was a problem hiding this comment.
This sqlalchemy_docs_url redirects to clickhouse.com/docs/integrations/sql-clients/superset, which returns a 404, so the generated SQLAlchemy documentation link for ClickHouse Connect sends users to a missing page. Could it point at the live Superset guide that docs_url further down already reaches (https://clickhouse.com/docs/integrations/connectors/data-visualization/superset-and-clickhouse)?
There was a problem hiding this comment.
Updated both sqlalchemy_docs_url and docs_url in ClickHouseConnectEngineSpec to the live documentation guide:
https://clickhouse.com/docs/integrations/connectors/data-visualization/superset-and-clickhouse.
| connect_args={"schema": "finance"}, | ||
| ) | ||
| assert adj_url_saved.query["schema"] == "finance" | ||
| assert adj_args_saved["schema"] == "finance" |
There was a problem hiding this comment.
The or uri.query.get("schema") fallback in adjust_engine_params isn't pinned by any case: this one supplies connect_args["schema"], which wins first, and the other URI-schema cases switch catalogs or have no schema in the query. Deleting that fallback would still pass every assertion here, while /legacy?catalog=main&schema=finance with connect_args={} would then resolve to legacy. Could a case with that URI assert that both the returned URL query and connect_args carry finance?
There was a problem hiding this comment.
Added test case in test_databricks_native_adjust_engine_params exercising make_url("databricks+connector://token:abc@my_hostname:1234/legacy?catalog=main&schema=finance") with empty connect_args, verifying that uri.query["schema"] is preserved and passed to both the returned URL and connect_args.
…fallback (apache#43695) - Resolve dialect catalog dynamically during get_columns and get_table_names when no catalog is configured on connection handshake. - Distinguish server default catalog from genuine catalog switches when original catalog is unconfigured. - Align ClickHouse Connect doc URLs to official live documentation guide. - Assert ast.Tuple evaluation in test_lint_metadata. - Add test coverage for catalog switch and no-catalog reflection. Signed-off-by: Francesco Castaldi <info@francescocastaldi.it>
|
@sadpandajoe Thank you for the detailed review. All items have been addressed in the latest commit:
|
Summary
This PR improves Database Engine Specs completeness, fixes AST metadata linting for newer Python versions, and adds unit test suites for database engine specs lacking dedicated coverage.
Key Changes
lint_metadata.py):AnnAssign).Tuple,Set, andUnaryOpevaluation.tests/unit_tests/db_engine_specs/for 22 dialects (including Cloudflare D1, Arc, IBM i, Parseable, Ocient, MaxCompute/ODPS, Hologres, Ascend, IoTDB, Exasol, MonetDB, Solr, RisingWave, Phoenix, Netezza, Greenplum, YugabyteDB, Shillelagh, Teradata, Vertica, Spark SQL, and Superset meta DB).Testing Instructions
python superset/db_engine_specs/lint_metadata.pytests/unit_tests/db_engine_specs/