feat: Replace information schema with describe calls#1415
Open
tejassp-db wants to merge 5 commits intomainfrom
Open
feat: Replace information schema with describe calls#1415tejassp-db wants to merge 5 commits intomainfrom
tejassp-db wants to merge 5 commits intomainfrom
Conversation
Add alternate code path for metadata fetching in _describe_relation methods using DESCRIBE TABLE EXTENDED AS JSON (DBR 17.3+). Replaces 4 information_schema queries in IncrementalTableAPI and get_view_description in MaterializedViewAPI/ViewAPI. Falls back to legacy queries when capability is unavailable. PECOBLR-2546
tejassp-db
changed the title
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace expensive
information_schemaqueries with a singleDESCRIBE TABLE EXTENDED ... AS JSONcall for metadata fetching in_describe_relationmethods. Gated behindDBRCapability.DESCRIBE_TABLE_EXTENDED_AS_JSON(DBR 17.3+), falls back to existing info_schema queries on older runtimes.Changes:
IncrementalTableAPI._describe_relation: replaces 4 info_schema queries (PK, FK, non-null constraints, column masks) with AS JSON parsingMaterializedViewAPI._describe_relation: replacesget_view_description(info_schema.views) with AS JSON parsingViewAPI._describe_relation: same as MVDatabricksDescribeJsonMetadataparser class with composite PK/FK supportis_describe_as_json_supported()gating method (checks HMS, foreign table, capability)is_foreign_tableproperty onDatabricksRelationdescribe_table_extended_as_jsonJinja macroTesting:
is_describe_as_json_supportedunit tests (UC, HMS, foreign table, no capability)Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.PECOBLR-2546