feat(cookbooks): add Knowledge Catalog cookbook suite - #284
Open
hyunuk wants to merge 14 commits into
Open
Conversation
- Add verified metadata_as_code_dark_data.ipynb cookbook with 3-layer test verification - Add initial draft cookbooks (curated_data_products, lineage_graph_observability) and sample manuals
- Add verified curated_data_products.ipynb with 3-layer test verification - Add data contract SLA aspect and AI golden queries grounding
- Add verified lineage_graph_observability.ipynb with 3-layer test verification - Add programmatic Lineage API querying, topology visualization, and root-cause drift isolation
… and branding - Remove unused tabulate dependency and BadRequest import - Eliminate directional words and 1st-person plural pronouns - Replace em dashes with standard commas per style guide - Update Universal Catalog references to Knowledge Catalog
…a_products with style guide - Add architectural rationale for zero-copy data mesh packaging, machine-readable SLAs, and protobuf integer indexing - Explain golden queries grounding for AI agents and dual-consumer patterns - Eliminate directional words and 1st-person plural pronouns per Google Developer Style Guide - Replace em dashes with standard punctuation - Remove unused pyyaml dependency from setup
…ry data, HITL governance, and schema drift RCA - Ingest authentic e-commerce dataset from bigquery-public-data.thelook_ecommerce into 3-tier pipeline - Implement human-in-the-loop (HITL) metadata review and approval promotion - Inject upstream schema drift into users_raw and detect impact across lineage graph - Implement recursive root-cause analysis (trace_root_cause) to isolate originating fault - Add lineage explainability auditing with BigQuery dry-run syntax verification - Align with Google Developer Style Guide and pass 3-layer test suite
…uganizer IDs - Set utm_campaign=b543855766 for metadata_as_code_dark_data.ipynb - Set utm_campaign=b543859711 for curated_data_products.ipynb - Set utm_campaign=b543859650 for lineage_graph_observability.ipynb
…ing layers on live GCP - Set default LOCATION to US for seamless bigquery-public-data multi-region CTAS - Robustify provision_dataset to ensure dataset location matches target region - Use pure pandas DataFrame row-dict conversion for preview queries - Pass Layer 1 Unit Tests (16/16 tests), Layer 2 Cleanup, and Layer 3 E2E Execution on live environment
…nt automatically - Add explicit Callout Note in Cell 3 explaining bigquery-public-data US multi-region requirement - Automatically normalize LOCATION to US in Cell 4 for public data pipeline execution - Pass all 3 testing layers top-to-bottom on live GCP
…bformat v4 schema compliance
… with Gemini 3.6 Flash - Upgrades reasoning model to gemini-3.6-flash and removes deprecated sampling arguments - Aligns markdown headings with Google Developer Style Guide sentence-case conventions - Enhances fail-fast parameter validation with descriptive remediation instructions - Implements BigQuery, OpenLineage, Dataplex aspect types, and cryptographic audit certificates
…prise_governance_observability
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This pull request adds cookbooks for Google Cloud Knowledge Catalog. These recipes demonstrate end-to-end patterns for enterprise data governance, curated data products, and multimodal dark data extraction.
Included Cookbooks
metadata_as_code_dark_data.ipynb• Strict Pydantic schema validation
• Programmatic Knowledge Catalog custom aspect registration and tagging
• Round-trip aspect verification
curated_data_products.ipynb• Machine-readable contract SLAs
• Golden queries aspect for AI grounding
• Grounded BigQuery SQL generation.
enterprise_governance_observability.ipynb• OpenLineage lifecycle events & schema facets
• Automated data quality diagnostic gate & poisoning prevention
• Custom aspect governance.