diff --git a/Allfiles/01-explore-azure-databricks.ipynb b/Allfiles/01-explore-azure-databricks.ipynb index af6e358..8b46625 100644 --- a/Allfiles/01-explore-azure-databricks.ipynb +++ b/Allfiles/01-explore-azure-databricks.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "82c6d5ed", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7a601857-78c8-4bab-b2a6-4389ddc11845", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Running Python Code\n", "\n", @@ -17,25 +25,80 @@ "2. Prints a formatted message that summarises the route. For example:\n", " `Route: City Express | From: Central Station → To: Airport Terminal`\n", "\n", - "> 💬 **Genie Code tip:** Open Genie Code panel (click the ![assistant-icon](https://raw.githubusercontent.com/MicrosoftLearning/DP-750T00-Implement-Data-Engineering-Solutions-using-Azure-Databricks/refs/heads/main/Allfiles/media/databricks-assistant.svg) icon on the cell) and type:\n", + "> \uD83D\uDCAC **Genie Code tip:** Open Genie Code panel (click the ![assistant-icon](https://raw.githubusercontent.com/MicrosoftLearning/DP-750T00-Implement-Data-Engineering-Solutions-using-Azure-Databricks/refs/heads/main/Allfiles/media/databricks-assistant.svg) icon on the cell) and type:\n", "> *\"Write Python code to define three route variables and print a formatted summary message\"*\n", "> Use the suggestion as a starting point, then personalise the values." ] }, { "cell_type": "code", - "execution_count": null, - "id": "ef6b149a", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "784b33d8-e96a-4ecf-912c-b4c1ea50a4cc", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 2" + } + }, "outputs": [], "source": [ - "# TODO: Define your route variables and print a formatted summary message" + "%python\n", + "\n", + "route_data =[]\n", + "route_id = ['R001' 'R002', 'R003']\n", + "route_name = ['City Express', 'Harbor Line', 'Metro']\n", + "origin = ['Central Station', 'Downtown', 'Airport Terminal']\n", + "destination = ['Airport Terminal', 'Airport Terminal', 'City Center']\n", + "\n", + "for i in range(len(route_id)):\n", + " route_data.append([route_id[i],route_name[i],origin[i],destination[i]])\n", + "\n", + "route_data\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "374888a6-5fa5-4ad1-a90a-6dab15ad7252", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 3" + } + }, + "outputs": [], + "source": [ + "for i in range(len(route_data)):\n", + " print(f\"Route: {route_data[i][1]} | From: {route_data[i][2]} → To: {route_data[i][3]}\")" ] }, { "cell_type": "markdown", - "id": "0d2df199", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "55d3b919-ace8-4586-9f93-ae3b06e46e27", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Running SQL Commands\n", "\n", @@ -50,26 +113,46 @@ "\n", "The result should be a single-row table with at least two columns.\n", "\n", - "> 💬 **Genie Code tip:** Ask Genie Code:\n", + "> \uD83D\uDCAC **Genie Code tip:** Ask Genie Code:\n", "> *\"Write a SQL SELECT statement in a Databricks notebook that returns the current timestamp and a string literal as named columns\"*\n", "> Remember to start your cell with `%sql`." ] }, { "cell_type": "code", - "execution_count": null, - "id": "01a150db", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "57a32746-f97d-44ec-9832-fcb1b9f76ebd", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Use the %sql magic command and write a SQL query\n", - "# Hint: start with %sql on the first line, then write your SELECT statement" + "%sql\n", + "SELECT current_timestamp() AS current_datetime, 'CityMoves Transit' AS system_name\n" ] }, { "cell_type": "markdown", - "id": "2a577b30", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "63912b42-24eb-43e2-8b7f-3ad3fc22e35a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Documenting with Markdown\n", "\n", @@ -83,23 +166,88 @@ "2. A short paragraph (2–3 sentences) describing the purpose of the notebook in the context of CityMoves Transit.\n", "3. A bulleted list (`-`) of at least three things a reader will find in this notebook.\n", "\n", - "> 💬 **Genie Code tip:** Select this markdown cell and ask Genie Code:\n", + "> \uD83D\uDCAC **Genie Code tip:** Select this markdown cell and ask Genie Code:\n", "> *\"Write a Markdown introduction cell for a data engineering notebook about a public transit system called CityMoves Transit\"*\n", "> Refine the generated text as you see fit." ] }, { "cell_type": "markdown", - "id": "974bf85b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6d007e6b-f716-477d-9bc1-1c92b36fdd3b", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, + "source": [ + "# CityMoves Transit — Data Engineering Notebook\n", + "\n", + "This notebook serves as an introductory exploration of the CityMoves Transit data platform built on Azure Databricks. It demonstrates core notebook capabilities that the data engineering team will use daily for route analytics and operational reporting.\n", + "\n", + "- Python code for defining and summarising transit route data\n", + "- SQL queries for retrieving system metadata and timestamps\n", + "- Markdown documentation for clear, shareable notebook structure" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "f398ad50-590e-48ac-874e-bba7e6217801", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, + "source": [] + }, + { + "cell_type": "markdown", + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ee66acac-aec1-451c-a96b-c4498421ece1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 3202781778147245, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "01-explore-azure-databricks (1)", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/02-select-and-configure-compute.ipynb b/Allfiles/02-select-and-configure-compute.ipynb index 62deb25..516fe49 100644 --- a/Allfiles/02-select-and-configure-compute.ipynb +++ b/Allfiles/02-select-and-configure-compute.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "f611b9a4", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "47edc795-82c4-4438-a096-8eaa26cc876a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Install Libraries Notebook-Scoped\n", "\n", @@ -16,14 +24,22 @@ }, { "cell_type": "markdown", - "id": "47d5e4b7", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6e2f07ae-d264-45b2-abfc-b9d960754fc3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.1 — Install the `faker` library notebook-scoped\n", "\n", "Your team relies on `faker` to generate synthetic patient records for pipeline testing. Install it as a notebook-scoped library so the package is available to this notebook only and doesn't affect other users sharing the same compute.\n", "\n", - "> 🤖 **Genie Code tip:** Open Genie Code panel (click the ![assistant-icon](https://raw.githubusercontent.com/MicrosoftLearning/DP-750T00-Implement-Data-Engineering-Solutions-using-Azure-Databricks/refs/heads/main/Allfiles/media/genie-code.svg) icon on the cell) and use the prompt below to get started:\n", + "> \uD83E\uDD16 **Genie Code tip:** Open Genie Code panel (click the ![assistant-icon](https://raw.githubusercontent.com/MicrosoftLearning/DP-750T00-Implement-Data-Engineering-Solutions-using-Azure-Databricks/refs/heads/main/Allfiles/media/genie-code.svg) icon on the cell) and use the prompt below to get started:\n", "> *\"How do I install a Python package notebook-scoped in Databricks using a magic command?\"*\n", "\n", "**Hint:** Use a `%pip` magic command to install the package. Pinning an exact version (e.g., `faker==40.8.0`) is a good practice for reproducibility." @@ -31,18 +47,37 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "667ecc3f", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "8ef69aa3-d5a0-44d9-ab99-1bb0cc8aa8a2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Install faker==40.8.0 notebook-scoped" + "%pip install faker==40.8.0" ] }, { "cell_type": "markdown", - "id": "5145c763", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "d90646dc-cf99-442e-852c-dcd3ad8f1ae9", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.2 — Verify the installation\n", "\n", @@ -52,24 +87,48 @@ "2. Creating a `Faker` instance.\n", "3. Printing a randomly generated **full name** and **date of birth**.\n", "\n", - "> 🤖 **Genie Code tip:**\n", + "> \uD83E\uDD16 **Genie Code tip:**\n", "> *\"Show me a Python example of generating a random name and date of birth using the Faker library.\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "df305ccc", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "916c5cad-4b4b-4946-ac40-11bd951ddea6", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Import Faker, create an instance, and print a generated name and date of birth" + "from faker import Faker\n", + "\n", + "fake = Faker()\n", + "\n", + "print(f\"Full Name: {fake.name()}\")\n", + "print(f\"Date of Birth: {fake.date_of_birth(minimum_age=20, maximum_age=80)}\")" ] }, { "cell_type": "markdown", - "id": "537b38a5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "3f613c1f-763b-44d4-a070-d283dd24b382", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Generate and Analyze Synthetic Patient Data\n", "\n", @@ -80,8 +139,16 @@ }, { "cell_type": "markdown", - "id": "ee221a7b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "677df061-0d6b-40d3-a2ef-d2de6128f607", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.1 — Generate a synthetic patient admissions DataFrame\n", "\n", @@ -97,7 +164,7 @@ "\n", "Display the first 10 rows of the resulting DataFrame.\n", "\n", - "> 🤖 **Genie Code tip:**\n", + "> \uD83E\uDD16 **Genie Code tip:**\n", "> *\"How do I generate a list of Python dictionaries using the Faker library, then create a PySpark DataFrame from that list?\"*\n", ">\n", "> **Hint:** Use `faker.date_of_birth()` with `minimum_age` and `maximum_age` parameters, or `faker.date_between()` with `start_date` and `end_date`. Use `random.choice()` for the diagnosis code." @@ -105,19 +172,53 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "b38ec3d4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "9d3b3ee7-b41b-477f-844e-e9237abd9be0", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Generate 100 synthetic patient records and create a Spark DataFrame\n", - "# Display the first 10 rows" + "from faker import Faker\n", + "import random\n", + "from datetime import date\n", + "fake = Faker()\n", + "diagnosis_codes = [\"I21\", 'J18', 'E11', 'K80', 'N39']\n", + "\n", + "records = []\n", + "for i in range(1, 1001):\n", + " records.append({\n", + " \"patient_id\": i, \n", + " \"full_name\": fake.name(), \n", + " \"date_of_birth\": str(fake.date_between(start_date = date(1940, 1, 1), end_date = date(2005, 12, 31))), \n", + " \"admission_date\":str(fake.date_between(start_date=date(2023, 1, 1), end_date= date(2025, 12, 31))), \n", + " \"diagnosis_code\": random.choice(diagnosis_codes)\n", + " })\n", + "df = spark.createDataFrame(records)\n", + "display(df)" ] }, { "cell_type": "markdown", - "id": "034453d6", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "947491cd-51e9-48c7-8f48-f342f6c12935", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.2 — Analyze admissions by diagnosis code\n", "\n", @@ -127,26 +228,53 @@ "2. Order the results from **most to least** admissions.\n", "3. Display the result.\n", "\n", - "> 🤖 **Genie Code tip:**\n", + "> \uD83E\uDD16 **Genie Code tip:**\n", "> *\"How do I group by a column, count occurrences, and sort the result in descending order using PySpark DataFrame API?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "0115d27c", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "cba9771b-976e-40e3-ba2c-11c7d771481b", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Count admissions per diagnosis_code, ordered from most to least" + "from pyspark.sql.functions import col \n", + "\n", + "\n", + "diagnosis_counts = df.groupBy(\"diagnosis_code\").count().orderBy(col(\"count\").desc())\n", + "display(diagnosis_counts)" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": null, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "pythonIndentUnit": 4 + }, + "notebookName": "02-select-and-configure-compute", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 + "nbformat_minor": 0 } diff --git a/Allfiles/03-create-and-organize-objects-in-unity-catalog.ipynb b/Allfiles/03-create-and-organize-objects-in-unity-catalog.ipynb index 7e49032..fddca91 100644 --- a/Allfiles/03-create-and-organize-objects-in-unity-catalog.ipynb +++ b/Allfiles/03-create-and-organize-objects-in-unity-catalog.ipynb @@ -2,8 +2,19 @@ "cells": [ { "cell_type": "markdown", - "id": "86081d58", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d85d70ca-f48a-4fd3-9fd6-d4f3d60167fc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Set Up the Catalog Structure\n", "\n", @@ -16,7 +27,7 @@ "```txt\n", "\n", "Unity Catalog\n", - "└── edu_dev [catalog]\n", + "└── edu_dev63853039 [catalog]\n", " │ tags: environment=development | university=lakeside | data_classification=internal\n", " │\n", " ├── bronze [schema]\n", @@ -61,18 +72,29 @@ }, { "cell_type": "markdown", - "id": "293e733e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "bf42b44b-1e8b-4b2a-930f-d5c2f97b88d1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.1 — Create the development catalog\n", "\n", - "Create a catalog called `edu_dev` with a descriptive comment. This catalog serves as the **development environment** for Lakeside University's data platform.\n", + "Create a catalog called `edu_dev63853039` with a descriptive comment. This catalog serves as the **development environment** for Lakeside University's data platform.\n", "\n", "**Naming convention reminders:**\n", "- Use lowercase with underscores\n", "- No periods, spaces, forward slashes, or control characters\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code:\n", "> *\"How do I create a Unity Catalog catalog in Databricks SQL with a comment?\"*\n", "\n", "**Hint:** Use the SQL command `CREATE CATALOG`." @@ -80,9 +102,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "9933f5b5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "56373874-bc35-479c-8712-45bb01547a79", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -101,23 +134,48 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "e38aeda3", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "47ea25a1-9872-4855-9ad6-6c41b049141e", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 5" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the edu_dev catalog with a descriptive comment" + "CREATE CATALOG IF NOT EXISTS edu_dev63853039\n", + "MANAGED LOCATION 'abfss://unity-catalog-storage@dbstoragep3bbxu5v55osy.dfs.core.windows.net/7405616271049399'\n", + "COMMENT 'Development catalog for Lakeside University data platform';" ] }, { "cell_type": "markdown", - "id": "3470e517", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f6b3c97c-ac3b-433b-8114-ecc45c3aefd2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.2 — Create medallion schemas\n", "\n", - "Within the `edu_dev` catalog, create three schemas that implement the **medallion architecture**:\n", + "Within the `edu_dev63853039` catalog, create three schemas that implement the **medallion architecture**:\n", "\n", "| Schema | Purpose |\n", "|--------|---------|\n", @@ -125,7 +183,7 @@ "| `silver` | Cleaned, validated, and enriched data |\n", "| `gold` | Aggregated, analytics-ready data |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"Show me how to create multiple schemas in a Databricks Unity Catalog catalog using SQL.\"*\n", "\n", "**Hint:** Use `CREATE SCHEMA IF NOT EXISTS catalog.schema_name` for each schema." @@ -133,19 +191,49 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "f82b0009", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d668e74c-1ef6-4a77-b33e-c496f517ee52", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the bronze, silver, and gold schemas inside edu_dev" + "CREATE SCHEMA IF NOT EXISTS edu_dev63853039.bronze\n", + "COMMENT 'Raw ingested data - unmodified source files'; \n", + "\n", + "CREATE SCHEMA IF NOT EXISTS edu_dev63853039.silver\n", + "COMMENT 'Cleaned, validated, and enriched data';\n", + "\n", + "CREATE SCHEMA IF NOT EXISTS edu_dev63853039.gold\n", + "COMMENT 'Aggregated and ready for analysis';\n" ] }, { "cell_type": "markdown", - "id": "fc2c333a", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b14d3471-b8a1-4906-a503-ea9e4ab61181", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Create Tables with Constraints\n", "\n", @@ -154,12 +242,23 @@ }, { "cell_type": "markdown", - "id": "43cc1d95", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "55175eca-57fa-4eda-bd30-203cc34d41ac", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.1 — Create the `students` table\n", "\n", - "Create a managed Delta table `edu_dev.silver.students` to store student records with the following columns:\n", + "Create a managed Delta table `edu_dev63853039.silver.students` to store student records with the following columns:\n", "\n", "| Column | Type | Description |\n", "|--------|------|-------------|\n", @@ -170,7 +269,7 @@ "| `enrollment_year` | INT | Year the student enrolled |\n", "| `program` | STRING | Degree program |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a managed Delta table with a primary key constraint in Unity Catalog?\"*\n", "\n", "**Hint:** Add `CONSTRAINT students_pk PRIMARY KEY (student_id)` at the end of the column list." @@ -178,23 +277,55 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "c84aa6e8", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "545e9034-88b2-4640-89f1-b511280c027e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create edu_dev.silver.students with a primary key constraint" + "CREATE TABLE IF NOT EXISTS edu_dev63853039.silver.students (\n", + "\n", + " student_id BIGINT NOT NULL,\n", + " first_name STRING, \n", + " last_name STRING, \n", + " email STRING, \n", + " enrolled_date TIMESTAMP, \n", + " program STRING, \n", + " CONSTRAINT students_pk PRIMARY KEY(student_id)\n", + ");" ] }, { "cell_type": "markdown", - "id": "62473788", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "2142e2db-871d-442e-a8cb-018e428970cc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.2 — Create the `courses` table\n", "\n", - "Create `edu_dev.silver.courses` with the following columns:\n", + "Create `edu_dev63853039.silver.courses` with the following columns:\n", "\n", "| Column | Type | Description |\n", "|--------|------|-------------|\n", @@ -203,29 +334,58 @@ "| `department` | STRING | Academic department offering the course |\n", "| `credits` | INT | Number of academic credits |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"Write a CREATE TABLE statement for a courses table in Databricks Unity Catalog with a primary key constraint.\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "12121b41", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "a684072c-2646-4b8d-91ff-eb898d9411c8", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create edu_dev.silver.courses with a primary key constraint" + "CREATE TABLE IF NOT EXISTS edu_dev63853039.silver.courses (\n", + " course_id BIGINT NOT NULL, \n", + " course_name STRING, \n", + " department STRING, \n", + " credits INT, \n", + " CONSTRAINT courses_pk PRIMARY KEY(course_id)\n", + "); " ] }, { "cell_type": "markdown", - "id": "1a2238fd", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b0465fe0-40c2-4322-9d9e-dcf776f63955", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.3 — Create the `enrollments` table with foreign keys\n", "\n", - "Create `edu_dev.silver.enrollments` to record which students are enrolled in which courses. This table must reference both `students` and `courses` via foreign key constraints.\n", + "Create `edu_dev63853039.silver.enrollments` to record which students are enrolled in which courses. This table must reference both `students` and `courses` via foreign key constraints.\n", "\n", "| Column | Type | Description |\n", "|--------|------|-------------|\n", @@ -235,7 +395,7 @@ "| `semester` | STRING | Academic semester (e.g., `Spring 2023`) |\n", "| `grade` | DECIMAL(4,2) | Numerical grade on a 0.0–10.0 scale |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I add foreign key constraints referencing other tables when creating a table in Databricks Unity Catalog?\"*\n", "\n", "**Hint:** Use `CONSTRAINT FOREIGN KEY () REFERENCES ..()` for each foreign key." @@ -243,19 +403,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "aaf6a330", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "cbed427e-ab8d-41f7-89b7-a5c83975fe0c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create edu_dev.silver.enrollments with primary key and two foreign key constraints" + "CREATE TABLE IF NOT EXISTS edu_dev63853039.silver.enrollments (\n", + " enrollment_id BIGINT NOT NULL, \n", + " student_id BIGINT, \n", + " course_id BIGINT, \n", + " semester STRING, \n", + " grade DECIMAL(4,2), \n", + " CONSTRAINT enrollments_pk PRIMARY KEY(enrollment_id),\n", + " CONSTRAINT enrollments_students_fk FOREIGN KEY(student_id) REFERENCES edu_dev63853039.silver.students(student_id),\n", + " CONSTRAINT enrollments_courses_fk FOREIGN KEY(course_id) REFERENCES edu_dev63853039.silver.courses(course_id)\n", + ");" ] }, { "cell_type": "markdown", - "id": "e06d70c9", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "eac7be32-e47e-4ae5-9a7b-b1d802893636", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Sample data — run this cell\n", "\n", @@ -264,14 +456,26 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "3de44411", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "5a612473-233c-4115-a415-2da3a4da9fc4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Populate students\n", - "INSERT INTO edu_dev.silver.students VALUES\n", + "INSERT INTO edu_dev63853039.silver.students VALUES\n", "(1001, 'Emma', 'Watson', 'emma.watson@lakeside.edu', 2022, 'Computer Science'),\n", "(1002, 'Liam', 'Ahmed', 'liam.ahmed@lakeside.edu', 2023, 'Data Science'),\n", "(1003, 'Sofia', 'Chen', 'sofia.chen@lakeside.edu', 2021, 'Mathematics'),\n", @@ -284,7 +488,7 @@ "(1010, 'Jake', 'OBrien', 'jake.obrien@lakeside.edu', 2020, 'Computer Science');\n", "\n", "-- Populate courses\n", - "INSERT INTO edu_dev.silver.courses VALUES\n", + "INSERT INTO edu_dev63853039.silver.courses VALUES\n", "(101, 'Introduction to Programming', 'Computer Science', 3),\n", "(102, 'Data Structures', 'Computer Science', 4),\n", "(103, 'Statistics for Data Science', 'Data Science', 3),\n", @@ -293,7 +497,7 @@ "(106, 'Linear Algebra', 'Mathematics', 3);\n", "\n", "-- Populate enrollments\n", - "INSERT INTO edu_dev.silver.enrollments VALUES\n", + "INSERT INTO edu_dev63853039.silver.enrollments VALUES\n", "(1, 1001, 101, 'Spring 2023', 8.50),\n", "(2, 1001, 102, 'Fall 2023', 7.80),\n", "(3, 1002, 103, 'Spring 2023', 9.20),\n", @@ -313,8 +517,19 @@ }, { "cell_type": "markdown", - "id": "4333ac8a", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "497190f2-178f-4375-b070-04575c16b6a7", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Create Views\n", "\n", @@ -323,12 +538,23 @@ }, { "cell_type": "markdown", - "id": "da6bf6d7", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "945011ce-9172-404c-a691-5a918c45884a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.1 — Create a standard view\n", "\n", - "Create a view `edu_dev.silver.vw_student_enrollments` that presents a combined, human-readable view of student enrollment records. It should include:\n", + "Create a view `edu_dev63853039.silver.vw_student_enrollments` that presents a combined, human-readable view of student enrollment records. It should include:\n", "\n", "- Student full name (concatenate `first_name` and `last_name` with a space between them)\n", "- Student `email`\n", @@ -337,7 +563,7 @@ "- `semester`\n", "- `grade`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"Write a SQL view in Azure Databricks that joins three tables: students, courses, and enrollments\"*\n", "\n", "**Hint:** Use `CONCAT(first_name, ' ', last_name)` or the `||` operator to combine names. You need to join all three silver tables." @@ -345,24 +571,70 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "36092d06", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f708c615-da8a-49c1-a866-3f3e2b3841ea", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 18" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the edu_dev.silver.vw_student_enrollments view\n", - "-- Join students, courses, and enrollments; include full_name, email, course_name, department, semester, grade" + "-- TODO: Create the edu_dev63853039.silver.vw_student_enrollments view\n", + "-- Join students, courses, and enrollments; include full_name, email, course_name, department, semester, grade\n", + "\n", + "DROP TABLE IF EXISTS edu_dev63853039.silver.vw_student_enrollments;\n", + "\n", + "CREATE OR REPLACE VIEW edu_dev63853039.silver.vw_student_enrollments AS (\n", + " SELECT \n", + " CONCAT(s.first_name, ' ', s.last_name) \n", + " AS full_name,\n", + " s.email,\n", + " c.course_name,\n", + " c.department,\n", + " e.semester,\n", + " e.grade\n", + " FROM \n", + " edu_dev63853039.silver.enrollments e \n", + " INNER JOIN \n", + " edu_dev63853039.silver.students s\n", + " ON \n", + " e.student_id = s.student_id\n", + " INNER JOIN \n", + " edu_dev63853039.silver.courses c\n", + " ON \n", + " e.course_id = c.course_id\n", + ");" ] }, { "cell_type": "markdown", - "id": "304bd8cb", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "489032f4-79bb-40b0-8811-6184c425096d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.2 — Create a materialized view for department statistics\n", "\n", - "Create a materialized view `edu_dev.gold.vw_department_enrollment_stats` that pre-computes the following per academic department:\n", + "Create a materialized view `edu_dev63853039.gold.vw_department_enrollment_stats` that pre-computes the following per academic department:\n", "\n", "| Output column | Description |\n", "|--------------|-------------|\n", @@ -371,7 +643,7 @@ "| `avg_grade` | Average grade, rounded to 2 decimal places |\n", "| `distinct_students` | Number of unique students enrolled |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"What is the syntax for creating a materialized view in Azure Databricks that aggregates data from two joined tables?\"*\n", "\n", "**Hint:** Use `CREATE MATERIALIZED VIEW`, join `enrollments` with `courses`, and use `COUNT()`, `ROUND(AVG(...), 2)`, and `COUNT(DISTINCT ...)`.\n", @@ -382,19 +654,58 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "589ad9da", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b607649d-b674-4d19-b938-02c58e26869c", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 20" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the edu_dev.gold.vw_department_enrollment_stats materialized view" + "-- TODO: Create the edu_dev63853039.gold.vw_department_enrollment_stats materialized view\n", + "\n", + "CREATE MATERIALIZED VIEW IF NOT EXISTS edu_dev63853039.gold.vw_department_enrollment_stats AS (\n", + " SELECT \n", + " c.department,\n", + " COUNT(*) AS total_enrollments,\n", + " ROUND(AVG(e.grade), 2) AS avg_grade,\n", + " COUNT(DISTINCT e.student_id) AS distinct_students\n", + " FROM \n", + " edu_dev63853039.silver.enrollments e\n", + " INNER JOIN \n", + " edu_dev63853039.silver.courses c\n", + " ON \n", + " e.course_id = c.course_id\n", + " GROUP BY \n", + " c.department\n", + ");" ] }, { "cell_type": "markdown", - "id": "4ab4b66e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "fc765d60-2062-416e-9db2-62509ffdcc14", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "**✅ Test your materialized view** — Run the following query to verify it returns per-department aggregates:\n", "\n", @@ -403,19 +714,42 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "d3b2354e", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f38a40de-a840-4b78-9af6-9230ae7584cb", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "SELECT * FROM edu_dev.gold.vw_department_enrollment_stats;" + "SELECT * FROM edu_dev63853039.gold.vw_department_enrollment_stats;" ] }, { "cell_type": "markdown", - "id": "77ee534b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "fc9a15af-bfb9-4244-b301-23b82b2fdfc8", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Create a Volume and Read Files\n", "\n", @@ -424,14 +758,25 @@ }, { "cell_type": "markdown", - "id": "a8fca075", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "be2a335a-742a-49e5-8b43-5ebaee21e67c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.1 — Create a managed volume\n", "\n", - "Create a managed volume named `raw_files` in the `edu_dev.bronze` schema. This will serve as the landing area for raw data files.\n", + "Create a managed volume named `raw_files` in the `edu_dev63853039.bronze` schema. This will serve as the landing area for raw data files.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a managed volume in Unity Catalog using a SQL statement?\"*\n", "\n", "**Hint:** Use `CREATE VOLUME IF NOT EXISTS ..`." @@ -439,19 +784,43 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "c21a6290", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c67111e9-a135-47e9-a880-9e643608a798", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the raw_files managed volume in edu_dev.bronze" + "-- TODO: Create the raw_files managed volume in edu_dev63853039.bronze\n", + "\n", + "CREATE VOLUME IF NOT EXISTS edu_dev63853039.bronze.raw_files" ] }, { "cell_type": "markdown", - "id": "5252fe12", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d2132d9c-6c9c-462b-abfd-a6be4afbae5c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.2 — Write a CSV file to the volume\n", "\n", @@ -460,9 +829,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "6c348308", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "a047fe3b-0fb4-4ca3-92ac-85789743bd24", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Write a sample student CSV file to the volume — run this cell without modification\n", @@ -478,20 +858,31 @@ "1009,Mei,Lindqvist,mei.lindqvist@lakeside.edu,2023,Mathematics\n", "1010,Jake,OBrien,jake.obrien@lakeside.edu,2020,Computer Science\"\"\"\n", "\n", - "dbutils.fs.put(\"/Volumes/edu_dev/bronze/raw_files/students.csv\", csv_content, overwrite=True)\n", - "print(\"students.csv written to /Volumes/edu_dev/bronze/raw_files/\")" + "dbutils.fs.put(\"/Volumes/edu_dev63853039/bronze/raw_files/students.csv\", csv_content, overwrite=True)\n", + "print(\"students.csv written to /Volumes/edu_dev63853039/bronze/raw_files/\")" ] }, { "cell_type": "markdown", - "id": "6823ffb1", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d7554b95-5c1c-4cd9-8b49-a7d779f40c67", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.3 — Read the CSV from the volume into a DataFrame\n", "\n", - "Read the CSV file you just placed in the volume into a Spark DataFrame and display its contents. The file is at `/Volumes/edu_dev/bronze/raw_files/students.csv`.\n", + "Read the CSV file you just placed in the volume into a Spark DataFrame and display its contents. The file is at `/Volumes/edu_dev63853039/bronze/raw_files/students.csv`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I read a CSV file with a header row from a Unity Catalog volume path in PySpark?\"*\n", "\n", "**Hint:** Use `spark.read.csv(path, header=True, inferSchema=True)` then call `.display()` or `.show()` on the result." @@ -499,18 +890,41 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4f2b6ef8", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "2e2ecc7c-205e-4959-a66e-ece8c2b5a98c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Read /Volumes/edu_dev/bronze/raw_files/students.csv into a Spark DataFrame and display it" + "# TODO: Read /Volumes/edu_dev63853039/bronze/raw_files/students.csv into a Spark DataFrame and display it\n", + "spark.read.csv('/Volumes/edu_dev63853039/bronze/raw_files/students.csv', header=True, inferSchema=True).display()" ] }, { "cell_type": "markdown", - "id": "d9848d3c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f6988ea6-f83f-4725-94ca-e19b2c64a3dc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 5: Create a Reusable SQL Function\n", "\n", @@ -519,12 +933,23 @@ }, { "cell_type": "markdown", - "id": "efc0eb82", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "cf215b86-565c-4e54-ae30-9db6468e76c2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 5.1 — Create a grade classification function\n", "\n", - "Create a SQL scalar function `edu_dev.silver.get_grade_classification` that accepts a `grade DECIMAL(4,2)` and returns a `STRING` letter classification based on the scale used at Lakeside University:\n", + "Create a SQL scalar function `edu_dev63853039.silver.get_grade_classification` that accepts a `grade DECIMAL(4,2)` and returns a `STRING` letter classification based on the scale used at Lakeside University:\n", "\n", "| Grade range | Classification |\n", "|-------------|----------------|\n", @@ -534,7 +959,7 @@ "| ≥ 4.0 | `'D'` |\n", "| < 4.0 | `'F'` |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a SQL scalar function in Unity Catalog that uses a CASE expression to return a letter grade based on a numeric score?\"*\n", "\n", "**Hint:** Use `CREATE FUNCTION ... RETURNS STRING RETURN CASE WHEN ... END`." @@ -542,61 +967,140 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "abd7299f", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "4e00efc2-8578-4d14-9191-aca2527edc74", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the edu_dev.silver.get_grade_classification function" + "-- TODO: Create the edu_dev63853039.silver.get_grade_classification function\n", + "\n", + "CREATE OR REPLACE FUNCTION get_grade_classification(grade\n", + "DECIMAL(4, 2))\n", + "RETURNS STRING\n", + "RETURN CASE\n", + " WHEN grade >= 8.5 THEN 'A'\n", + " WHEN grade >= 7.0 THEN 'B'\n", + " WHEN grade >= 5.5 THEN 'C'\n", + " WHEN grade >= 4.0 THEN 'D'\n", + " ELSE 'F'\n", + "END;" ] }, { "cell_type": "markdown", - "id": "c96c404a", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "a1a7118b-7f8b-4741-a35b-05fcfbca6d46", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 5.2 — Test the function\n", "\n", - "Query `edu_dev.silver.enrollments` and apply your new function to produce a `grade_classification` column for each row. Include: `enrollment_id`, `student_id`, `course_id`, `semester`, `grade`, and `grade_classification`.\n", + "Query `edu_dev63853039.silver.enrollments` and apply your new function to produce a `grade_classification` column for each row. Include: `enrollment_id`, `student_id`, `course_id`, `semester`, `grade`, and `grade_classification`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I call a Unity Catalog user-defined function in a SELECT statement in Azure Databricks?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "60f10718", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "27ab739a-3b97-4770-b159-26d561be1a1c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Query edu_dev.silver.enrollments and call get_grade_classification on the grade column" + "-- TODO: Query edu_dev63853039.silver.enrollments and call get_grade_classification on the grade column\n", + "\n", + "SELECT \n", + " enrollment_id, \n", + " student_id, \n", + " course_id,\n", + " semester,\n", + " grade,\n", + " get_grade_classification(grade) AS grade_classification\n", + "FROM \n", + " edu_dev63853039.silver.enrollments;\n", + " \n" ] }, { "cell_type": "markdown", - "id": "ed144885", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "ccef6d90-975b-442d-b5c3-7d92c3c855d5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 6: DDL Operations\n", "\n", "Unity Catalog objects evolve over time as requirements change. In this exercise, you use `ALTER` statements to:\n", "1. Extend the `students` table with a new column.\n", - "2. Apply governance **tags** to the `edu_dev` catalog." + "2. Apply governance **tags** to the `edu_dev63853039` catalog." ] }, { "cell_type": "markdown", - "id": "c6b30ba9", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "4b8e4332-a4df-4e0e-b126-d0bdae092aec", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 6.1 — Add a column to the students table\n", "\n", - "The student registration system has been updated to capture phone numbers. Add a nullable `phone_number` column of type `STRING` to `edu_dev.silver.students`.\n", + "The student registration system has been updated to capture phone numbers. Add a nullable `phone_number` column of type `STRING` to `edu_dev63853039.silver.students`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I add a new nullable column to an existing Delta table in Azure Databricks using SQL?\"*\n", "\n", "**Hint:** Use `ALTER TABLE`." @@ -604,23 +1108,49 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "85c7df17", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "461fd0b3-e8fd-40cb-8127-ac459821105e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Add a phone_number (STRING) column to edu_dev.silver.students" + "-- TODO: Add a phone_number (STRING) column to edu_dev63853039.silver.students\n", + "\n", + "ALTER TABLE edu_dev63853039.silver.students\n", + "ADD COLUMN phone_number STRING" ] }, { "cell_type": "markdown", - "id": "706ba53d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d7eb1f67-364e-454f-bfb0-16e35961221d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 6.2 — Apply tags to the catalog\n", "\n", - "Apply the following tags to the `edu_dev` catalog to support governance and discoverability:\n", + "Apply the following tags to the `edu_dev63853039` catalog to support governance and discoverability:\n", "\n", "| Tag key | Tag value |\n", "|---------|----------|\n", @@ -628,25 +1158,55 @@ "| `university` | `'lakeside'` |\n", "| `data_classification` | `'internal'` |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I set metadata tags on a Unity Catalog catalog using the ALTER CATALOG statement in Databricks?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "bae85a69", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "681f838d-681a-4e71-bd6d-07f6cbf3a46e", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 39" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Apply the three tags to the edu_dev catalog using ALTER CATALOG ... SET TAGS" + "-- TODO: Apply the three tags to the edu_dev63853039 catalog using ALTER CATALOG ... SET TAGS\n", + "\n", + "ALTER CATALOG edu_dev63853039\n", + "SET TAGS (\n", + " 'environment' = 'education',\n", + " 'university' = 'lakeside',\n", + " 'data_classification' = 'internal'\n", + ")" ] }, { "cell_type": "markdown", - "id": "338f23f8", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "4acd1fc3-97b3-4d32-b730-26a9bd7b5ffa", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 6.3 — Verify your work\n", "\n", @@ -655,58 +1215,117 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "178ae537", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f51485bd-a21a-4d69-b815-80ae0a822029", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Show all tables and views in the silver schema\n", - "SHOW TABLES IN edu_dev.silver;" + "SHOW TABLES IN edu_dev63853039.silver;" ] }, { "cell_type": "code", - "execution_count": null, - "id": "24d433ae", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "68b77758-4e42-48d6-b6e5-4d310a89bf7b", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Describe the students table — verify the new phone_number column is present\n", - "DESCRIBE TABLE edu_dev.silver.students;" + "DESCRIBE TABLE edu_dev63853039.silver.students;" ] }, { "cell_type": "code", - "execution_count": null, - "id": "44e96056", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "bbadd540-0ecf-467b-aa16-7314ad053899", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Show catalog details\n", - "DESCRIBE CATALOG EXTENDED edu_dev;" + "DESCRIBE CATALOG EXTENDED edu_dev63853039;" ] }, { "cell_type": "code", - "execution_count": null, - "id": "28dab7de", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "47b34975-b8fb-4526-ba4f-8325f5c218c2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql \n", "-- Query tags via information schema (structured way)\n", "SELECT *\n", "FROM system.information_schema.catalog_tags\n", - "WHERE catalog_name = 'edu_dev';" + "WHERE catalog_name = 'edu_dev63853039';" ] }, { "cell_type": "markdown", - "id": "5b9e4d0e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c58cba83-b616-4c8f-b8fe-6fd2bdb0217f", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "---\n", "\n", @@ -719,10 +1338,40 @@ } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": { + "hardware": { + "accelerator": null, + "gpuPoolId": null, + "memory": null + }, + "software": { + "pinSparkToX86": null + } + }, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": -1, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "03-create-and-organize-objects-in-unity-catalog", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/04-secure-unity-catalog-objects.ipynb b/Allfiles/04-secure-unity-catalog-objects.ipynb index af16699..8a3ed0b 100644 --- a/Allfiles/04-secure-unity-catalog-objects.ipynb +++ b/Allfiles/04-secure-unity-catalog-objects.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "246cbfa5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "89d52cf3-35b6-48d5-8389-3b7bfd0ddf16", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Access Control\n", "\n", @@ -14,15 +22,23 @@ }, { "cell_type": "markdown", - "id": "7109cd52", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "c27b7625-8cf2-4dc7-b412-1a26451f9b60", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.1 — Create the catalog, schema, and customers table\n", "\n", "The cell below sets up all the objects needed for this exercise:\n", - "- Creates a catalog called `retail_catalog` to hold all NorthMart data assets.\n", + "- Creates a catalog called `retail_catalog63874195` to hold all NorthMart data assets.\n", "- Creates a schema called `security_lab` inside it.\n", - "- Creates a managed Delta table `retail_catalog.security_lab.customers` and populates it with six sample customer records. These rows will be used throughout the rest of this exercise to demonstrate row filtering and column masking.\n", + "- Creates a managed Delta table `retail_catalog63874195.security_lab.customers` and populates it with six sample customer records. These rows will be used throughout the rest of this exercise to demonstrate row filtering and column masking.\n", "\n", "| customer_id | customer_name | email | region | total_spend | tier |\n", "|-------------|-----------------|-------------------------|--------|-------------|----------|\n", @@ -38,9 +54,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "cb0ec373", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c3f1f958-9a9f-4ca6-8607-b1e8dc9e28a5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -57,7 +84,7 @@ "print (f\"Storage root: {storage_root}\")\n", "\n", "spark.sql(f\"\"\"\n", - " CREATE CATALOG IF NOT EXISTS retail_catalog\n", + " CREATE CATALOG IF NOT EXISTS retail_catalog63874195\n", " MANAGED LOCATION '{storage_root}'\n", " COMMENT 'Catalog for NorthMart Retail data assets'\n", "\"\"\")" @@ -65,16 +92,28 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "6bbd6143", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "05a04dbf-9f24-4421-a355-4d91c904de4d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "CREATE SCHEMA IF NOT EXISTS retail_catalog.security_lab\n", + "CREATE SCHEMA IF NOT EXISTS retail_catalog63874195.security_lab\n", " COMMENT 'Schema used for the security lab exercises';\n", "\n", - "CREATE OR REPLACE TABLE retail_catalog.security_lab.customers AS\n", + "CREATE OR REPLACE TABLE retail_catalog63874195.security_lab.customers AS\n", "SELECT * FROM VALUES\n", " (1, 'Alice Johnson', 'alice@northmart.com', 'North', 5200.00, 'Standard'),\n", " (2, 'Bob Martinez', 'bob@northmart.com', 'South', 8100.00, 'Premium'),\n", @@ -87,18 +126,26 @@ }, { "cell_type": "markdown", - "id": "e28ea3dd", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "034db656-bd70-4761-bab3-3178e6cc65c2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.2 — Grant permissions to the retail-analysts group\n", "\n", "The `retail-analysts` group needs to query the `customers` table. Grant the necessary privileges at all required levels of the Unity Catalog hierarchy:\n", "\n", - "- `USE CATALOG` on `retail_catalog`\n", - "- `USE SCHEMA` on `retail_catalog.security_lab`\n", - "- `SELECT` on `retail_catalog.security_lab.customers`\n", + "- `USE CATALOG` on `retail_catalog63874195`\n", + "- `USE SCHEMA` on `retail_catalog63874195.security_lab`\n", + "- `SELECT` on `retail_catalog63874195.security_lab.customers`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"What SQL grants are needed to allow a Databricks group to SELECT from a Unity Catalog table?\"*\n", "\n", "**Hint:** Use `GRANT ON TO \\`retail-analysts\\``." @@ -106,25 +153,49 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "7d849128", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "3452e891-b644-4b57-b582-f3d462fff5df", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Verify the granted permissions" + "-- TODO: Verify the granted permissions\n", + "\n", + "GRANT ALL PRIVILEGES ON CATALOG retail_catalog63874195 TO `retail-analysts`;\n", + "GRANT USE SCHEMA ON SCHEMA retail_catalog63874195.security_lab TO `retail-analysts`;\n", + "GRANT SELECT ON TABLE retail_catalog63874195.security_lab.customers TO `retail-analysts`;" ] }, { "cell_type": "markdown", - "id": "bed7d1e5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "8105adca-4565-4a47-ab58-f616e80015ce", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.3 — Verify the granted permissions\n", "\n", "Verify that the privileges were correctly assigned. Use `SHOW GRANTS` to display all privileges on the `customers` table.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I use SHOW GRANTS to verify permissions on a Unity Catalog table?\"*\n", "\n", "**Expected result:** You should see `retail-analysts` listed with the `SELECT` privilege." @@ -132,19 +203,41 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4a4dc08c", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "0403d8f2-86c4-4c05-90d3-9b0d2bedad8e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Show all grants on retail_catalog.security_lab.customers" + "-- TODO: Show all grants on retail_catalog63874195.security_lab.customers\n", + "\n", + "SHOW GRANTS ON TABLE retail_catalog63874195.security_lab.customers" ] }, { "cell_type": "markdown", - "id": "54004c07", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "58417853-8983-4f8d-8e9d-76692dd1315a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Row Filtering\n", "\n", @@ -155,18 +248,26 @@ }, { "cell_type": "markdown", - "id": "5c40704e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "3fe59099-d4a5-44e0-9117-a86580226d04", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.1 — Create a row filter function\n", "\n", - "Create a SQL function `retail_catalog.security_lab.region_filter` that takes a `region` parameter of type `STRING` and returns `TRUE` if:\n", + "Create a SQL function `retail_catalog63874195.security_lab.region_filter` that takes a `region` parameter of type `STRING` and returns `TRUE` if:\n", "- the `region` is `'North'`, **OR**\n", "- the current user is your own account (replace `your@email.com` with your actual email).\n", "\n", "This means: unless you are the privileged user, only `North` region rows will be visible.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a row filter function in Databricks Unity Catalog that uses current_user() to control access?\"*\n", "\n", "**Hint:** Use `CREATE OR REPLACE FUNCTION ... RETURN `." @@ -174,26 +275,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1aa7f33a", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "3aac1aca-fcce-49f6-819d-5a0f1388b8f8", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the region_filter function in retail_catalog.security_lab\n", - "-- Replace 'your@email.com' with your actual Databricks user email" + "-- TODO: Create the region_filter function in retail_catalog63874195.security_lab\n", + "-- Replace 'your@email.com' with your actual Databricks user email\n", + "\n", + "CREATE OR REPLACE FUNCTION retail_catalog63874195.security_lab.region_filter(region STRING)\n", + "RETURNS BOOLEAN \n", + "RETURN (region = 'North' OR current_user() = 'user1-63874195@lodsprodmca.onmicrosoft.com'\n", + ");" ] }, { "cell_type": "markdown", - "id": "f8c13df8", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "b5cd7b09-be18-454c-84f5-77333b83a39c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.2 — Apply the row filter to the customers table\n", "\n", "Attach the `region_filter` function to the `customers` table so that it is automatically applied to every query against that table.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I set a row filter on a Unity Catalog table in Azure Databricks?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... SET ROW FILTER ON ()`." @@ -201,19 +327,42 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "44a27dc5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "918736d0-4782-4546-8e1f-e5aca51f07be", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Apply region_filter to retail_catalog.security_lab.customers on the region column" + "-- TODO: Apply region_filter to retail_catalog63874195.security_lab.customers on the region column\n", + "\n", + "ALTER TABLE retail_catalog63874195.security_lab.customers\n", + "SET ROW FILTER retail_catalog63874195.security_lab.region_filter ON(region);\n" ] }, { "cell_type": "markdown", - "id": "064a0a5c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "4e1c00df-b483-4744-8929-78b7c7c2333c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.3 — Test the row filter\n", "\n", @@ -221,42 +370,72 @@ "\n", "If you had logged in as someone else (not your email), you would only see the 2 `North` region rows.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How can I verify which user is currently running a query in Databricks using SQL?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "d8081299", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b66d2c2b-7512-49fe-b366-7a77037b3dc5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Verify the current user\n", "SELECT current_user();\n", "\n", - "-- TODO: Query retail_catalog.security_lab.customers and observe the results" + "-- TODO: Query retail_catalog63874195.security_lab.customers and observe the results\n", + "\n", + "SELECT * FROM retail_catalog63874195.security_lab.customers;\n" ] }, { "cell_type": "markdown", - "id": "07c44057", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "cf9eaf07-ce75-46fc-99bf-54de53d42e6a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "➡️ Tip: update the email address in the row filter function you created earlier, and test again. Observe the output when querying the customers table." ] }, { "cell_type": "markdown", - "id": "e2ae227c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "37dd7f85-a769-4f3c-a532-fe28396f0d44", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.4 — Remove the row filter\n", "\n", "Now remove the row filter from the `customers` table. After removing it, query the table again to confirm **all 6 rows** are now returned regardless of user context.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I remove a row filter from a Unity Catalog table?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... DROP ROW FILTER`." @@ -264,21 +443,44 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1ade042c", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "719f44ea-4111-498b-9a8c-40e312c45dc5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Remove the row filter from retail_catalog.security_lab.customers\n", + "-- TODO: Remove the row filter from retail_catalog63874195.security_lab.customers\n", + "ALTER TABLE retail_catalog63874195.security_lab.customers DROP ROW FILTER;\n", "\n", - "-- TODO: Query the table to confirm all 6 rows are visible" + " \n", + "-- TODO: Create the mask_email function in retail_catalog63874195.security_lab\n", + "SELECT * FROM retail_catalog63874195.security_lab.customers;" ] }, { "cell_type": "markdown", - "id": "32632e1e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1a9021db-4a77-4d9b-8cce-a4211002e6be", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Column Masking\n", "\n", @@ -287,16 +489,24 @@ }, { "cell_type": "markdown", - "id": "60241d37", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "df68fcab-38e8-44e9-998a-4e0675f18fd4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.1 — Create a column mask function\n", "\n", - "Create a SQL function `retail_catalog.security_lab.mask_email` that takes an `email` parameter of type `STRING` and:\n", + "Create a SQL function `retail_catalog63874195.security_lab.mask_email` that takes an `email` parameter of type `STRING` and:\n", "- Returns the **full email** if the current user is your own account.\n", "- Returns a **masked value** (e.g. `'***@***.com'`) for all other users.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a column masking function in Databricks Unity Catalog that uses current_user() to decide whether to return the real or masked value?\"*\n", "\n", "**Hint:** Use a `CASE WHEN current_user() = '...' THEN email ELSE '***@***.com' END` pattern inside the function body." @@ -304,26 +514,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "f7e64496", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "5ce679c5-046b-475c-b814-024440f12b64", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create the mask_email function in retail_catalog.security_lab\n", - "-- Replace 'your@email.com' with your actual Databricks user email" + "-- TODO: Create the mask_email function in retail_catalog63874195.security_lab\n", + "-- Replace 'your@email.com' with your actual Databricks user email\n", + "\n", + "CREATE OR REPLACE FUNCTION retail_catalog63874195.security_lab.mask_email(email STRING)\n", + "RETURNS STRING\n", + "RETURN CASE WHEN current_user = 'user1-63874195@lodsprodmca.onmicrosoft.com'\n", + "THEN email ELSE '***@***.com' END;" ] }, { "cell_type": "markdown", - "id": "f0f0d2f8", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "da0d4fa7-f5a9-45af-af89-008f937e8fdf", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.2 — Apply the column mask to the email column\n", "\n", "Apply the `mask_email` function as a column mask on the `email` column of the `customers` table.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I set a column mask on a Unity Catalog table column in Azure Databricks using SQL?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... ALTER COLUMN email SET MASK `." @@ -331,19 +566,42 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "7c3bd607", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c83e8bdd-f8e8-43da-afce-59d89b004af6", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Apply mask_email to the email column of retail_catalog.security_lab.customers" + "-- TODO: Apply mask_email to the email column of retail_catalog63874195.security_lab.customers\n", + "\n", + "ALTER TABLE retail_catalog63874195.security_lab.customers\n", + "ALTER COLUMN email SET MASK retail_catalog63874195.security_lab.mask_email" ] }, { "cell_type": "markdown", - "id": "06673f52", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "fdc63555-9014-4165-8fa4-c0807909b838", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.3 — Test the column mask\n", "\n", @@ -354,25 +612,46 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "9557db8c", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "16ae39da-525e-476d-a804-8a282c6b2be6", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Query retail_catalog.security_lab.customers and observe the email column" + "-- TODO: Query retail_catalog63874195.security_lab.customers and observe the email column\n", + "SELECT * FROM retail_catalog63874195.security_lab.customers" ] }, { "cell_type": "markdown", - "id": "b268d979", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6a2377ea-ab09-43ac-a29f-4c34f74c3c03", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.4 — Remove the column mask\n", "\n", "Remove the column mask from the `email` column, then query the table again to confirm emails are now fully visible to all users.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I drop a column mask from a Unity Catalog table column?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... ALTER COLUMN email DROP MASK`." @@ -380,21 +659,43 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "7d425b20", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c329740e-d69e-40f4-831e-fd5ccf6b4ee1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Remove the column mask from the email column\n", - "\n", - "-- TODO: Query the table to confirm email addresses are now unmasked" + "ALTER TABLE retail_catalog63874195.security_lab.customers\n", + "ALTER COLUMN email DROP MASK;\n", + "-- TODO: Query the table to confirm email addresses are now unmasked\n", + "SELECT * FROM retail_catalog63874195.security_lab.customers" ] }, { "cell_type": "markdown", - "id": "75a2120a", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "b216559c-c988-4f7a-a738-007fbf60c949", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Azure Key Vault Secrets\n", "\n", @@ -405,14 +706,22 @@ }, { "cell_type": "markdown", - "id": "bc54e26e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "d5406ec4-8d3a-4323-92d5-f3b2dc187922", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.1 — List available secrets in the scope\n", "\n", "List all secrets available in the `retail-kv-scope` secret scope to confirm that `loyalty-api-key` is accessible.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I list the secrets in a Databricks secret scope using dbutils?\"*\n", "\n", "**Hint:** Use `dbutils.secrets.list(\"scope-name\")`. This returns key names only — never secret values." @@ -420,18 +729,38 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "7b36e969", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "981c6f36-d0c6-4307-8a59-18a755dd7c88", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: List all secrets in the retail-kv-scope and display the output" + "# TODO: List all secrets in the retail-kv-scope and display the output\n", + "dbutils.secrets.list(\"retail-kv-scope\")" ] }, { "cell_type": "markdown", - "id": "49ba2ffb", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "575831f4-a306-4386-8879-33ae67caddd4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.2 — Retrieve the API key securely in Python\n", "\n", @@ -439,50 +768,95 @@ "\n", "Observe how Azure Databricks **automatically redacts** the value in the notebook output — you will see `[REDACTED]` instead of the actual key.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I retrieve a secret from a Databricks secret scope using dbutils.secrets.get and why is the output redacted?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "47000f12", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "3b519b1b-4b98-49aa-90e9-0895863d20b1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# TODO: Retrieve the loyalty-api-key from retail-kv-scope\n", - "# TODO: Print the api_key variable and observe that the output is [REDACTED]" + "\n", + "api_key = dbutils.secrets.get(\"retail-kv-scope\", \"loyalty-api-key\")\n", + "\n", + "# TODO: Print the api_key variable and observe that the output is [REDACTED]\n", + "print(api_key)" ] }, { "cell_type": "markdown", - "id": "5d8d2940", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "2062d8dd-542c-442e-b927-bbd8a55aef43", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.3 — Use the secret to simulate a connection string\n", "\n", "Construct a connection string that embeds the API key, simulating how it would be used to connect to NorthMart's loyalty platform. Print the connection string and observe that the secret value is still redacted.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I safely use a Databricks secret inside an f-string for a connection string without exposing the value?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "61c36ca0", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "9b94e389-73ac-49b9-b207-28af7d2f5492", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# TODO: Build a connection string using the api_key variable\n", "# Example format: \"https://loyalty.northmart.com/api?key=\"\n", - "# TODO: Print the connection string" + "\n", + "connection_string = f\"https://loyalty.northmart.com/api?key={api_key}\"\n", + "# TODO: Print the connection string\n", + "print(connection_string)" ] }, { "cell_type": "markdown", - "id": "0b3f910f", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7a5b1dfc-2b08-4bbc-b0fd-432b64ead4bc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.4 — Retrieve the secret using SQL\n", "\n", @@ -490,25 +864,46 @@ "\n", "Note that the output is also redacted in SQL results.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I use the secret() function in Databricks SQL to retrieve a secret from a scope?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "c7cb70ca", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "27afa2d8-cb47-4f6b-a840-08ac7aee8089", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Use the secret() SQL function to retrieve loyalty-api-key from retail-kv-scope" + "-- TODO: Use the secret() SQL function to retrieve loyalty-api-key from retail-kv-scope\n", + "SELECT secret('retail-kv-scope','loyalty-api-key')" ] }, { "cell_type": "markdown", - "id": "e8c931f1", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "64d34933-9eea-4f18-82fd-43cec5c39427", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Cleanup\n", "\n", @@ -517,28 +912,61 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "0e4e500a", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "816389ad-198b-4cb8-b084-0fa4573d65ca", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Remove all lab objects\n", - "ALTER TABLE retail_catalog.security_lab.customers DROP ROW FILTER;\n", - "ALTER TABLE retail_catalog.security_lab.customers ALTER COLUMN email DROP MASK;\n", - "DROP TABLE IF EXISTS retail_catalog.security_lab.customers;\n", - "DROP FUNCTION IF EXISTS retail_catalog.security_lab.region_filter;\n", - "DROP FUNCTION IF EXISTS retail_catalog.security_lab.mask_email;\n", - "DROP SCHEMA IF EXISTS retail_catalog.security_lab;\n", - "DROP CATALOG IF EXISTS retail_catalog CASCADE;" + "ALTER TABLE retail_catalog63874195.security_lab.customers DROP ROW FILTER;\n", + "ALTER TABLE retail_catalog63874195.security_lab.customers ALTER COLUMN email DROP MASK;\n", + "DROP TABLE IF EXISTS retail_catalog63874195.security_lab.customers;\n", + "DROP FUNCTION IF EXISTS retail_catalog63874195.security_lab.region_filter;\n", + "DROP FUNCTION IF EXISTS retail_catalog63874195.security_lab.mask_email;\n", + "DROP SCHEMA IF EXISTS retail_catalog63874195.security_lab;\n", + "DROP CATALOG IF EXISTS retail_catalog63874195 CASCADE;" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 4629336142089078, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "04-secure-unity-catalog-objects", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/05-govern-unity-catalog-objects.ipynb b/Allfiles/05-govern-unity-catalog-objects.ipynb index f8fbe1f..2d62b00 100644 --- a/Allfiles/05-govern-unity-catalog-objects.ipynb +++ b/Allfiles/05-govern-unity-catalog-objects.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "5ce4abca", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "fc2d7e7e-0450-40fe-8b20-75c03074bcf2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Set Up the AutoSphere Data Platform\n", "\n", @@ -14,8 +22,16 @@ }, { "cell_type": "markdown", - "id": "dc2a60c0", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "8f0966c1-e3fc-458e-9b49-7bf1b5a9c12e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Setup — Create the catalog, schema, and tables\n", "\n", @@ -31,15 +47,26 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4ad1a39f", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "ba5fe687-96bc-4c93-b771-f11e5c051e4f", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 3" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", "# Use the current catalog to reliably find the workspace default catalog,\n", "# regardless of its naming convention.\n", - "default_catalog = spark.catalog.currentCatalog()\n", + "default_catalog = [c.catalog for c in spark.sql(\"SHOW CATALOGS\").collect() if c.catalog.startswith(\"adb_\")][0]\n", "\n", "storage_root = (\n", " spark.sql(f\"DESCRIBE CATALOG EXTENDED {default_catalog}\")\n", @@ -50,7 +77,7 @@ "print (f\"Storage root: {storage_root}\")\n", "\n", "spark.sql(f\"\"\"\n", - " CREATE CATALOG IF NOT EXISTS automotive_catalog\n", + " CREATE CATALOG IF NOT EXISTS automotive_catalog2\n", " MANAGED LOCATION '{storage_root}'\n", " COMMENT 'Connected vehicle platform catalog for AutoSphere AG — stores registration, telemetry, and service data.'\n", "\"\"\")" @@ -58,52 +85,84 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "8e55df83", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "eec52119-9b86-4f40-8097-fa9bdf875a81", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 4" + } + }, "outputs": [], "source": [ - "%sql\n", - "CREATE SCHEMA IF NOT EXISTS automotive_catalog.governance_lab\n", - " COMMENT 'Governance lab schema for hands-on Unity Catalog exercises.';\n", + "spark.sql(\"USE CATALOG automotive_catalog2\")\n", + "\n", + "spark.sql(\"\"\"\n", + " CREATE SCHEMA IF NOT EXISTS automotive_catalog2.governance_lab\n", + " COMMENT 'Governance lab schema for hands-on Unity Catalog exercises.'\n", + "\"\"\")\n", + "print(\"✓ Schema created\")\n", "\n", - "-- Create the customer_registrations table\n", - "CREATE TABLE IF NOT EXISTS automotive_catalog.governance_lab.customer_registrations\n", + "spark.sql(\"\"\"\n", + " CREATE TABLE IF NOT EXISTS automotive_catalog2.governance_lab.customer_registrations\n", " COMMENT 'One row per registered AutoSphere customer and their associated vehicle.'\n", - "AS SELECT * FROM VALUES\n", - " (1, 'Lukas Bauer', 'lukas.bauer@autosphere.de', 'DE-LB-4821', 'DE', 'VH-001'),\n", - " (2, 'Sophie Martin', 'sophie.martin@autosphere.fr', 'FR-SM-9034', 'FR', 'VH-002'),\n", - " (3, 'James Clarke', 'james.clarke@autosphere.uk', 'UK-JC-1172', 'UK', 'VH-003'),\n", - " (4, 'Yuki Tanaka', 'yuki.tanaka@autosphere.jp', 'JP-YT-5561', 'JP', 'VH-004'),\n", - " (5, 'Maria Santos', 'maria.santos@autosphere.pt', 'PT-MS-8843', 'PT', 'VH-005'),\n", - " (6, 'Carlos Rivera', 'carlos.rivera@autosphere.es', 'ES-CR-3309', 'ES', 'VH-006')\n", - "AS t(customer_id, full_name, email, driver_license_no, country, vehicle_id);\n", - "\n", - "-- Create the vehicle_telemetry table\n", - "CREATE TABLE IF NOT EXISTS automotive_catalog.governance_lab.vehicle_telemetry (\n", - " event_id BIGINT COMMENT 'Unique telemetry event identifier',\n", - " vehicle_id STRING,\n", - " event_time TIMESTAMP,\n", - " speed_kmh INT COMMENT 'Instantaneous vehicle speed in kilometres per hour',\n", - " battery_level_pct INT COMMENT 'State of charge as a percentage (0–100)',\n", - " latitude DOUBLE,\n", - " longitude DOUBLE,\n", - " country STRING\n", - ");\n", - "\n", - "INSERT INTO automotive_catalog.governance_lab.vehicle_telemetry VALUES\n", - " (1, 'VH-001', TIMESTAMP '2026-03-01 08:15:00', 112, 78, 48.8566, 2.3522, 'DE'),\n", - " (2, 'VH-002', TIMESTAMP '2026-03-01 08:16:00', 95, 55, 51.5074, -0.1278, 'FR'),\n", - " (3, 'VH-003', TIMESTAMP '2026-03-01 08:17:00', 130, 91, 40.4168, -3.7038, 'UK'),\n", - " (4, 'VH-004', TIMESTAMP '2026-03-01 08:18:00', 88, 34, 35.6762, 139.6503, 'JP'),\n", - " (5, 'VH-005', TIMESTAMP '2026-03-01 08:19:00', 105, 62, 38.7223, -9.1393, 'PT'),\n", - " (6, 'VH-006', TIMESTAMP '2026-03-01 08:20:00', 77, 47, 41.3851, 2.1734, 'ES');" + " AS SELECT * FROM VALUES\n", + " (1, 'Lukas Bauer', 'lukas.bauer@autosphere.de', 'DE-LB-4821', 'DE', 'VH-001'),\n", + " (2, 'Sophie Martin', 'sophie.martin@autosphere.fr', 'FR-SM-9034', 'FR', 'VH-002'),\n", + " (3, 'James Clarke', 'james.clarke@autosphere.uk', 'UK-JC-1172', 'UK', 'VH-003'),\n", + " (4, 'Yuki Tanaka', 'yuki.tanaka@autosphere.jp', 'JP-YT-5561', 'JP', 'VH-004'),\n", + " (5, 'Maria Santos', 'maria.santos@autosphere.pt', 'PT-MS-8843', 'PT', 'VH-005'),\n", + " (6, 'Carlos Rivera', 'carlos.rivera@autosphere.es', 'ES-CR-3309', 'ES', 'VH-006')\n", + " AS t(customer_id, full_name, email, driver_license_no, country, vehicle_id)\n", + "\"\"\")\n", + "print(\"✓ customer_registrations table created\")\n", + "\n", + "spark.sql(\"\"\"\n", + " CREATE TABLE IF NOT EXISTS automotive_catalog2.governance_lab.vehicle_telemetry (\n", + " event_id BIGINT COMMENT 'Unique telemetry event identifier',\n", + " vehicle_id STRING,\n", + " event_time TIMESTAMP,\n", + " speed_kmh INT COMMENT 'Instantaneous vehicle speed in kilometres per hour',\n", + " battery_level_pct INT COMMENT 'State of charge as a percentage (0-100)',\n", + " latitude DOUBLE,\n", + " longitude DOUBLE,\n", + " country STRING\n", + " )\n", + "\"\"\")\n", + "print(\"✓ vehicle_telemetry table created\")\n", + "\n", + "spark.sql(\"\"\"\n", + " INSERT INTO automotive_catalog2.governance_lab.vehicle_telemetry VALUES\n", + " (1, 'VH-001', TIMESTAMP '2026-03-01 08:15:00', 112, 78, 48.8566, 2.3522, 'DE'),\n", + " (2, 'VH-002', TIMESTAMP '2026-03-01 08:16:00', 95, 55, 51.5074, -0.1278, 'FR'),\n", + " (3, 'VH-003', TIMESTAMP '2026-03-01 08:17:00', 130, 91, 40.4168, -3.7038, 'UK'),\n", + " (4, 'VH-004', TIMESTAMP '2026-03-01 08:18:00', 88, 34, 35.6762, 139.6503, 'JP'),\n", + " (5, 'VH-005', TIMESTAMP '2026-03-01 08:19:00', 105, 62, 38.7223, -9.1393, 'PT'),\n", + " (6, 'VH-006', TIMESTAMP '2026-03-01 08:20:00', 77, 47, 41.3851, 2.1734, 'ES')\n", + "\"\"\")\n", + "print(\"✓ Telemetry data inserted\")\n", + "print(\"\\nAll objects created successfully in automotive_catalog2.governance_lab\")" ] }, { "cell_type": "markdown", - "id": "7bd11ac1", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "2e01185f-414b-4d54-b97c-9f5e65a400e3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Tag Data Assets for Governance\n", "\n", @@ -114,8 +173,16 @@ }, { "cell_type": "markdown", - "id": "00c87158", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "f010d9b9-e1d9-4220-a2e9-260147c038cc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.1 — Add table-level tags\n", "\n", @@ -124,7 +191,7 @@ "- `domain` = `customer`\n", "- `data_classification` = `confidential`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I add key-value tags to a Unity Catalog table in Databricks SQL?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... SET TAGS ('key' = 'value', 'key2' = 'value2')`." @@ -132,20 +199,45 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "ae5833f4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "ce6078ba-dfd5-4ee8-a338-e002d6d66044", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Add table-level tags to automotive_catalog.governance_lab.customer_registrations\n", + "\n", + "ALTER TABLE automotive_catalog2.governance_lab.customer_registrations SET TAGS (\n", + " 'domain'= 'customer',\n", + " 'data_classification' = 'confidential'\n", + ")\n", "-- Tags: domain = customer, data_classification = confidential" ] }, { "cell_type": "markdown", - "id": "8d96e379", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "f7033da3-5c4d-4a2e-b328-d41a9d241cd2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.2 — Add column-level PII tags\n", "\n", @@ -154,7 +246,7 @@ "- Column `email` → `pii` = `email`\n", "- Column `driver_license_no` → `pii` = `driver_license`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I add a tag to a specific column in a Unity Catalog table using SQL in Databricks?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... ALTER COLUMN SET TAGS (...)`." @@ -162,56 +254,117 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "32304672", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "bd034de2-6011-4bca-9665-9d80c93b2b4a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Tag the email column with pii = email\n", - "\n", - "-- TODO: Tag the driver_license_no column with pii = driver_license" + "ALTER TABLE automotive_catalog2.governance_lab.customer_registrations ALTER COLUMN email SET TAGS ('pii'='email');\n", + "-- TODO: Tag the driver_license_no column with pii = driver_license\n", + "ALTER TABLE automotive_catalog2.governance_lab.customer_registrations ALTER COLUMN driver_license_no SET TAGS ('pii' = 'driver_license')" ] }, { "cell_type": "markdown", - "id": "154ded38", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "856185cd-26a0-4b02-8173-79db505ff9be", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.3 — Verify the tags\n", "\n", "Use `system.information_schema.table_tags` to confirm the table-level tags have been applied. Then describe the columns to confirm the column tags are present. Use `system.information_schema.column_tags` for this.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How can I verify tags on a Unity Catalog table and its columns in Databricks SQL?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "d6913f7e", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "ca3a2ff1-f7cc-440a-8211-41715342f06d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Show table-level tags for customer_registrations" + "-- Show table-level tags for customer_registrations\n", + "SELECT * FROM system.information_schema.table_tags \n", + "WHERE catalog_name = 'automotive_catalog2'\n", + "AND schema_name = 'governance_lab'\n", + "AND table_name = 'customer_registrations'" ] }, { "cell_type": "code", - "execution_count": null, - "id": "2ec89502", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b6cd9020-36c0-4ebc-900e-c95a7e47dfa2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Show column-level tags for customer_registrations" + "-- Show column-level tags for customer_registrations\n", + "SELECT * FROM system.information_schema.column_tags\n", + "WHERE catalog_name = 'automotive_catalog2'\n", + "AND schema_name = 'governance_lab'\n", + "AND table_name = 'customer_registrations';" ] }, { "cell_type": "markdown", - "id": "660110e6", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "93dbbb3f-50a1-4c8b-841e-7172ed78b2e3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Configure Data Retention\n", "\n", @@ -226,8 +379,16 @@ }, { "cell_type": "markdown", - "id": "0ed2394b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "57c148b2-bb9b-4f01-9c4e-fce12f7003dc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.1 — Configure retention properties\n", "\n", @@ -236,7 +397,7 @@ "- `delta.logRetentionDuration` = `interval 14 days`\n", "- `delta.deletedFileRetentionDuration` = `interval 14 days`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I set Delta Lake retention duration properties on a Unity Catalog table in Databricks?\"*\n", "\n", "**Hint:** Use `ALTER TABLE ... SET TBLPROPERTIES (...)`." @@ -244,20 +405,44 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "23ccba01", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "fc3bc7ee-e30d-4aeb-89eb-5dd17d15840c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Set logRetentionDuration and deletedFileRetentionDuration to 14 days\n", - "-- on automotive_catalog.governance_lab.vehicle_telemetry" + "-- on automotive_catalog.governance_lab.vehicle_telemetry\n", + "ALTER TABLE automotive_catalog2.governance_lab.vehicle_telemetry SET TBLPROPERTIES (\n", + " delta.logRetentionDuration = '14 days',\n", + " delta.deletedFileRetentionDuration = '14 days'\n", + ")\n" ] }, { "cell_type": "markdown", - "id": "98e8791e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "acca0119-a28b-41ef-a6b3-0f18c647fdbe", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.2 — Simulate a GDPR deletion request\n", "\n", @@ -265,27 +450,49 @@ "\n", "After deletion, verify the records are gone by querying the table.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I delete specific rows from a Delta Lake table in Databricks SQL?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "fcee14fb", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d07d6dde-e3c3-4408-a17f-114514c659c5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Delete all rows for vehicle_id = 'VH-003' from vehicle_telemetry\n", - "\n", - "-- TODO: Verify the deletion by querying the table" + "DELETE FROM automotive_catalog2.governance_lab.vehicle_telemetry\n", + "WHERE vehicle_id = 'VH-003'; \n", + "-- TODO: Verify the deletion by querying the table\n", + "SELECT * FROM automotive_catalog2.governance_lab.vehicle_telemetry;" ] }, { "cell_type": "markdown", - "id": "7ad2c419", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "98eeea53-6b66-40fd-8e07-6501177688cd", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.3 — Run VACUUM to purge deleted data files\n", "\n", @@ -293,7 +500,7 @@ "\n", "Use the default retention of **168 hours (7 days)**. This is safe, works on Serverless compute, and is appropriate for a lab environment.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I run VACUUM on a Delta Lake table in Databricks, and what does the RETAIN option do?\"*\n", "\n", "**Note:** On Serverless compute, overriding the minimum retention duration is not supported. Always specify a `RETAIN` value of at least 168 hours, or omit `RETAIN` entirely to use the default." @@ -301,26 +508,47 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1f9654e4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "14f4a0bc-3a2b-4fe3-971d-84b1fbe63236", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Run VACUUM on automotive_catalog.governance_lab.vehicle_telemetry\n", - "-- Use RETAIN 168 HOURS (7 days) or omit RETAIN to use the default" + "VACUUM automotive_catalog2.governance_lab.vehicle_telemetry RETAIN 168 HOURS\n", + "-- Use RETAIN 168 HOURS (7 days) or omit RETAIN to use the default\n" ] }, { "cell_type": "markdown", - "id": "5b382bbd", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7bab7d93-fc33-4715-aa1c-02946f51cee3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.4 — Enable predictive optimization\n", "\n", "Enable **predictive optimization** on the `governance_lab` schema so that future `VACUUM` and `OPTIMIZE` maintenance runs automatically without manual scheduling.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I enable predictive optimization on a Unity Catalog schema in Databricks?\"*\n", "\n", "**Hint:** Use `ALTER SCHEMA ... ENABLE PREDICTIVE OPTIMIZATION`." @@ -328,37 +556,67 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "881c3b88", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "e0abc3fe-e693-44d2-9ff1-9afdbcd6b82b", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Enable predictive optimization on automotive_catalog.governance_lab" + "-- TODO: Enable predictive optimization on automotive_catalog.governance_lab\n", + "\n", + "ALTER SCHEMA automotive_catalog2.governance_lab ENABLE PREDICTIVE OPTIMIZATION;" ] }, { "cell_type": "markdown", - "id": "c280c455", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7e9ec799-d1fe-44c5-856d-49cf30346a5a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Query Data Lineage Programmatically\n", "\n", - "> 📋 **Before running this exercise**: If you haven't already done so, pause here and follow the **Catalog Explorer lineage steps** in the lab setup instructions to explore lineage visually in the UI. Then return here to query lineage data using SQL.\n", + "> \uD83D\uDCCB **Before running this exercise**: If you haven't already done so, pause here and follow the **Catalog Explorer lineage steps** in the lab setup instructions to explore lineage visually in the UI. Then return here to query lineage data using SQL.\n", "\n", "Unity Catalog captures all read and write events and stores them in the `system.access.table_lineage` system table. This lets you programmatically answer questions like: *\"Which tables in our automotive catalog were most queried this week?\"*" ] }, { "cell_type": "markdown", - "id": "2c4f5d42", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "734e68ba-1259-43fa-8da6-f0fc5ab07cab", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.1 — Find recently accessed tables in your catalog\n", "\n", "Query `system.access.table_lineage` to find all **write events** (source tables) within the past 7 days that relate to `automotive_catalog`. Return the table name and the count of distinct events, ordered by the most active tables first.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I query system.access.table_lineage to find recently accessed Unity Catalog tables in Databricks?\"*\n", "\n", "**Hint:** Filter on `source_table_full_name LIKE 'automotive_catalog%'` and group by table name." @@ -366,45 +624,123 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "363d78dd", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "e4edea03-8f98-400f-9910-ba2092573325", + "showTitle": true, + "tableResultSettingsMap": {}, + "title": "Cell 24" + } + }, + "outputs": [], + "source": [ + "%sql\n", + "GRANT USE SCHEMA ON SCHEMA system.access TO `user1-63885547@lodsprodmca.onmicrosoft.com`;\n", + "GRANT SELECT ON SCHEMA system.access TO `user1-63885547@lodsprodmca.onmicrosoft.com`;" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "6de3b319-3871-4159-809b-72412f420a82", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Query system.access.table_lineage for tables in automotive_catalog accessed in the last 7 days\n", "-- Return: source_table_full_name, count of events\n", - "-- Order by most active first" + "-- Order by most active first\n", + "SELECT \n", + " source_table_full_name,\n", + " COUNT(DISTINCT event_id) AS event_count, \n", + " MAX(event_time)\n", + "FROM \n", + " system.access.table_lineage\n", + "WHERE \n", + " event_date > CURRENT_DATE() - INTERVAL 7 DAYS\n", + " AND(\n", + " source_table_full_name LIKE 'automotive_catalog2%'\n", + " )\n", + "GROUP BY source_table_full_name\n", + "ORDER BY last_event_time DESC;" ] }, { "cell_type": "markdown", - "id": "f6acecab", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "e4fac3dd-7b10-45e2-aa1d-e2bad4b1b794", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.2 — View the history of vehicle_telemetry\n", "\n", "Use `DESCRIBE HISTORY` to see the full Delta Lake operation history for `vehicle_telemetry`. Identify the version where the deletion occurred (`DELETE` operation).\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I view the full version history of a Delta Lake table in Databricks SQL?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "eaafacd9", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "3a77b86e-4b2a-4443-9434-e2dff2e65511", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Describe the history of automotive_catalog.governance_lab.vehicle_telemetry" + "-- TODO: Describe the history of automotive_catalog.governance_lab.vehicle_telemetry\n", + "DESCRIBE HISTORY automotive_catalog2.governance_lab.vehicle_telemetry;" ] }, { "cell_type": "markdown", - "id": "1a505bac", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "54b037d9-0bfb-4453-a81c-a6b1a8051b96", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 5: Audit Logging\n", "\n", @@ -415,14 +751,22 @@ }, { "cell_type": "markdown", - "id": "dfb2a448", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7e900a68-a3d6-48e3-b5b8-774aef38e2e4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 5.1 — Find who accessed the customer_registrations table\n", "\n", "Query `system.access.audit` to list all users who accessed `automotive_catalog.governance_lab.customer_registrations` in the past 30 days. Include the action name and the time of each access, ordered by most recent first.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I query the Databricks audit log system table to find who accessed a specific Unity Catalog table?\"*\n", "\n", "**Hint:** Filter on `request_params.full_name_arg` and `action_name IN ('getTable', 'createTable')`. Use `user_identity.email` for the user column." @@ -430,27 +774,59 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4b237cd6", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "ff234ae9-9cb2-4566-8aab-b62700ba8e50", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Query system.access.audit for access events on automotive_catalog.governance_lab.customer_registrations\n", "-- Return: user email, action_name, event_time\n", - "-- Filter to the past 30 days, order by event_time DESC" + "-- Filter to the past 30 days, order by event_time DESC\n", + "\n", + "SELECT \n", + " user_identity.email AS\n", + " user_email,\n", + " action_name,\n", + " event_time\n", + "FROM system.access.audit\n", + "WHERE request_params.full_name_arg = 'automotive_catalog2.governance_lab.customer_registrations'\n", + "AND action_name IN\n", + "('getTable', 'createTable')\n", + "AND event_date > CURRENT_DATE() - INTERVAL 30 DAYS\n", + "ORDER BY event_time DESC;\n" ] }, { "cell_type": "markdown", - "id": "dbd2e60c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "e49aafea-5ba4-4723-8d86-fa715806d7f5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 5.2 — Detect permission changes across your catalog\n", "\n", "Query `system.access.audit` to list all **permission change events** (`updatePermissions` action) on securable objects in `automotive_catalog` during the past 30 days. Include who made the change and what object was affected.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I find all permission changes in the Databricks audit log for a specific Unity Catalog catalog?\"*\n", "\n", "**Hint:** Filter on `service_name = 'unityCatalog'`, `action_name = 'updatePermissions'`, and `request_params.securable_full_name LIKE 'automotive_catalog%'`." @@ -458,50 +834,135 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "bde1da29", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "dcae3ef8-26f4-4994-af5e-6b56875335c1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Query system.access.audit for permission change events on automotive_catalog objects\n", "-- Return: event_time, user email, securable_type, securable_full_name, changes\n", - "-- Filter to the past 30 days, order by event_time DESC" + "-- Filter to the past 30 days, order by event_time DESC\n", + "\n", + "SELECT \n", + " event_time,\n", + " user_identity.email AS user_email,\n", + " request_params.securable_type,\n", + " request_params.securable_full_name,\n", + " request_params.changes\n", + "FROM system.access.audit\n", + "WHERE service_name = 'unityCatalog'\n", + " AND action_name = 'updatePermissions'\n", + " AND request_params.securable_full_name LIKE 'automotive_catalog2%'\n", + " AND event_date > CURRENT_DATE() - INTERVAL 30 DAYS\n", + "ORDER BY event_time DESC;\n" ] }, { "cell_type": "markdown", - "id": "f9458263", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "26a948ce-f28b-4dc7-8af7-aa035e69466d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 5.3 — Summarise audit activity by user\n", "\n", "Create a summary query that counts the total number of audit events per user for Unity Catalog actions in the past 7 days. This gives the compliance team a quick overview of who was most active on the platform.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I aggregate the Databricks audit log to count events per user for Unity Catalog activity?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "2e3e9bb9", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "2f679f7a-425d-46a2-89ec-ee52ccafc8c5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Count total audit events per user from system.access.audit\n", "-- Filter to service_name = 'unityCatalog' and the last 7 days\n", "-- Return: user email, event count\n", - "-- Order by most active user first" + "-- Order by most active user first\n", + "\n", + "\n", + "SELECT \n", + " user_identity.email AS user_email,\n", + " COUNT(*) AS event_count\n", + "FROM system.access.audit\n", + "WHERE service_name = 'unityCatalog'\n", + " AND event_date > CURRENT_DATE() - INTERVAL 7 DAYS\n", + "GROUP BY user_identity.email\n", + "ORDER BY event_count DESC;" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": { + "hardware": { + "accelerator": null, + "gpuPoolId": null, + "memory": null + }, + "software": { + "pinSparkToX86": null + } + }, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 8684257317553162, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "05-govern-unity-catalog-objects", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/06-design-implement-data-modeling-unity-catalog.ipynb b/Allfiles/06-design-implement-data-modeling-unity-catalog.ipynb index d2c7f39..6dac302 100644 --- a/Allfiles/06-design-implement-data-modeling-unity-catalog.ipynb +++ b/Allfiles/06-design-implement-data-modeling-unity-catalog.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "20435367", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "3a0ad445-81b1-480b-8142-f1da873b310e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Create the Northbank Financial Data Model\n", "\n", @@ -12,21 +20,56 @@ "After completing this exercise, return to the lab setup page to explore the tables you created in **Catalog Explorer**." ] }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "35660014-0b42-4712-83bb-d93434a5aa95", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, + "outputs": [], + "source": [] + }, { "cell_type": "markdown", - "id": "f2eb3610", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "4071462e-d34a-4426-b3ec-a2b2555e6b63", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.1 — Create the catalog and schema\n", "\n", - "Create a Unity Catalog **catalog** named `banking_lab` with an appropriate comment. Inside it, create a **schema** named `silver`." + "Create a Unity Catalog **catalog** named `banking_lab63940554` with an appropriate comment. Inside it, create a **schema** named `silver`." ] }, { "cell_type": "code", - "execution_count": null, - "id": "b07608af", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "fe2bd1ab-c9d0-44b7-a608-2636511c1231", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -43,7 +86,7 @@ "print (f\"Storage root: {storage_root}\")\n", "\n", "spark.sql(f\"\"\"\n", - " CREATE CATALOG IF NOT EXISTS banking_lab\n", + " CREATE CATALOG IF NOT EXISTS banking_lab63940554\n", " MANAGED LOCATION '{storage_root}'\n", " COMMENT 'Northbank Financial lakehouse platform for customer analytics and regulatory reporting'\n", "\"\"\")" @@ -51,24 +94,44 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "22cf004d", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "2406f7f8-5d05-4df2-ba95-2dcd1bcca379", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "CREATE SCHEMA IF NOT EXISTS banking_lab.silver\n", + "CREATE SCHEMA IF NOT EXISTS banking_lab63940554.silver\n", " COMMENT 'Cleansed and enriched banking data — customer dimensions and transaction facts';" ] }, { "cell_type": "markdown", - "id": "6a701a07", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1e183fbf-9246-4278-b04e-6af26f264404", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.2 — Create the `dim_customer` SCD Type 2 table\n", "\n", - "Create a managed Delta table `banking_lab.silver.dim_customer`. This table tracks **customer dimension history** using the SCD Type 2 pattern.\n", + "Create a managed Delta table `banking_lab63940554.silver.dim_customer`. This table tracks **customer dimension history** using the SCD Type 2 pattern.\n", "\n", "The table must include:\n", "\n", @@ -87,7 +150,7 @@ "\n", "Apply **liquid clustering** on `customer_id` and enable **Change Data Feed**.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a Delta table in Databricks with GENERATED ALWAYS AS IDENTITY, CLUSTER BY, and Change Data Feed enabled in TBLPROPERTIES?\"*\n", "\n", "**Hints:**\n", @@ -98,24 +161,61 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "47b35520", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "6967d5fd-a2bf-4839-bc2f-64be1f94b699", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create banking_lab.silver.dim_customer as a managed Delta table\n", - "-- Include all SCD Type 2 columns, CLUSTER BY (customer_id), and enable Change Data Feed" + "-- TODO: Create banking_lab63940554.silver.dim_customer as a managed Delta table\n", + "-- Include all SCD Type 2 columns, CLUSTER BY (customer_id), and enable Change Data Feed\n", + "\n", + "CREATE TABLE IF NOT EXISTS banking_lab63940554.silver.dim_customer(\n", + "\n", + " customer_sk BIGINT GENERATED ALWAYS AS IDENTITY,\n", + " customer_id STRING NOT NULL,\n", + " full_name STRING,\n", + " email STRING,\n", + " city STRING,\n", + " segment STRING, \n", + " account_type STRING,\n", + " valid_from TIMESTAMP NOT NULL,\n", + " valid_to TIMESTAMP NOT NULL,\n", + " is_current BOOLEAN\n", + ")\n", + "USING DELTA\n", + "CLUSTER BY (customer_id)\n", + "TBLPROPERTIES (delta.enableChangeDataFeed = true)" ] }, { "cell_type": "markdown", - "id": "e04d3467", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "fbc54768-b5ae-4db5-ab02-3722b19afc08", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1.3 — Create the `fact_transactions` table\n", "\n", - "Create a managed Delta table `banking_lab.silver.fact_transactions` to store individual banking payments.\n", + "Create a managed Delta table `banking_lab63940554.silver.fact_transactions` to store individual banking payments.\n", "\n", "| Column | Type | Notes |\n", "|---|---|---|\n", @@ -130,7 +230,7 @@ "\n", "Apply **liquid clustering** on `customer_id, transaction_date` to support the expected query patterns (filtering by customer and/or date range). Enable **Change Data Feed** for FCA compliance auditing.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a Delta Lake table in Databricks SQL with liquid clustering and Change Data Feed enabled?\"*\n", "\n", "**Hints:**\n", @@ -141,20 +241,54 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "6d48a6ae", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "7a298804-a13b-4f26-94e3-d6a45f7a130a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Create banking_lab.silver.fact_transactions as a managed Delta table\n", - "-- Apply CLUSTER BY (customer_id, transaction_date) and enable Change Data Feed" + "-- TODO: Create banking_lab63940554.silver.fact_transactions as a managed Delta table\n", + "-- Apply CLUSTER BY (customer_id, transaction_date) and enable Change Data Feed\n", + "\n", + "CREATE TABLE IF NOT EXISTS banking_lab63940554.silver.fact_transactions(\n", + " transaction_id STRING NOT NULL,\n", + " account_id STRING,\n", + " customer_id STRING,\n", + " transaction_type STRING,\n", + " amount DECIMAL(10,2),\n", + " currency STRING DEFAULT 'GBP',\n", + " transaction_date DATE,\n", + " description STRING\n", + ")\n", + "USING DELTA\n", + "CLUSTER BY (customer_id, transaction_date)\n", + "TBLPROPERTIES (delta.enableChangeDataFeed = true, 'delta.feature.allowColumnDefaults'='supported')" ] }, { "cell_type": "markdown", - "id": "a07f6c28", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1a865ec8-e629-4d54-a98b-31cca18ec67e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Implement SCD Type 2 for the Customer Dimension\n", "\n", @@ -163,26 +297,45 @@ }, { "cell_type": "markdown", - "id": "66c5d62e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "dde723c1-881a-464b-8bf0-552d5f795ab5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.1 — Load the initial customer records\n", "\n", - "The cell below defines the **initial dataset** for 8 Northbank customers as a PySpark DataFrame. Your task is to **write this DataFrame to `banking_lab.silver.dim_customer`** in append mode.\n", + "The cell below defines the **initial dataset** for 8 Northbank customers as a PySpark DataFrame. Your task is to **write this DataFrame to `banking_lab63940554.silver.dim_customer`** in append mode.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I write a PySpark DataFrame to an existing Delta table in Databricks using saveAsTable in append mode?\"*\n", "\n", "**Hints:**\n", - "- Use `df.write.mode(\"append\").saveAsTable(\"banking_lab.silver.dim_customer\")`\n", + "- Use `df.write.mode(\"append\").saveAsTable(\"banking_lab63940554.silver.dim_customer\")`\n", "- The data and DataFrame setup are already provided — add the write statement after the DataFrame is built" ] }, { "cell_type": "code", - "execution_count": null, - "id": "18589255", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "7f45eaf2-bc94-405b-90fd-d98c330d7d5f", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "from pyspark.sql.functions import lit, to_timestamp\n", @@ -210,27 +363,49 @@ " .withColumn(\"is_current\", lit(True))\n", ")\n", "\n", - "# TODO: Write df_initial to banking_lab.silver.dim_customer (append mode)" + "# TODO: Write df_initial to banking_lab63940554.silver.dim_customer (append mode)\n", + "\n", + "df_initial.write.mode(\"append\").saveAsTable(\"banking_lab63940554.silver.dim_customer\")" ] }, { "cell_type": "code", - "execution_count": null, - "id": "1199b217", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "dccaa3f2-6fbd-4993-94df-c5edfe2117e8", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Verify: you should see 8 customer records\n", "SELECT customer_sk, customer_id, full_name, city, segment, is_current\n", - "FROM banking_lab.silver.dim_customer\n", + "FROM banking_lab63940554.silver.dim_customer\n", "ORDER BY customer_sk;" ] }, { "cell_type": "markdown", - "id": "cbe8ff36", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "a1c3f2d7-d662-47bd-a571-3512cae17650", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.2 — Load staging data with customer changes\n", "\n", @@ -248,9 +423,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "cff50d6f", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "42d67aaa-8646-4202-a27f-49dff8dde62e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "staging_data = [\n", @@ -271,8 +457,16 @@ }, { "cell_type": "markdown", - "id": "3c73f6b1", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "fbb48129-ba26-4d18-9e85-7f6d0ce870c7", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.3 — Apply SCD Type 2: Step 1 — Close changed records\n", "\n", @@ -280,7 +474,7 @@ "- `valid_to = current_timestamp()`\n", "- `is_current = false`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"Write a Databricks SQL MERGE statement that closes SCD Type 2 dimension records by setting is_current=false and valid_to=current_timestamp() when any tracked attribute changes.\"*\n", "\n", "**Hints:**\n", @@ -291,21 +485,54 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "c35b36a2", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "ed110e16-6aa9-4817-9b33-02e0cbd8c250", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Step 1: Close the current version of changed customer records\n", - "-- MERGE INTO banking_lab.silver.dim_customer ...\n", - "-- TODO: Write the MERGE statement" + "-- MERGE INTO banking_lab63940554.silver.dim_customer ...\n", + "-- TODO: Write the MERGE statement\n", + "\n", + "MERGE INTO banking_lab63940554.silver.dim_customer target\n", + "USING staging_customers AS source\n", + "ON target.customer_id = source.customer_id AND target.is_current = true\n", + "WHEN MATCHED AND (\n", + " target.city != source.city\n", + " OR target.segment != source.segment \n", + " OR target.email != source.email\n", + " OR target.account_type != source.account_type\n", + ")\n", + "THEN UPDATE SET\n", + " valid_to = current_timestamp(), \n", + " is_current = false;" ] }, { "cell_type": "markdown", - "id": "7b0b54fc", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "5af947bb-a43d-493c-a983-8bd877a61182", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.3 — Apply SCD Type 2: Step 2 — Insert new versions and new customers\n", "\n", @@ -315,29 +542,86 @@ "\n", "New rows should have `valid_from = current_timestamp()`, `valid_to = TIMESTAMP '9999-12-31'`, and `is_current = true`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"Write a Databricks SQL INSERT statement to add new current SCD Type 2 rows from a staging table, only for customers where no is_current=true record exists.\"*\n", "\n", - "**Hint:** Use a `WHERE NOT EXISTS (SELECT 1 FROM banking_lab.silver.dim_customer d WHERE d.customer_id = s.customer_id AND d.is_current = true)` to identify which staging records need a new row inserted." + "**Hint:** Use a `WHERE NOT EXISTS (SELECT 1 FROM banking_lab63940554.silver.dim_customer d WHERE d.customer_id = s.customer_id AND d.is_current = true)` to identify which staging records need a new row inserted." ] }, { "cell_type": "code", - "execution_count": null, - "id": "de442db2", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "eeeb9070-f9fc-4b7d-95e4-81a2b11fc9bb", + "showTitle": false, + "tableResultSettingsMap": { + "0": { + "dataGridStateBlob": "{\"version\":1,\"tableState\":{\"columnPinning\":{\"left\":[\"#row_number#\"],\"right\":[]},\"columnSizing\":{},\"columnVisibility\":{}},\"settings\":{\"columns\":{}},\"syncTimestamp\":1785844721892}", + "filterBlob": null, + "queryPlanFiltersBlob": null, + "tableResultIndex": 0 + } + }, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Step 2: Insert new current versions for changed customers and the new customer\n", - "-- INSERT INTO banking_lab.silver.dim_customer ...\n", - "-- TODO: Write the INSERT statement" + "-- INSERT INTO banking_lab63940554.silver.dim_customer ...\n", + "-- TODO: Write the INSERT statement\n", + "\n", + "INSERT INTO banking_lab63940554.silver.dim_customer (\n", + " customer_id,\n", + " full_name,\n", + " city,\n", + " segment,\n", + " account_type,\n", + " valid_from, \n", + " valid_to, \n", + " is_current\n", + " )\n", + "SELECT \n", + " s.customer_id,\n", + " s.full_name,\n", + " s.city,\n", + " s.segment,\n", + " s.account_type,\n", + " current_timestamp(),\n", + " TIMESTAMP '9999-12-31',\n", + " true\n", + "FROM staging_customers s\n", + "WHERE NOT EXISTS (\n", + " SELECT 1 FROM banking_lab63940554.silver.dim_customer d\n", + " WHERE s.customer_id = d.customer_id AND d.is_current = true\n", + ");\n", + "\n", + "-- Verify: you should see 10 customer records\n", + "SELECT customer_sk, customer_id, full_name, city, segment, is_current\n", + "FROM banking_lab63940554.silver.dim_customer\n", + "ORDER BY customer_sk" ] }, { "cell_type": "markdown", - "id": "d75af494", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "55e86686-4fc4-4858-8974-bdab5f6b6651", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.4 — Query the full version history of a customer\n", "\n", @@ -345,7 +629,7 @@ "- Version 1: London / Retail (expired)\n", "- Version 2: Oxford / Premium (current)\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I query all SCD Type 2 versions for a specific customer in a Databricks Delta table?\"*\n", "\n", "**Hint:** Filter on `customer_id = 'C-1001'` and `ORDER BY valid_from ASC`." @@ -353,19 +637,43 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "abbcdcde", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c5a0a546-9e3a-4d92-a25a-afe5e8e12877", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Show all versions of customer C-1001, ordered by valid_from" + "-- TODO: Show all versions of customer C-1001, ordered by valid_from\n", + "\n", + "SELECT * FROM banking_lab63940554.silver.dim_customer \n", + "WHERE customer_id = 'C-1001'\n", + "ORDER BY valid_from ASC\n" ] }, { "cell_type": "markdown", - "id": "78599bd9", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "94302318-89b2-4f84-a4ab-fa807b770bf7", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.5 — Point-in-time customer lookup\n", "\n", @@ -373,7 +681,7 @@ "\n", "Write a query that returns one row per customer showing their profile on that date. Use the `valid_from` and `valid_to` columns to select the version that was active on 2020-06-15.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I write a point-in-time query on an SCD Type 2 table using valid_from and valid_to date range filtering?\"*\n", "\n", "**Hint:** A version was active on date `D` when `valid_from <= D AND valid_to > D`. Expected result: all 8 original customers in their initial state (no Oxford, no London for Sophia)." @@ -381,20 +689,44 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "f59729c4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "517ffce7-d510-4533-ade2-3a36ea32c23d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Return the customer profile as it existed on 2020-06-15\n", - "-- Expected: 8 rows showing the original city and segment values" + "-- Expected: 8 rows showing the original city and segment values\n", + "\n", + "SELECT * FROM banking_lab63940554.silver.dim_customer\n", + "WHERE valid_from <= '2020-06-15' AND valid_to > '2020-06-15'\n", + "ORDER BY customer_id;" ] }, { "cell_type": "markdown", - "id": "88d4d48f", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7a8a1a39-2919-4248-a0ac-4484347002f3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Change Data Feed & FCA Audit Trail\n", "\n", @@ -403,8 +735,16 @@ }, { "cell_type": "markdown", - "id": "ee57b25d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "563a10d7-7ef2-4f44-bbf9-a1f2bec389cb", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.1 — Load initial transactions\n", "\n", @@ -413,13 +753,25 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1b2603b3", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "4f697159-2021-4b48-9c6d-a4210a36e862", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "INSERT INTO banking_lab.silver.fact_transactions VALUES\n", + "INSERT INTO banking_lab63940554.silver.fact_transactions VALUES\n", " ('TXN-001', 'ACC-001', 'C-1001', 'Credit', 1500.00, 'GBP', '2025-11-01', 'Salary payment'),\n", " ('TXN-002', 'ACC-001', 'C-1001', 'Debit', 120.50, 'GBP', '2025-11-03', 'Amazon purchase'),\n", " ('TXN-003', 'ACC-002', 'C-1002', 'Debit', 250.00, 'GBP', '2025-11-05', 'Grocery store'),\n", @@ -434,8 +786,16 @@ }, { "cell_type": "markdown", - "id": "397b9ed8", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "42fd7380-b34e-4291-b2e0-6e6f76c93971", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.2 — Simulate transaction corrections\n", "\n", @@ -444,61 +804,111 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "8a824a8a", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "185333ee-7d95-4749-a58b-4144ac188a1c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Correction 1: TXN-003 amount was mis-keyed (correct: £275.50, not £250.00)\n", - "UPDATE banking_lab.silver.fact_transactions\n", + "UPDATE banking_lab63940554.silver.fact_transactions\n", "SET amount = 275.50, description = 'Grocery store (corrected)'\n", "WHERE transaction_id = 'TXN-003';\n", "\n", "-- Correction 2: TXN-007 amount is under dispute\n", - "UPDATE banking_lab.silver.fact_transactions\n", + "UPDATE banking_lab63940554.silver.fact_transactions\n", "SET amount = 1450.00, description = 'Business expense (disputed amount)'\n", "WHERE transaction_id = 'TXN-007';" ] }, { "cell_type": "markdown", - "id": "a382bc48", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "5cb5c54d-f2ae-4f63-a79c-5d74f2f789e2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.3 — Query the Change Data Feed to build an audit trail\n", "\n", "Write a query using `table_changes()` to retrieve **all change events** recorded for `fact_transactions` since the beginning of its history. Include the `_change_type`, `_commit_version`, and `_commit_timestamp` metadata columns.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I query the Change Data Feed on a Delta table in Databricks using table_changes()? Show change type and commit metadata columns.\"*\n", "\n", - "**Hint:** Use `table_changes('banking_lab.silver.fact_transactions', 0)` — the second argument is the starting version number." + "**Hint:** Use `table_changes('banking_lab63940554.silver.fact_transactions', 0)` — the second argument is the starting version number." ] }, { "cell_type": "code", - "execution_count": null, - "id": "17077832", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "5bf03549-3ab3-4b78-80f5-8e985d1e58e4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Query all changes to fact_transactions from version 0 onwards\n", "-- Include: transaction_id, amount, description, _change_type, _commit_version, _commit_timestamp\n", - "-- ORDER BY _commit_timestamp, transaction_id" + "-- ORDER BY _commit_timestamp, transaction_id\n", + "\n", + "SELECT \n", + " transaction_id, \n", + " amount, \n", + " description, \n", + " _change_type, \n", + " _commit_version, \n", + " _commit_timestamp \n", + "FROM table_changes('banking_lab63940554.silver.fact_transactions', 0)\n", + "ORDER BY _commit_timestamp, transaction_id;\n" ] }, { "cell_type": "markdown", - "id": "8e2e5962", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "dfc4680b-1538-4f2c-aad9-ad52a5499729", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.4 — Isolate corrected records for FCA reporting\n", "\n", "For FCA reporting, the compliance team needs a view showing only the **post-correction state** of modified transactions. Filter the Change Data Feed to show only `update_postimage` records.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I filter table_changes() results in Databricks to show only update_postimage records?\"*\n", "\n", "**Hint:** Each `UPDATE` operation generates two rows: `update_preimage` (before) and `update_postimage` (after the change). Filter for `_change_type = 'update_postimage'`. Expected result: TXN-003 with £275.50 and TXN-007 with £1450.00." @@ -506,20 +916,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "72909401", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "1c5ceec5-53f8-493e-b940-a964a9ef333e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Show only the post-correction state of updated transactions\n", - "-- Filter table_changes() results for _change_type = 'update_postimage'" + "-- Filter table_changes() results for _change_type = 'update_postimage'\n", + "\n", + "SELECT \n", + " transaction_id, \n", + " amount, \n", + " description, \n", + " _change_type, \n", + " _commit_version, \n", + " _commit_timestamp \n", + "FROM table_changes('banking_lab63940554.silver.fact_transactions', 0)\n", + "WHERE _change_type = 'update_postimage'\n", + "ORDER BY _commit_timestamp, transaction_id; " ] }, { "cell_type": "markdown", - "id": "d3209538", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "c8f5270a-7a6a-4fc9-8116-322a23a99dd1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Delta Lake Time Travel\n", "\n", @@ -528,14 +969,22 @@ }, { "cell_type": "markdown", - "id": "5e09e479", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "f020eacd-bd6a-43a0-94c0-6ed63543101a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.1 — Inspect the transaction history\n", "\n", "Use `DESCRIBE HISTORY` to view the complete operation log of `fact_transactions`. The output shows every version of the table, including the operation type, timestamp, and user.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I view the full operation history of a Delta table in Databricks, including all version numbers?\"*\n", "\n", "Note the **version numbers** in the output — you will use them in the next tasks." @@ -543,25 +992,47 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4df02696", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "17e0d083-0af8-46a1-b836-2741314f36fb", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Show the full operation history of banking_lab.silver.fact_transactions" + "-- TODO: Show the full operation history of banking_lab63940554.silver.fact_transactions\n", + "\n", + "DESCRIBE HISTORY banking_lab63940554.silver.fact_transactions" ] }, { "cell_type": "markdown", - "id": "5c832d3d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "693c1ff4-aacd-49e4-a8b7-500f012df549", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.2 — Query original transaction amounts before corrections\n", "\n", "Using `VERSION AS OF`, retrieve `TXN-003` and `TXN-007` **as they appeared after the initial INSERT but before the corrections** were applied.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I query a previous version of a Delta table in Databricks using VERSION AS OF in a SELECT statement?\"*\n", "\n", "**Hint:** The initial INSERT corresponds to version **1** (version 0 is the table creation). Expected result: TXN-003 with £250.00 and TXN-007 with £1500.00." @@ -569,20 +1040,46 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4c602e48", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "288a1f96-c749-4e71-ae0d-c3da2a7b7785", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Query fact_transactions VERSION AS OF 1\n", - "-- Show transaction_id, amount, description for TXN-003 and TXN-007" + "-- Show transaction_id, amount, description for TXN-003 and TXN-007\n", + "SELECT \n", + " transaction_id, \n", + " amount, \n", + " description \n", + "FROM banking_lab63940554.silver.fact_transactions VERSION AS OF 1\n", + "WHERE transaction_id IN ('TXN-003', 'TXN-007');" ] }, { "cell_type": "markdown", - "id": "ff7880da", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "82c9363d-f046-4214-bbab-14f2d1dc30b7", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.3 — Restore the table to its pre-correction state\n", "\n", @@ -590,41 +1087,93 @@ "\n", "After restoring, verify that TXN-003 and TXN-007 show their original amounts.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I use RESTORE TABLE in Databricks to roll back a Delta table to a specific version number?\"*\n", "\n", - "**Hint:** Use `RESTORE TABLE banking_lab.silver.fact_transactions TO VERSION AS OF 1;`" + "**Hint:** Use `RESTORE TABLE banking_lab63940554.silver.fact_transactions TO VERSION AS OF 1;`" ] }, { "cell_type": "code", - "execution_count": null, - "id": "e9ed572b", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "49a26bb9-7611-43b8-b77d-1e8efea09807", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Restore fact_transactions to VERSION AS OF 1" + "-- TODO: Restore fact_transactions to VERSION AS OF 1\n", + "RESTORE TABLE banking_lab63940554.silver.fact_transactions TO VERSION AS OF 1" ] }, { "cell_type": "code", - "execution_count": null, - "id": "3e6f2687", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "e0e38438-a601-42d1-949e-6a9caa7863ca", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Verify the original amounts are restored for TXN-003 and TXN-007\n", - "-- Expected: TXN-003 = £250.00, TXN-007 = £1500.00" + "-- Expected: TXN-003 = £250.00, TXN-007 = £1500.00\n", + "SELECT \n", + " transaction_id, \n", + " amount, \n", + " description \n", + "FROM banking_lab63940554.silver.fact_transactions\n", + "WHERE transaction_id IN ('TXN-003', 'TXN-007');" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 4756691258415467, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "06-design-implement-data-modeling-unity-catalog", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/07-ingest-data-into-unity-catalog.ipynb b/Allfiles/07-ingest-data-into-unity-catalog.ipynb index 71051a2..284f6ce 100644 --- a/Allfiles/07-ingest-data-into-unity-catalog.ipynb +++ b/Allfiles/07-ingest-data-into-unity-catalog.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "0d15f5f3", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6c4dcfcb-4e71-40cf-afcc-1086ea5ed2f2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Set up the Solaris Energy Data Platform\n", "\n", @@ -14,8 +22,16 @@ }, { "cell_type": "markdown", - "id": "438d0a04", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "d5be5d50-b20a-4eea-ab90-ea71a08bc9fd", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: Set up the Unity Catalog structure\n", "\n", @@ -30,9 +46,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "51fd3672", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "87e1e0d1-2bae-4f7d-9c6b-b6ff3b900dc1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -49,7 +76,7 @@ "print (f\"Storage root: {storage_root}\")\n", "\n", "spark.sql(f\"\"\"\n", - " CREATE CATALOG IF NOT EXISTS solaris_lab\n", + " CREATE CATALOG IF NOT EXISTS solaris_lab2\n", " MANAGED LOCATION '{storage_root}'\n", " COMMENT 'Solaris Energy analytics platform — solar farms and wind turbines across Europe'\n", "\"\"\")" @@ -57,26 +84,46 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "b427659b", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c96dcd30-064e-4c9f-8210-95862023bccd", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "CREATE SCHEMA IF NOT EXISTS solaris_lab.bronze\n", + "CREATE SCHEMA IF NOT EXISTS solaris_lab2.bronze\n", " COMMENT 'Raw ingested data — unmodified, as received from field systems';\n", "\n", - "CREATE SCHEMA IF NOT EXISTS solaris_lab.silver\n", + "CREATE SCHEMA IF NOT EXISTS solaris_lab2.silver\n", " COMMENT 'Cleansed and enriched data — ready for analytics and reporting';\n", " \n", - "CREATE VOLUME IF NOT EXISTS solaris_lab.bronze.raw_files\n", + "CREATE VOLUME IF NOT EXISTS solaris_lab2.bronze.raw_files\n", " COMMENT 'Landing zone for raw CSV files from SCADA systems and field sensors';" ] }, { "cell_type": "markdown", - "id": "48a84036", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "9019fa9d-6d1b-4ad6-b762-2914acfaa462", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### ✅ Provided: Populate the volume with sample data\n", "\n", @@ -89,9 +136,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "3312d797", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "20a341b1-ac9d-4da7-b113-681d8a7c5b9d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — it writes sample data files to your volume\n", @@ -157,19 +215,27 @@ "EVT020,T-DK-005,WIND_DK_001,2025-06-02 16:50:00,Maintenance,Medium,Slip ring inspection and cleaning completed,true\n", "\"\"\"\n", "\n", - "with open(\"/Volumes/solaris_lab/bronze/raw_files/solar_readings.csv\", \"w\") as f:\n", + "with open(\"/Volumes/solaris_lab2/bronze/raw_files/solar_readings.csv\", \"w\") as f:\n", " f.write(solar_csv.strip())\n", "\n", - "with open(\"/Volumes/solaris_lab/bronze/raw_files/turbine_events.csv\", \"w\") as f:\n", + "with open(\"/Volumes/solaris_lab2/bronze/raw_files/turbine_events.csv\", \"w\") as f:\n", " f.write(turbine_csv.strip())\n", "\n", - "print(\"✅ Sample data written to /Volumes/solaris_lab/bronze/raw_files/\")" + "print(\"✅ Sample data written to /Volumes/solaris_lab2/bronze/raw_files/\")" ] }, { "cell_type": "markdown", - "id": "bda6010b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ba577e59-9faa-4af5-9bf9-17eb6caa1edf", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Batch Ingestion with PySpark DataFrames\n", "\n", @@ -180,14 +246,22 @@ }, { "cell_type": "markdown", - "id": "8016db7e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "17ecac1a-0081-4857-b81e-de84888b1984", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.1 — Read the solar readings CSV into a DataFrame\n", "\n", "Use the `spark.read` API to load `solar_readings.csv` from the volume. The file has a header row and you should let Spark infer the column data types automatically.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I read a CSV file from a Unity Catalog volume into a PySpark DataFrame with header and schema inference enabled?\"*\n", "\n", "**Hints:**\n", @@ -197,26 +271,55 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "91e73fc0", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c2fa8cf0-8c88-45eb-a620-b6935938195f", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# TODO: Read solar_readings.csv from the volume into a DataFrame named df_solar\n", "\n", - "df_solar = ..." + "df_solar = (spark.read\n", + " .format(\"csv\")\n", + " .option(\"header\", \"true\")\n", + " .option(\"inferSchema\", \"true\")\n", + " .load(\"/Volumes/solaris_lab2/bronze/raw_files/solar_readings.csv\"))\n", + "# TODO: Read turbine_events.csv from the volume into a DataFrame named df_turbine)\n", + "df_turbine = (spark.read\n", + " .format(\"csv\")\n", + " .option(\"header\", \"true\")\n", + " .option(\"inferSchema\", \"true\")\n", + " .load(\"/Volumes/solaris_lab2/bronze/raw_files/turbine_events.csv\"))" ] }, { "cell_type": "markdown", - "id": "de66658c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "80258dcb-035b-460e-a501-8db6939a2e72", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.2 — Inspect the schema and sample rows\n", "\n", "Before writing data to a table, it is good practice to verify the inferred schema and check a sample of records. Confirm that `power_kw`, `irradiance_w_m2`, and `temperature_c` are numeric types.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I print the schema of a PySpark DataFrame and display the first few rows?\"*\n", "\n", "**Hints:**\n", @@ -226,21 +329,41 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "a093b302", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "16bdc5e9-3362-4945-963e-80b86cb23331", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# TODO: Print the schema of df_solar\n", + "df_solar.printSchema()\n", "\n", - "\n", - "# TODO: Display a sample of the data" + "# TODO: Display a sample of the data\n", + "display(df_solar)" ] }, { "cell_type": "markdown", - "id": "41589f5e", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "8a5069b3-ea74-4767-9419-1a0dd2bf874a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.3 — Write the DataFrame to a Delta table\n", "\n", @@ -248,7 +371,7 @@ "\n", "Use `overwrite` mode so the table is fully replaced each time the cell runs (appropriate for an initial full load).\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I write a PySpark DataFrame to a Unity Catalog Delta table using saveAsTable in overwrite mode?\"*\n", "\n", "**Hints:**\n", @@ -257,24 +380,44 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "02cb4df8", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "457948a4-adae-4cf2-ae31-2538a7cabe11", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# TODO: Write df_solar to solaris_lab.bronze.solar_readings in overwrite mode" + "# TODO: Write df_solar to solaris_lab.bronze.solar_readings in overwrite mode\n", + "df_solar.write.mode(\"overwrite\").saveAsTable(\"solaris_lab2.bronze.solar_readings\")" ] }, { "cell_type": "markdown", - "id": "c0675d33", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ee067caa-5055-4803-a976-954a0ef8c427", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2.4 — Verify the ingested table\n", "\n", "Query the Delta table you just created to confirm all rows were loaded. Also filter for rows where `status = 'Fault'` to check that fault readings from panel `P-ES-004` are present.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I count rows in a Databricks SQL table and filter by a column value?\"*\n", "\n", "**Hints:**\n", @@ -284,22 +427,47 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "b3acbab6", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "53d94c84-bcc2-4b4d-9aa8-421887151d04", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Count all rows in solaris_lab.bronze.solar_readings\n", + "-- TODO: Count all rows in solaris_lab.bronze.solar_readings \n", + "-- TODO: Create the Delta table solaris_lab.bronze.\n", + "-- TODO: Count only rows where status = 'Fault'\n", "\n", + "SELECT COUNT(*) as total_rows \n", + "FROM solaris_lab2.bronze.solar_readings; \n", "\n", - "-- TODO: Count only rows where status = 'Fault'" + "SELECT COUNT(*) AS fault_rows FROM solaris_lab2.bronze.solar_readings\n", + "WHERE status = 'Fault'" ] }, { "cell_type": "markdown", - "id": "2848ff66", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "eedec013-4545-4a9c-9a10-b3c59b7ba4f2", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: SQL-Based Ingestion\n", "\n", @@ -308,8 +476,16 @@ }, { "cell_type": "markdown", - "id": "edf2d531", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "19709d12-920c-4a43-9e66-fbe25e0ed486", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.1 — Create the turbine events target table\n", "\n", @@ -326,7 +502,7 @@ "| `description` | `STRING` |\n", "| `resolved` | `BOOLEAN` |\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I create a Delta table in Databricks SQL with explicit column types, using CREATE TABLE IF NOT EXISTS?\"*\n", "\n", "**Hints:**\n", @@ -336,26 +512,57 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "9f25d0c0", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "0d170751-ebfe-451c-9d59-8fb191dfdbb7", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Create the Delta table solaris_lab.bronze.turbine_events\n", - "-- Include all 8 columns listed above with correct data types" + "-- Include all 8 columns listed above with correct data types\n", + "CREATE TABLE IF NOT EXISTS solaris_lab2.bronze.turbine_events(\n", + " event_id STRING, \n", + " turbine_id STRING, \n", + " site_id STRING, \n", + " event_timestamp TIMESTAMP, \n", + " event_type STRING, \n", + " severity STRING, \n", + " description STRING, \n", + " resolved BOOLEAN\n", + ")\n", + "USING DELTA;" ] }, { "cell_type": "markdown", - "id": "572fb404", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ae01273b-bae0-4421-8bf8-ee8fda8ca910", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.2 — Load turbine events with COPY INTO\n", "\n", "Use `COPY INTO` to load `turbine_events.csv` from the volume. `COPY INTO` is idempotent — it tracks which files have already been loaded, so re-running the command will not create duplicate rows.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I use COPY INTO in Databricks SQL to load a CSV file from a Unity Catalog volume, including FORMAT_OPTIONS for header and type inference?\"*\n", "\n", "**Hints:**\n", @@ -366,25 +573,50 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "3c62bc58", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "6f17fb85-b328-4bf5-94d1-31c886930390", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Use COPY INTO to load turbine_events.csv into solaris_lab.bronze.turbine_events" + "-- TODO: Use COPY INTO to load turbine_events.csv into solaris_lab.bronze.turbine_events\n", + "\n", + "COPY INTO solaris_lab2.bronze.turbine_events\n", + "FROM '/Volumes/solaris_lab2/bronze/raw_files/turbine_events.csv'\n", + "FILEFORMAT = CSV\n", + "FORMAT_OPTIONS ( 'header' = 'true', 'inferSchema' = 'true')" ] }, { "cell_type": "markdown", - "id": "70632be7", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "396d4560-b126-4b5f-bcd4-1db4b2898cfe", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.3 — Verify idempotency\n", "\n", "Count the rows in `turbine_events` now, then run `COPY INTO` again, and count the rows a second time. You should see the same count both times — demonstrating that `COPY INTO` skips already-loaded files.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How does COPY INTO track which files have already been loaded in Databricks?\"*\n", "\n", "**Hints:**\n", @@ -394,41 +626,91 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1abac4b4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "fd2ec385-cb0a-4d76-b224-33f0ddc18cfe", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Count rows in turbine_events (first check)" + "-- TODO: Count rows in turbine_events (first check)\n", + "SELECT COUNT(*) FROM solaris_lab2.bronze.turbine_events" ] }, { "cell_type": "code", - "execution_count": null, - "id": "e74c5fa6", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "5954d336-9ecd-4c7a-88b7-00811b9eb434", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Run COPY INTO again (same command as Task 3.2)" + "-- TODO: Run COPY INTO again (same command as Task 3.2)\n", + "COPY INTO solaris_lab2.bronze.turbine_events\n", + "FROM '/Volumes/solaris_lab2/bronze/raw_files/turbine_events.csv'\n", + "FILEFORMAT = CSV\n", + "FORMAT_OPTIONS ( 'header' = 'true', 'inferSchema' = 'true')" ] }, { "cell_type": "code", - "execution_count": null, - "id": "d47a71cc", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "98588010-23ea-4c75-8268-e7bd471993cb", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- TODO: Count rows again — confirm the count is unchanged" + "-- TODO: Count rows again — confirm the count is unchanged\n", + "SELECT COUNT(*) FROM solaris_lab2.bronze.turbine_events" ] }, { "cell_type": "markdown", - "id": "c32af310", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7fefe6c2-26ae-4be6-aa94-60e3a11e7153", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3.4 — Build a silver summary table with CTAS\n", "\n", @@ -441,7 +723,7 @@ "- `avg_irradiance` — average `irradiance_w_m2`\n", "- `fault_count` — count of readings where `status = 'Fault'`\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I use CREATE TABLE AS SELECT in Databricks SQL to aggregate data from an existing table, grouping by date and site?\"*\n", "\n", "**Hints:**\n", @@ -453,33 +735,79 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1326e9e5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c3061628-a4bf-417a-8d5e-953808282ece", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Create or replace solaris_lab.silver.daily_solar_production\n", "-- Aggregate from solaris_lab.bronze.solar_readings\n", - "-- Include: reading_date, site_id, total_power_kw, avg_irradiance, fault_count" + "-- Include: reading_date, site_id, total_power_kw, avg_irradiance, fault_count\n", + "CREATE OR REPLACE TABLE solaris_lab2.silver.daily_solar_production AS (\n", + "SELECT\n", + " CAST(reading_timestamp AS DATE) AS reading_date,\n", + " site_id,\n", + " SUM(CASE WHEN status = 'Normal'\n", + " THEN power_kw ELSE 0 END) AS total_power_kw,\n", + " AVG(irradiance_w_m2) AS avg_irradiance,\n", + " COUNT(CASE WHEN status = 'fault' THEN 1 END) AS fault_count\n", + "FROM solaris_lab2.bronze.solar_readings\n", + "GROUP BY reading_date, site_id\n", + ")\n" ] }, { "cell_type": "code", - "execution_count": null, - "id": "e8caf394", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c9d3df14-25a9-4cb5-b39c-501275338091", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Query the summary table to verify the results\n", - "-- Order by reading_date and site_id" + "-- Order by reading_date and site_id\n", + "\n", + "SELECT * FROM solaris_lab2.silver.daily_solar_production\n", + "ORDER BY reading_date, site_id" ] }, { "cell_type": "markdown", - "id": "e251dccb", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1a8757c4-0ca4-4c50-9c6b-7271b056b62d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Continuous File Ingestion with Auto Loader\n", "\n", @@ -490,8 +818,16 @@ }, { "cell_type": "markdown", - "id": "75893a04", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ce608237-e817-4ad8-8552-4675c592f473", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### ✅ Provided: Simulate incoming data files\n", "\n", @@ -500,16 +836,27 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "3ec0f4c3", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b7b7a38f-6d60-437c-a3c9-c640a30a93e8", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — it writes three incoming data files\n", "\n", "import os\n", "\n", - "incoming_dir = \"/Volumes/solaris_lab/bronze/raw_files/incoming\"\n", + "incoming_dir = \"/Volumes/solaris_lab2/bronze/raw_files/incoming\"\n", "os.makedirs(incoming_dir, exist_ok=True)\n", "\n", "wave1 = \"\"\"reading_id,site_id,panel_id,reading_timestamp,power_kw,irradiance_w_m2,temperature_c,status\n", @@ -545,14 +892,22 @@ }, { "cell_type": "markdown", - "id": "81d7c63b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ffc4b8d4-36b7-4a65-ab77-9a4e60c72798", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.1 — Configure the Auto Loader read stream\n", "\n", "Use `spark.readStream` with the `cloudFiles` format to create a streaming DataFrame that monitors the `incoming/` directory for new CSV files. Auto Loader will automatically track which files it has processed.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I configure Auto Loader in PySpark using spark.readStream and the cloudFiles format to read CSV files from a Unity Catalog volume, with schemaLocation?\"*\n", "\n", "**Hints:**\n", @@ -565,9 +920,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "b80c5f63", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "37907b0e-5d8c-4d1f-a323-e43e85bddca6", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# TODO: Configure the Auto Loader read stream\n", @@ -577,13 +943,27 @@ "df_autoloader = (\n", " spark.readStream\n", " # ... add your configuration here\n", + " .format(\"cloudFiles\")\n", + " .option(\"cloudFiles.format\", \"csv\")\n", + " .option(\"cloudFiles.schemaLocation\", \"/Volumes/solaris_lab2/bronze/raw_files/schema\")\n", + " .option(\"cloudFiles.inferColumnTypes\", \"true\")\n", + " .option(\"header\", \"true\")\n", + " .load(\"/Volumes/solaris_lab2/bronze/raw_files/incoming\")\n", ")" ] }, { "cell_type": "markdown", - "id": "e40bd7c6", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "a9df5486-0172-447e-a397-779245417db9", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.2 — Write the stream to a Delta table\n", "\n", @@ -591,7 +971,7 @@ "\n", "You must also specify a **checkpoint location** so that Auto Loader can track progress and resume correctly if the job restarts.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How do I write an Auto Loader (spark.readStream cloudFiles) stream to a Unity Catalog table using writeStream, trigger(availableNow=True), and a checkpoint location?\"*\n", "\n", "**Hints:**\n", @@ -604,9 +984,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "11ef242a", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "891e5cc6-096e-4958-b496-dea2011f469f", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# TODO: Write the Auto Loader stream to solaris_lab.bronze.solar_readings_stream\n", @@ -614,7 +1005,9 @@ "\n", "query = (\n", " df_autoloader.writeStream\n", - " # ... add your configuration here\n", + " .trigger(availableNow=True)\n", + " .option(\"checkpointLocation\", \"/Volumes/solaris_lab2/bronze/raw_files/checkpoint\")\n", + " .toTable(\"solaris_lab2.bronze.solar_readings_stream\")\n", ")\n", "\n", "query.awaitTermination()" @@ -622,14 +1015,22 @@ }, { "cell_type": "markdown", - "id": "20081177", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "d0f824ba-0919-4c84-8ceb-613bbeef8559", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4.3 — Verify the Auto Loader ingestion\n", "\n", "Query `solaris_lab.bronze.solar_readings_stream` to confirm all 15 rows (5 per wave × 3 waves) were loaded. Then run the write stream cell a second time to confirm that no duplicate rows are created — demonstrating Auto Loader's exactly-once guarantee.\n", "\n", - "> 🤖 **Genie Code tip:** Ask:\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask:\n", "> *\"How can I verify that Auto Loader ingested the correct number of rows and didn't create duplicates?\"*\n", "\n", "**Hints:**\n", @@ -639,22 +1040,44 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "86975607", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "6970dad5-31eb-47c2-af8d-061f1b54ee43", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- TODO: Count all rows in solar_readings_stream\n", + "SELECT COUNT(*) FROM solaris_lab2.bronze.solar_readings_stream;\n", "\n", - "\n", - "-- TODO: Show a sample of rows ordered by reading_timestamp" + "-- TODO: Show a sample of rows ordered by reading_timestamp\n", + "SELECT * FROM solaris_lab2.bronze.solar_readings_stream \n", + "ORDER BY reading_timestamp" ] }, { "cell_type": "markdown", - "id": "f1be3037", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "de32284c-42ba-4245-89f7-56662842e441", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## ✅ Clean up (optional)\n", "\n", @@ -665,9 +1088,17 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "c2b2e088", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "b61d5e10-5207-43b6-8991-283314d48a0e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", @@ -677,10 +1108,31 @@ } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 8250340951300221, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "07-ingest-data-into-unity-catalog", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/08-cleanse-transform-load-data-into-unity-catalog.ipynb b/Allfiles/08-cleanse-transform-load-data-into-unity-catalog.ipynb index 645ada8..d4e08d9 100644 --- a/Allfiles/08-cleanse-transform-load-data-into-unity-catalog.ipynb +++ b/Allfiles/08-cleanse-transform-load-data-into-unity-catalog.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "1ef5332c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "42449a38-dace-4110-acfb-38ee9221b1ec", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Set up the Pristine Properties Platform\n", "\n", @@ -12,23 +20,42 @@ }, { "cell_type": "markdown", - "id": "99eb0acb", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "81c3d0c8-bc5f-4b6e-b51d-49f84a8935f3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### ✅ Provided: Create the Unity Catalog structure\n", "\n", "Run the cell below to create:\n", - "- A catalog named `realestate_lab`\n", + "- A catalog named `realestate_lab2`\n", "- A **bronze** schema for raw ingested data\n", "- A **silver** schema for cleansed and transformed data\n", - "- A managed volume `raw_files` inside `realestate_lab.bronze`" + "- A managed volume `raw_files` inside `realestate_lab2.bronze`" ] }, { "cell_type": "code", - "execution_count": null, - "id": "6ea9c1c4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b5992b07-5e6b-4d64-89bb-fe7212c0fed4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -45,7 +72,7 @@ "print (f\"Storage root: {storage_root}\")\n", "\n", "spark.sql(f\"\"\"\n", - " CREATE CATALOG IF NOT EXISTS realestate_lab\n", + " CREATE CATALOG IF NOT EXISTS realestate_lab2\n", " MANAGED LOCATION '{storage_root}'\n", " COMMENT 'Pristine Properties — real estate analytics platform for Dutch cities'\n", "\"\"\")" @@ -53,26 +80,46 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "72c57462", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d9e77abb-7d5f-419b-a87b-b995d8e1a5f4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "CREATE SCHEMA IF NOT EXISTS realestate_lab.bronze\n", + "CREATE SCHEMA IF NOT EXISTS realestate_lab2.bronze\n", " COMMENT 'Raw ingested data — as received from agent offices';\n", "\n", - "CREATE SCHEMA IF NOT EXISTS realestate_lab.silver\n", + "CREATE SCHEMA IF NOT EXISTS realestate_lab2.silver\n", " COMMENT 'Cleansed and transformed data — ready for analytics';\n", "\n", - "CREATE VOLUME IF NOT EXISTS realestate_lab.bronze.raw_files\n", + "CREATE VOLUME IF NOT EXISTS realestate_lab2.bronze.raw_files\n", " COMMENT 'Landing zone for raw CSV files from agent systems';" ] }, { "cell_type": "markdown", - "id": "b6aea65d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6b17ee5e-33f1-497d-aeeb-c8dba7865f2d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### ✅ Provided: Populate the volume with sample data\n", "\n", @@ -87,9 +134,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "8f8d4ed0", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f828190a-fef5-4def-9ba1-1840a6bf00a1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — it writes sample CSV files to your volume\n", @@ -153,7 +211,7 @@ "Groningen,195000,198000,203000,208000,213000,218000,222000,219000,214000,208000,201000,196000\n", "Breda,245000,249000,255000,262000,268000,274000,279000,275000,268000,260000,253000,247000\"\"\"\n", "\n", - "base = \"/Volumes/realestate_lab/bronze/raw_files\"\n", + "base = \"/Volumes/realestate_lab2/bronze/raw_files\"\n", "dbutils.fs.put(f\"{base}/property_listings.csv\", listings_csv, overwrite=True)\n", "dbutils.fs.put(f\"{base}/real_estate_agents.csv\", agents_csv, overwrite=True)\n", "dbutils.fs.put(f\"{base}/property_sales.csv\", sales_csv, overwrite=True)\n", @@ -163,19 +221,38 @@ }, { "cell_type": "markdown", - "id": "fc56cbbb", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "b6e39daa-95c7-44ff-92ea-6ccf213de260", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### ✅ Provided: Load data into bronze Delta tables\n", "\n", - "Run the cell below to read all four CSV files from the volume and save them as Delta tables in the `realestate_lab.bronze` schema." + "Run the cell below to read all four CSV files from the volume and save them as Delta tables in the `realestate_lab2.bronze` schema." ] }, { "cell_type": "code", - "execution_count": null, - "id": "3670b684", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "65ef4753-4651-48ff-9aab-7dea267daacc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — it loads the CSV files into bronze Delta tables\n", @@ -188,88 +265,161 @@ " .write.mode(\"overwrite\")\n", " .saveAsTable(table))\n", "\n", - "base = \"/Volumes/realestate_lab/bronze/raw_files\"\n", - "load_csv(f\"{base}/property_listings.csv\", \"realestate_lab.bronze.listings\")\n", - "load_csv(f\"{base}/real_estate_agents.csv\", \"realestate_lab.bronze.agents\")\n", - "load_csv(f\"{base}/property_sales.csv\", \"realestate_lab.bronze.sales\")\n", - "load_csv(f\"{base}/market_stats_wide.csv\", \"realestate_lab.bronze.market_stats\")\n", + "base = \"/Volumes/realestate_lab2/bronze/raw_files\"\n", + "load_csv(f\"{base}/property_listings.csv\", \"realestate_lab2.bronze.listings\")\n", + "load_csv(f\"{base}/real_estate_agents.csv\", \"realestate_lab2.bronze.agents\")\n", + "load_csv(f\"{base}/property_sales.csv\", \"realestate_lab2.bronze.sales\")\n", + "load_csv(f\"{base}/market_stats_wide.csv\", \"realestate_lab2.bronze.market_stats\")\n", "print(\"All tables loaded successfully.\")" ] }, { "cell_type": "markdown", - "id": "d55d53ec", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "e00172e8-bcfb-4c98-8d64-913e59e0fc8b", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Profile the Listings Data\n", "\n", "Before transforming data, you need to understand what’s in it. In this exercise, you compute statistics for the `listings` table and generate a summary profile using both SQL and PySpark.\n", "\n", - "> 💬 **Non-notebook task:** After completing this exercise, follow the steps in the lab setup page to create a visual **Snapshot** profile for `realestate_lab.bronze.listings` in Catalog Explorer." + "> \uD83D\uDCAC **Non-notebook task:** After completing this exercise, follow the steps in the lab setup page to create a visual **Snapshot** profile for `realestate_lab2.bronze.listings` in Catalog Explorer." ] }, { "cell_type": "markdown", - "id": "57bec8b0", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "b2fa2a5b-9646-42f0-956c-7cafdf605213", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: Compute table statistics\n", "\n", - "Use `ANALYZE TABLE` to compute statistics for all columns in `realestate_lab.bronze.listings`. Then use `DESCRIBE TABLE EXTENDED` to view the collected metrics.\n", + "Use `ANALYZE TABLE` to compute statistics for all columns in `realestate_lab2.bronze.listings`. Then use `DESCRIBE TABLE EXTENDED` to view the collected metrics.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code what statistics `ANALYZE TABLE` collects and how they help the query optimizer.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code what statistics `ANALYZE TABLE` collects and how they help the query optimizer.\n", "> *Suggested prompt: \"What statistics does ANALYZE TABLE collect in Databricks and how does the query optimizer use them?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "e55d906d", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "4df95f46-115a-444e-ab17-cd0670faa5b3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Task 1a: Compute statistics for all columns\n" + "-- Task 1a: Compute statistics for all columns\n", + "\n", + "ANALYZE TABLE realestate_lab2.bronze.listings COMPUTE STATISTICS\n" ] }, { "cell_type": "code", - "execution_count": null, - "id": "761f9ada", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "80554e09-8a18-47c6-a2af-eda2db15cc30", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Task 1b: View the collected statistics\n" + "-- Task 1b: View the collected statistics\n", + "DESCRIBE TABLE EXTENDED realestate_lab2.bronze.listings\n" ] }, { "cell_type": "markdown", - "id": "1fe1d63d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "2b62dc6a-8a7e-4074-97b5-8e99572756da", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2: Generate summary statistics using PySpark\n", "\n", - "Load the `realestate_lab.bronze.listings` table into a PySpark DataFrame and use `.describe()` to view summary statistics for all columns.\n", + "Load the `realestate_lab2.bronze.listings` table into a PySpark DataFrame and use `.describe()` to view summary statistics for all columns.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code what `.describe()` returns and how it differs from `.summary()`.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code what `.describe()` returns and how it differs from `.summary()`.\n", "> *Suggested prompt: \"What is the difference between df.describe() and df.summary() in PySpark?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "8993c0ea", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b8620170-8e72-46ae-b094-c38bb854d593", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ - "# Task 2: Load listings and view summary statistics\n" + "# Task 2: Load listings and view summary statistics\n", + "df_listings = spark.table(\"realestate_lab2.bronze.listings\")\n", + "display(df_listings.describe())\n", + "\n" ] }, { "cell_type": "markdown", - "id": "6caa9f94", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "288d6afa-aec2-4589-99ea-31f98eeb9902", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Choosing the Right Data Types\n", "\n", @@ -278,24 +428,44 @@ }, { "cell_type": "markdown", - "id": "becb01e7", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "bd4d4275-c202-4503-b51d-51797dd28f73", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: DECIMAL vs DOUBLE for property prices\n", "\n", "Floating-point types like `DOUBLE` use binary representation internally, which can introduce small but significant rounding errors — a serious problem for financial data like property prices.\n", "\n", - "First, run the provided cell to observe this problem. Then write a `CREATE TABLE` statement that stores `list_price` as `DECIMAL(12,2)` by creating `realestate_lab.silver.listings_typed` from the bronze table.\n", + "First, run the provided cell to observe this problem. Then write a `CREATE TABLE` statement that stores `list_price` as `DECIMAL(12,2)` by creating `realestate_lab2.silver.listings_typed` from the bronze table.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code why `DOUBLE` causes precision errors with prices.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code why `DOUBLE` causes precision errors with prices.\n", "> *Suggested prompt: \"Why does storing a price like 449999.99 as DOUBLE in SQL lead to precision errors, and what type should I use instead?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "67a37b07", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "52517e01-af8c-4fde-8ad4-a6a23d64c85c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", @@ -310,20 +480,58 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "b30d2081", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "97abd0e8-552e-45c7-af2f-7045789e3038", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Task 1: Create realestate_lab.silver.listings_typed with CAST for each column\n", - "- Hint: CREATE OR REPLACE TABLE ... AS SELECT CAST(...) ...\n" + "-- Task 1: Create realestate_lab2.silver.listings_typed with CAST for each column\n", + "-- Hint: CREATE OR REPLACE TABLE ... AS SELECT CAST(...) ...\n", + "\n", + "CREATE OR REPLACE TABLE realestate_lab2.silver.listings_typed AS\n", + "SELECT\n", + " listing_id,\n", + " property_id,\n", + " agent_id,\n", + " city,\n", + " property_type,\n", + " bedrooms,\n", + " bathrooms,\n", + " CAST(list_price AS DECIMAL(10,2)) AS list_price,\n", + " sqm,\n", + " listing_date,\n", + " last_updated,\n", + " status\n", + "FROM realestate_lab2.bronze.listings\n", + " \n", + "-- \n" ] }, { "cell_type": "markdown", - "id": "ad1ecbcb", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "73cda371-4f30-4b25-b5e4-1ae283019ae1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2: TIMESTAMP vs TIMESTAMP_NTZ for viewing appointments\n", "\n", @@ -333,15 +541,34 @@ "\n", "Run the provided cell to see how the two types behave differently when the session timezone changes.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code when to prefer `TIMESTAMP_NTZ` over `TIMESTAMP`.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code when to prefer `TIMESTAMP_NTZ` over `TIMESTAMP`.\n", "> *Suggested prompt: \"When should I use TIMESTAMP_NTZ instead of TIMESTAMP in Azure Databricks for property viewing appointments?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "5091f1e8", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "c9c2d49d-ac86-4116-a5a3-b0ea07f4907a", + "showTitle": false, + "tableResultSettingsMap": { + "0": { + "dataGridStateBlob": "{\"version\":1,\"tableState\":{\"columnPinning\":{\"left\":[\"#row_number#\"],\"right\":[]},\"columnSizing\":{\"viewing_time_ts\":194},\"columnVisibility\":{}},\"settings\":{\"columns\":{}},\"syncTimestamp\":1785946948040}", + "filterBlob": null, + "queryPlanFiltersBlob": null, + "tableResultIndex": 0 + } + }, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", @@ -362,9 +589,21 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "90b88e5a", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d5499d87-8c38-41ea-9b8a-fbb1b1861cef", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", @@ -374,8 +613,16 @@ }, { "cell_type": "markdown", - "id": "34df076b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6cde4a38-84d6-4e43-addb-bde832b507e5", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3: Complex types — ARRAY, MAP, and STRUCT\n", "\n", @@ -389,36 +636,85 @@ "\n", "**Task 3b:** Query `listings_enriched` and access: the first amenity (index 0), the `energy_label` from features, and the `city` from address.\n", "\n", - "> 🤖 **Genie Code tip:** Ask how to define and access ARRAY, MAP, and STRUCT in Databricks SQL.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask how to define and access ARRAY, MAP, and STRUCT in Databricks SQL.\n", "> *Suggested prompt: \"How do I create a temp view in Databricks SQL with ARRAY, MAP, and STRUCT columns, and how do I access elements from each?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "9ea93927", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "5983e005-33c1-446e-8fcc-84af9080ceff", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Task 3a: Create listings_enriched temp view with ARRAY, MAP, and STRUCT columns\n" + "-- Task 3a: Create listings_enriched temp view with ARRAY, MAP, and STRUCT columns\n", + "-- Task 3a: Create listings_enriched temp view with ARRAY, MAP, and STRUCT columns\n", + "\n", + "CREATE OR REPLACE TEMP VIEW listings_enriched AS\n", + "SELECT *\n", + "FROM VALUES\n", + " ('PROP-001', ARRAY('Parking', 'Balcony', 'Storage'), MAP('energy_label', 'A', 'heating', 'central'), named_struct('street', 'Keizersgracht 42', 'city', 'Amsterdam', 'postal_code', '1015 CR')),\n", + " ('PROP-002', ARRAY('Garden', 'Garage', 'Solar Panels'), MAP('energy_label', 'B', 'heating', 'gas'), named_struct('street', 'Witte de Withstraat 18', 'city', 'Rotterdam', 'postal_code', '3012 BM')),\n", + " ('PROP-003', ARRAY('Elevator', 'Gym'), MAP('energy_label', 'C', 'heating', 'electric'), named_struct('street', 'Oudegracht 55', 'city', 'Utrecht', 'postal_code', '3511 AP'))\n", + "AS t(property_id, amenities, features, address);\n", + " \n" ] }, { "cell_type": "code", - "execution_count": null, - "id": "622aea2b", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "62443e2d-8c4c-493b-9ee3-02a0bbdb1d3d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "-- Task 3b: Select from listings_enriched accessing: amenities[0], features['energy_label'], address.city\n" + "-- Task 3b: Select from listings_enriched accessing: amenities[0], features['energy_label'], address.city\n", + "\n", + "SELECT \n", + " property_id,\n", + " amenities[0] AS first_amenity, \n", + " features['energy_label'] AS energy_label,\n", + " address.city AS city\n", + "FROM listings_enriched" ] }, { "cell_type": "markdown", - "id": "2980750c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ee9fae2a-0322-424a-9f79-ecc4d18b6741", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Handling Duplicates and Missing Values\n", "\n", @@ -427,82 +723,172 @@ }, { "cell_type": "markdown", - "id": "ad087907", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "ca622bcf-86cb-4564-bf77-bd57d764ee51", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: Identify duplicate property listings\n", "\n", "Write a SQL query using `GROUP BY` and `HAVING` to find all `property_id` values that appear more than once.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code for a SQL pattern to find duplicate values.\n", - "> *Suggested prompt: \"Write a SQL query to find duplicate property_id values in realestate_lab.bronze.listings.\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code for a SQL pattern to find duplicate values.\n", + "> *Suggested prompt: \"Write a SQL query to find duplicate property_id values in realestate_lab2.bronze.listings.\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "a53dae2f", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "63e93050-fbba-42fe-a797-bb22e01679e4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Task 1: Find property_id values that appear more than once\n", - "--Hint: GROUP BY property_id HAVING COUNT(*) > 1\n" + "--Hint: GROUP BY property_id HAVING COUNT(*) > 1\n", + "\n", + "SELECT\n", + " property_id,\n", + " COUNT(*) AS count\n", + "FROM realestate_lab2.bronze.listings\n", + "GROUP BY property_id\n", + "HAVING COUNT(*) > 1\n" ] }, { "cell_type": "markdown", - "id": "95813c1d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "a886b3ea-8142-438e-9d8f-21be3070cda8", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2: Keep the most recently updated listing per property\n", "\n", "Use `ROW_NUMBER()` with the `QUALIFY` clause to keep only the most recently updated row for each `property_id`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code about `ROW_NUMBER()` with `QUALIFY` for deduplication.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code about `ROW_NUMBER()` with `QUALIFY` for deduplication.\n", "> *Suggested prompt: \"How do I use ROW_NUMBER() with QUALIFY in Databricks SQL to keep only the latest record per property_id ordered by last_updated?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "703c14bd", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "7500618f-a1d6-46f7-8a6e-aafd0ff4c694", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Task 2: Use ROW_NUMBER() with QUALIFY to keep the most recently updated row per property_id\n", - "-- Hint: QUALIFY ROW_NUMBER() OVER (PARTITION BY property_id ORDER BY last_updated DESC) = 1\n" + "-- Hint: QUALIFY ROW_NUMBER() OVER (PARTITION BY property_id ORDER BY last_updated DESC) = 1\n", + "\n", + "SELECT *\n", + "FROM realestate_lab2.bronze.listings\n", + "QUALIFY \n", + " ROW_NUMBER() \n", + "OVER \n", + " (PARTITION BY property_id ORDER BY last_updated DESC) = 1\n", + "\n" ] }, { "cell_type": "markdown", - "id": "69b881f1", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "eb30b870-71fd-4a5f-87b3-23d76ce4c3a0", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3: Deduplicate using PySpark\n", "\n", "Using PySpark, load the listings table, sort by `last_updated` descending, and call `dropDuplicates([\"property_id\"])`. Print the count before and after.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code how sorting interacts with `dropDuplicates`.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code how sorting interacts with `dropDuplicates`.\n", "> *Suggested prompt: \"How does orderBy work together with dropDuplicates in PySpark to control which duplicate row is kept?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "683e59fd", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "fe23843b-66db-4f72-9948-178fc4eb1b57", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Task 3: Sort by last_updated DESC and dropDuplicates(['property_id'])\n", - "# Print counts before and after, then display the result\n" + "# Print counts before and after, then display the result\n", + "\n", + "df_listings = spark.table(\"realestate_lab2.bronze.listings\")\n", + "print(f\"Before dedup: {df_listings.count()}\")\n", + "\n", + "df_deduped = df_listings.orderBy(\"last_updated\", ascending=False).dropDuplicates([\"property_id\"])\n", + "print(f\"After dedup: {df_deduped.count()}\")\n", + "\n", + "display(df_deduped)" ] }, { "cell_type": "markdown", - "id": "83402fc5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "856b622d-6b6a-4afe-bb77-44275c67364d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 4: Handle null and missing values\n", "\n", @@ -514,25 +900,56 @@ "\n", "Use `dropna()` and `fillna()`. Print counts before and after.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code how to chain dropna and fillna in PySpark.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code how to chain dropna and fillna in PySpark.\n", "> *Suggested prompt: \"How do I use dropna and fillna together in PySpark to drop rows where list_price is null and then fill other null columns?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "e8d18a43", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "abca492f-8e7a-4e12-b75d-27b18b985e4d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Task 4: Drop rows where list_price is null, then fillna for bedrooms, bathrooms, status\n", - "# Print original and cleaned counts, then display the result\n" + "# Print original and cleaned counts, then display the result\n", + "\n", + "df = spark.table(\"realestate_lab2.bronze.listings\")\n", + "print(f\"Before cleaning: {df.count()}\")\n", + "\n", + "df_cleaned = (df\n", + " .dropna(subset=[\"list_price\"])\n", + " .fillna(0, subset=[\"bedrooms\", \"bathrooms\"])\n", + " .fillna(\"Unknown\", subset=[\"status\"])\n", + " )\n", + "print(f\"After cleaning: {df_cleaned.count()}\")\n", + "\n", + "display(df_cleaned)\n" ] }, { "cell_type": "markdown", - "id": "a1a7b6aa", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "3e314a38-0c7c-43d8-b3fc-6d12683d7d02", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 5: Joining Listings with Agents and Sales Data\n", "\n", @@ -541,80 +958,183 @@ }, { "cell_type": "markdown", - "id": "51d14ef0", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1eb94b4c-9c5c-46de-8a0a-76682a3d8c04", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: Inner join listings with agents\n", "\n", - "Join `realestate_lab.bronze.listings` with `realestate_lab.bronze.agents` on `agent_id` using an **inner join**. Select: `listing_id`, `property_id`, `city` (from listings), `list_price`, `agent_name`, `agency`.\n", + "Join `realestate_lab2.bronze.listings` with `realestate_lab2.bronze.agents` on `agent_id` using an **inner join**. Select: `listing_id`, `property_id`, `city` (from listings), `list_price`, `agent_name`, `agency`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code for the correct PySpark join syntax with column aliases.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code for the correct PySpark join syntax with column aliases.\n", "> *Suggested prompt: \"How do I inner join two PySpark DataFrames on agent_id and select columns from both using DataFrame aliases?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "53b3ab2f", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "0661faff-eab4-4e89-9665-0e8c0b176767", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Task 1: Inner join listings with agents on agent_id\n", - "# Select: listing_id, property_id, city (from listings), list_price, agent_name, agency\n" + "# Select: listing_id, property_id, city (from listings), list_price, agent_name, agency\n", + "\n", + "df_listings = spark.table(\"realestate_lab2.bronze.listings\").alias(\"l\")\n", + "\n", + "df_agents = spark.table(\"realestate_lab2.bronze.agents\").alias(\"a\")\n", + "\n", + "df_joined = (df_listings\n", + " .join(df_agents, df_listings.agent_id == df_agents.agent_id, \"inner\")\n", + " .select(\"l.listing_id\", \"l.property_id\", \"l.city\", \"l.list_price\", \"a.agent_name\", \"a.agency\")\n", + " )\n", + "\n", + "display(df_joined)\n" ] }, { "cell_type": "markdown", - "id": "08271c25", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "fc60f8c8-e41e-469f-a807-a05f939255a1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2: Find agents with no listings\n", "\n", "Perform a **left join** with `agents` on the left and `listings` on the right. Filter for rows where `listing_id` is null to identify agents with no current listings.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code how a left join helps find records that exist in one table but not another.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code how a left join helps find records that exist in one table but not another.\n", "> *Suggested prompt: \"How do I use a left join in PySpark to find rows in agents that have no match in listings?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "99fcf017", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "be8654c4-e6b7-4f3e-9cc6-185431f453f7", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Task 2: Left join agents (left) with listings (right); filter where listing_id IS NULL\n", - "# Select: agent_id, agent_name, agency\n" + "# Select: agent_id, agent_name, agency\n", + "\n", + "from pyspark.sql.functions import col\n", + "\n", + "df_agents = spark.table(\"realestate_lab2.bronze.agents\").alias(\"a\")\n", + "\n", + "df_listings = spark.table(\"realestate_lab2.bronze.listings\").alias(\"l\")\n", + "\n", + "df_unmatched = (df_agents\n", + " .join(df_listings, df_agents.agent_id == df_listings.agent_id, \"left\")\n", + " .filter(col(\"l.listing_id\").isNull())\n", + " .select(\"a.agent_id\", \"a.agent_name\", \"a.agency\")\n", + " )\n", + "\n", + "display(df_unmatched)\n", + "\n" ] }, { "cell_type": "markdown", - "id": "5ba0eb2a", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "a8103d36-34e8-4506-a9f0-007a79e943cf", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3: Find unsold listings\n", "\n", "Join `listings` (left) with `sales` (right) on `property_id` using a **left join**. Filter for rows where `sale_id IS NULL` to find properties that have never been sold.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code about the left join + null filter pattern.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code about the left join + null filter pattern.\n", "> *Suggested prompt: \"How do I find rows in the listings table with no matching sale in the sales table using a PySpark left join?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "31793609", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d5d72866-6834-45d6-8635-f9e6e38a918c", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Task 3: Left join listings (left) with sales (right); filter where sale_id IS NULL\n", - "# Select: listing_id, property_id, city, property_type, list_price, listing_date\n" + "# Select: listing_id, property_id, city, property_type, list_price, listing_date\n", + "\n", + "df_listings = spark.table(\"realestate_lab2.bronze.listings\").alias(\"l\")\n", + "\n", + "df_sales = spark.table(\"realestate_lab2.bronze.sales\").alias(\"s\")\n", + "\n", + "df_unsold = (df_listings\n", + " .join(df_sales, df_listings.property_id == df_sales.property_id, \"left\")\n", + " .filter(col(\"s.sale_id\").isNull())\n", + " .select(\"l.listing_id\", \"l.property_id\", \"l.city\", \"l.property_type\", \"l.list_price\", \"l.listing_date\")\n", + " )\n", + "\n", + "display(df_unsold)\n" ] }, { "cell_type": "markdown", - "id": "05c63842", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "035554fd-0f7a-486d-8027-c05e18399757", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 6: Pivot and Unpivot Market Statistics\n", "\n", @@ -623,8 +1143,16 @@ }, { "cell_type": "markdown", - "id": "dd3da673", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "5a2c5d4c-7fe2-4082-b56b-99bece938267", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: Pivot listing counts by property type\n", "\n", @@ -636,52 +1164,108 @@ "|---|---|---|---|---|\n", "| Amsterdam | 3 | 1 | 1 | 1 |\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code for the `PIVOT` clause syntax in Databricks SQL.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code for the `PIVOT` clause syntax in Databricks SQL.\n", "> *Suggested prompt: \"How do I use the PIVOT clause in Databricks SQL to count active listings per city, with one column per property type?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "16d58cba", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "80f90989-ce1b-4bda-ba22-3629a8659e96", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Task 1: Pivot active listing counts by property_type\n", "-- Columns: city, Apartment, House, Studio, Villa\n", - "\n" + "\n", + "SELECT * \n", + "FROM (\n", + " SELECT city, property_type\n", + " FROM realestate_lab2.bronze.listings\n", + " WHERE status = 'Active'\n", + ")\n", + "PIVOT(\n", + " COUNT(*) FOR property_type IN ('Apartment', 'House', 'Studio', 'Villa')\n", + ")\n" ] }, { "cell_type": "markdown", - "id": "d15aa10b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "77a637bc-7ff6-4432-9477-496e7cc876e3", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2: Unpivot the monthly market statistics\n", "\n", - "The `realestate_lab.bronze.market_stats` table has one row per city and 12 monthly price columns. Use the `UNPIVOT` clause to transform it into long format with columns: `city`, `month`, `avg_price`.\n", + "The `realestate_lab2.bronze.market_stats` table has one row per city and 12 monthly price columns. Use the `UNPIVOT` clause to transform it into long format with columns: `city`, `month`, `avg_price`.\n", "\n", - "> 🤖 **Genie Code tip:** Ask Genie Code for the UNPIVOT syntax in Databricks SQL.\n", + "> \uD83E\uDD16 **Genie Code tip:** Ask Genie Code for the UNPIVOT syntax in Databricks SQL.\n", "> *Suggested prompt: \"How do I unpivot 12 monthly columns (Jan through Dec) into rows using the UNPIVOT clause in Databricks SQL?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "e14c95b7", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "5fd12210-a808-4e39-be1a-9d6d316481ff", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Task 2: Unpivot the 12 monthly columns into rows\n", - "-- Expected columns: city, month, avg_price\n" + "-- Expected columns: city, month, avg_price\n", + "\n", + "SELECT city, month, avg_price\n", + "FROM realestate_lab2.bronze.market_stats\n", + "UNPIVOT(\n", + " avg_price FOR month IN (`Jan`, `Feb`, `Mar`, `Apr`, `May`, `Jun`, `Jul`, `Aug`, `Sep`, `Oct`, `Nov`, `Dec`)\n", + ")\n", + "\n" ] }, { "cell_type": "markdown", - "id": "0b5c511b", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1602000c-a76c-427f-81d6-5767fe46f8fb", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Clean up\n", "\n", @@ -690,22 +1274,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "eb9c8e93", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "4b20abcc-8f69-4208-aafd-e3f30c109552", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Uncomment and run to remove all lab resources when you are done\n", - "-- DROP CATALOG IF EXISTS realestate_lab CASCADE;" + "-- DROP CATALOG IF EXISTS realestate_lab2 CASCADE;" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 5158275255374054, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "08-cleanse-transform-load-data-into-unity-catalog", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/09-implement-manage-data-quality-constraints-unity-catalog.ipynb b/Allfiles/09-implement-manage-data-quality-constraints-unity-catalog.ipynb index 6eb73ab..0605b46 100644 --- a/Allfiles/09-implement-manage-data-quality-constraints-unity-catalog.ipynb +++ b/Allfiles/09-implement-manage-data-quality-constraints-unity-catalog.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "a1b2c3d4", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "8d373cfc-afe0-4c41-9bd1-ee67eb6f1e63", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Set up the ClearCover Insurance Data Platform\n", "\n", @@ -14,26 +22,45 @@ }, { "cell_type": "markdown", - "id": "b1c2d3e4", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "9a98d3f7-a044-4c95-8ee5-9dece10fba35", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 1: Create the Unity Catalog structure\n", "\n", "This cell is provided for you — run it to prepare your environment.\n", "\n", "It creates the following Unity Catalog objects:\n", - "- A **catalog** named `insurance_lab` — the top-level namespace for the ClearCover Insurance platform\n", - "- A **bronze schema** inside `insurance_lab` — raw claims data, exactly as it arrives from source systems\n", - "- A **silver schema** inside `insurance_lab` — validated and type-safe claims\n", - "- A **gold schema** inside `insurance_lab` — aggregated reporting data\n", - "- A **volume** named `raw_files` inside `insurance_lab.bronze` — landing zone for incoming CSV claim files" + "- A **catalog** named `insurance_lab3` — the top-level namespace for the ClearCover Insurance platform\n", + "- A **bronze schema** inside `insurance_lab3` — raw claims data, exactly as it arrives from source systems\n", + "- A **silver schema** inside `insurance_lab3` — validated and type-safe claims\n", + "- A **gold schema** inside `insurance_lab3` — aggregated reporting data\n", + "- A **volume** named `raw_files` inside `insurance_lab3.bronze` — landing zone for incoming CSV claim files" ] }, { "cell_type": "code", - "execution_count": null, - "id": "a4d23c46", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "cad21f1a-85f6-4059-b729-b42efaa8a5c1", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -50,7 +77,7 @@ "print (f\"Storage root: {storage_root}\")\n", "\n", "spark.sql(f\"\"\"\n", - " CREATE CATALOG IF NOT EXISTS insurance_lab\n", + " CREATE CATALOG IF NOT EXISTS insurance_lab3\n", " MANAGED LOCATION '{storage_root}'\n", " COMMENT 'ClearCover Insurance analytics platform'\n", "\"\"\")" @@ -58,29 +85,49 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "444c9a4c", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "07e7253c-333e-4fe0-88de-6d9cc2edf337", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "CREATE SCHEMA IF NOT EXISTS insurance_lab.bronze\n", + "CREATE SCHEMA IF NOT EXISTS insurance_lab3.bronze\n", " COMMENT 'Raw claims data, unmodified as received from source systems';\n", "\n", - "CREATE SCHEMA IF NOT EXISTS insurance_lab.silver\n", + "CREATE SCHEMA IF NOT EXISTS insurance_lab3.silver\n", " COMMENT 'Validated claims, type-checked and quality-constrained';\n", "\n", - "CREATE SCHEMA IF NOT EXISTS insurance_lab.gold\n", + "CREATE SCHEMA IF NOT EXISTS insurance_lab3.gold\n", " COMMENT 'Aggregated reporting data for dashboards';\n", "\n", - "CREATE VOLUME IF NOT EXISTS insurance_lab.bronze.raw_files\n", + "CREATE VOLUME IF NOT EXISTS insurance_lab3.bronze.raw_files\n", " COMMENT 'Landing zone for raw CSV claim files';" ] }, { "cell_type": "markdown", - "id": "d1e2f3a4", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "17aba343-e24b-47f2-a63c-3b96e5473d51", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 2: Write the raw claims CSV to the volume\n", "\n", @@ -91,13 +138,24 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "e1f2a3b4", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "141cb298-eda0-4b30-9314-b2bc5e38f749", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — writes the raw claims CSV to the volume\n", - "volume_path = '/Volumes/insurance_lab/bronze/raw_files'\n", + "volume_path = '/Volumes/insurance_lab3/bronze/raw_files'\n", "\n", "rows = [\n", " 'claim_id,policy_id,customer_id,claim_date,incident_date,claim_amount,claim_type,status,coverage_amount,agent_code',\n", @@ -129,8 +187,16 @@ }, { "cell_type": "markdown", - "id": "f1a2b3c4", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "197c66fb-e313-4737-9c15-2fe4dbba68bf", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Task 3: Load the CSV into the bronze Delta table\n", "\n", @@ -139,9 +205,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "a2b3c4d5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "43c3b43b-a1a1-4c4f-abe0-7ef6fb040745", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — loads the CSV into the bronze Delta table\n", @@ -160,7 +237,7 @@ " StructField('agent_code', StringType(), True),\n", "])\n", "\n", - "volume_path = '/Volumes/insurance_lab/bronze/raw_files'\n", + "volume_path = '/Volumes/insurance_lab3/bronze/raw_files'\n", "\n", "df = (\n", " spark.read\n", @@ -174,17 +251,25 @@ " df.write\n", " .format('delta')\n", " .mode('overwrite')\n", - " .saveAsTable('insurance_lab.bronze.claims_raw')\n", + " .saveAsTable('insurance_lab3.bronze.claims_raw')\n", ")\n", "\n", - "print('✅ Table insurance_lab.bronze.claims_raw created')\n", + "print('✅ Table insurance_lab3.bronze.claims_raw created')\n", "print(f' Total rows loaded: {df.count()}')" ] }, { "cell_type": "markdown", - "id": "b2c3d4e5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "352a4aa2-bd20-49c0-9aab-3554165d5752", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### Verify: Inspect the raw data\n", "\n", @@ -195,32 +280,64 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "c2d3e4f5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "41759466-531e-48a0-9bad-5a7af81fb5ae", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "SELECT *\n", - "FROM insurance_lab.bronze.claims_raw\n", + "FROM insurance_lab3.bronze.claims_raw\n", "ORDER BY claim_id NULLS LAST;" ] }, { "cell_type": "code", - "execution_count": null, - "id": "d2e3f4a5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "implicitDf": true, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "426d6d0c-eeb5-41c0-a689-23df6031fb4e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", - "DESCRIBE TABLE insurance_lab.bronze.claims_raw;" + "DESCRIBE TABLE insurance_lab3.bronze.claims_raw;" ] }, { "cell_type": "markdown", - "id": "e2f3a4b5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "b51a51b8-6dcc-4cb5-964d-6ce920fe6a43", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "---\n", "\n", @@ -229,22 +346,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "f2a3b4c5", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "748384a6-fec3-45c2-a93d-3208ddd5a9fb", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "%sql\n", "-- Optional cleanup — uncomment to remove all lab resources when done\n", - "-- DROP CATALOG IF EXISTS insurance_lab CASCADE;" + "-- DROP CATALOG IF EXISTS insurance_lab3 CASCADE;" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": 4831926123628078, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "09-implement-manage-data-quality-constraints-unity-catalog", + "widgets": {} + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/Allfiles/09-implement-manage-data-quality-constraints.py b/Allfiles/09-implement-manage-data-quality-constraints.py index 8943189..7fb0da9 100644 --- a/Allfiles/09-implement-manage-data-quality-constraints.py +++ b/Allfiles/09-implement-manage-data-quality-constraints.py @@ -93,4 +93,74 @@ def claims_summary(): count('claim_id').alias('claim_count'), spark_sum('claim_amount').alias('total_claim_amount') ) - ) \ No newline at end of file + ) + + + +# ============================================================== +# ClearCover Insurance — Claims Data Quality Pipeline +# INSTRUCTOR ANSWER KEY +# ============================================================== + +from pyspark import pipelines as dp +from pyspark.sql.functions import expr, col, count, sum as spark_sum + + +# -------------------------------------------------------------- +# Exercise 3 + 4: Nullability, Status, and Data Type Validation +# -------------------------------------------------------------- + +@dp.table(name='silver.claims_validated') +@dp.expect_or_drop('valid_claim_id', 'claim_id IS NOT NULL') +@dp.expect_or_drop('valid_customer_id', 'customer_id IS NOT NULL') +@dp.expect( 'valid_status', "status IN ('OPEN', 'PENDING', 'CLOSED')") +@dp.expect_or_fail('valid_coverage', 'coverage_amount > 0') +@dp.expect_or_drop('valid_claim_date', 'claim_date IS NOT NULL') +@dp.expect_or_drop('valid_claim_amount', 'claim_amount IS NOT NULL') +@dp.expect_or_drop('non_negative_amount', 'claim_amount >= 0') +def claims_validated(): + '''Silver: validated insurance claims with full quality constraints applied.''' + return ( + spark.readStream + .table('insurance_lab3.bronze.claims_raw') + .withColumn('claim_date', expr('try_cast(claim_date AS date)')) + .withColumn('claim_amount', expr('try_cast(claim_amount AS decimal(12,2))')) + ) + + +# -------------------------------------------------------------- +# Exercise 5: Schema Drift — Rescued Data +# -------------------------------------------------------------- + +@dp.table(name='silver.claims_rescued') +def claims_rescued(): + '''Silver: raw claims loaded via Auto Loader with rescue schema evolution.''' + return ( + spark.readStream + .format('cloudFiles') + .option('cloudFiles.format', 'csv') + .option('header', 'true') + .option('cloudFiles.schemaLocation', '/Volumes/insurance_lab3/bronze/raw_files/_schema') + .option('cloudFiles.schemaEvolutionMode', 'rescue') + .option('rescuedDataColumn', '_rescued_data') + .option('cloudFiles.inferColumnTypes', 'true') + .load('/Volumes/insurance_lab3/bronze/raw_files/') + ) + + +# -------------------------------------------------------------- +# Gold: Claims Summary +# -------------------------------------------------------------- + +@dp.table(name='gold.claims_summary') +def claims_summary(): + '''Gold: aggregate claim counts and total amounts per type and status.''' + return ( + spark.read.table('insurance_lab3.silver.claims_validated') + .groupBy('claim_type', 'status') + .agg( + count('claim_id').alias('claim_count'), + spark_sum('claim_amount').alias('total_claim_amount') + ) + ) + diff --git a/Allfiles/10-design-implement-data-pipelines.ipynb b/Allfiles/10-design-implement-data-pipelines.ipynb index 6c73e2a..4b5986f 100644 --- a/Allfiles/10-design-implement-data-pipelines.ipynb +++ b/Allfiles/10-design-implement-data-pipelines.ipynb @@ -2,8 +2,16 @@ "cells": [ { "cell_type": "markdown", - "id": "5e4649f5", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "7a8b4d37-3730-4414-969f-aaae3e4894c6", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 1: Ingest Raw Booking Data (Bronze Layer)\n", "\n", @@ -14,13 +22,21 @@ }, { "cell_type": "markdown", - "id": "b296f73c", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "864d678e-1742-4dcd-99e9-95ba7b205dba", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "### ✅ Provided: Create the Unity Catalog structure and sample data\n", "\n", "Run the two cells below to:\n", - "- Create a catalog `hospitality_lab` with `bronze`, `silver`, and `gold` schemas\n", + "- Create a catalog `hospitality_lab2` with `bronze`, `silver`, and `gold` schemas\n", "- Create a sample raw bookings DataFrame containing deliberate quality issues\n", "\n", "Inspect the output carefully — you will need to understand the data issues before tackling Exercise 2." @@ -28,9 +44,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "c51589af", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b562ca14-4f86-4ca4-9f96-3f73fbab2b65", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# Since no default storage is enabled, we are inheriting the storage path from the default catalog's root.\n", @@ -49,14 +76,25 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "db1bf741", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "b804b8b7-39a4-44f3-9c32-e9b8a40bcc5b", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — creates the Unity Catalog structure\n", "\n", - "catalog = \"hospitality_lab\"\n", + "catalog = \"hospitality_lab2\"\n", "\n", "spark.sql(f\"CREATE CATALOG IF NOT EXISTS {catalog} MANAGED LOCATION '{storage_root}' COMMENT 'GlobStay hospitality analytics platform'\")\n", "spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {catalog}.bronze COMMENT 'Raw ingested booking data'\")\n", @@ -68,9 +106,20 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "bdda33dc", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "a1cb49d4-7833-4ca5-bc24-3f38a89f768a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# ✅ Run this cell — creates the raw bookings DataFrame\n", @@ -121,36 +170,67 @@ }, { "cell_type": "markdown", - "id": "3dcab995", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "a56b1a85-3232-4184-a072-098b25ec21e4", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge: Write raw data to a Bronze table\n", + "### \uD83C\uDFAF Challenge: Write raw data to a Bronze table\n", "\n", - "Write `raw_df` to a Delta table named **`bookings_bronze`** in the `hospitality_lab.bronze` schema.\n", + "Write `raw_df` to a Delta table named **`bookings_bronze`** in the `hospitality_lab2.bronze` schema.\n", "\n", "**Requirements:**\n", "- Use `overwrite` mode so the cell is re-runnable\n", "- After writing, verify the record count matches the raw DataFrame\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"How do I write a PySpark DataFrame to a Unity Catalog table using saveAsTable with overwrite mode?\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"How do I write a PySpark DataFrame to a Unity Catalog table using saveAsTable with overwrite mode?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "65c3f068", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "af70fc84-d414-4407-b1a6-f7d6912a4e9a", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", - "# Write raw_df to hospitality_lab.bronze.bookings_bronze (overwrite mode)\n", - "# Then verify the record count\n" + "# Write raw_df to hospitality_lab2.bronze.bookings_bronze (overwrite mode)\n", + "# Then verify the record count\n", + "raw_df.write.mode(\"overwrite\").saveAsTable(\"hospitality_lab2.bronze.bookings_bronze\")\n", + "\n", + "count= spark.table(\"hospitality_lab2.bronze.bookings_bronze\").count()\n", + "print(f\"Bronze table record count: {count}\")\n" ] }, { "cell_type": "markdown", - "id": "70c72564", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "8760880a-fa2d-4421-9f05-945079b42310", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 2: Clean and Validate Booking Data (Silver Layer)\n", "\n", @@ -168,12 +248,20 @@ }, { "cell_type": "markdown", - "id": "1ac54c18", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "712f8da4-af5a-49f9-9103-34496878dfd6", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge: Create the Silver table\n", + "### \uD83C\uDFAF Challenge: Create the Silver table\n", "\n", - "Read from `hospitality_lab.bronze.bookings_bronze`, apply all four cleaning rules, and write the result to **`hospitality_lab.silver.bookings_silver`** (overwrite mode).\n", + "Read from `hospitality_lab2.bronze.bookings_bronze`, apply all four cleaning rules, and write the result to **`hospitality_lab2.silver.bookings_silver`** (overwrite mode).\n", "\n", "**Hints:**\n", "- Use `.dropDuplicates([\"booking_id\"])` to deduplicate\n", @@ -183,25 +271,64 @@ "\n", "After writing, print both the Bronze and Silver record counts to confirm records were removed.\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"How do I filter PySpark rows where check_out_date is not after check_in_date using date comparison?\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"How do I filter PySpark rows where check_out_date is not after check_in_date using date comparison?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "6e092360", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "f55d9d6d-65ff-4de8-ac8e-721b1bcc2280", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "# Read from bronze, apply cleaning rules, write to silver\n", - "# Print bronze count vs silver count\n" + "# Print bronze count vs silver count\n", + "from pyspark.sql.functions import col, to_date, datediff\n", + "\n", + "\n", + "# Read from Bronze\n", + "bronze_df = spark.table(\"hospitality_lab2.bronze.bookings_bronze\")\n", + "\n", + "# Apply cleaning rules\n", + "silver_df = (\n", + " bronze_df.dropDuplicates([\"booking_id\"]).filter(col(\"room_type\").isNotNull()).filter(to_date(col(\"check_out_date\")) > to_date(col(\"check_in_date\"))).filter(col(\"rate_per_night\") > 0)\n", + ")\n", + "\n", + "# Write to Silver\n", + "silver_df.write.mode(\"overwrite\").saveAsTable(\"hospitality_lab2.silver.bookings_silver\")\n", + "\n", + "# Verify counts\n", + "bronze_count = bronze_df.count()\n", + "silver_count = spark.table(\"hospitality_lab2.silver.bookings_silver\").count()\n", + "print(f\"Bronze records: {bronze_count}\")\n", + "print(f\"Silver records: {silver_count}\")\n", + "print(f\"Records removed: {bronze_count - silver_count}\")\n" ] }, { "cell_type": "markdown", - "id": "7706b57d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "286dc521-be9e-47e0-a45d-b17d2ce5dc5e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 3: Build Analytics-Ready Data (Gold Layer)\n", "\n", @@ -213,68 +340,143 @@ }, { "cell_type": "markdown", - "id": "b6930a2f", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "fd152856-d5c2-4899-b47c-59ed7e883490", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge 3a: Property performance table\n", + "### \uD83C\uDFAF Challenge 3a: Property performance table\n", "\n", - "Create a Gold table `hospitality_lab.gold.property_performance` using Spark SQL.\n", + "Create a Gold table `hospitality_lab2.gold.property_performance` using Spark SQL.\n", "\n", "**Requirements:**\n", - "- Source: `hospitality_lab.silver.bookings_silver`\n", + "- Source: `hospitality_lab2.silver.bookings_silver`\n", "- Filter to `booking_status = 'confirmed'` only\n", "- Columns: `property_id`, `property_name`, `confirmed_bookings`, `total_nights`, `total_revenue`\n", "- Calculate `total_nights` using `DATEDIFF(check_out_date, check_in_date)`\n", "- Calculate `total_revenue` as `total_nights × rate_per_night`\n", "- Order by `total_revenue DESC`\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"Write a Spark SQL CREATE OR REPLACE TABLE statement that aggregates booking revenue per property using DATEDIFF and SUM.\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"Write a Spark SQL CREATE OR REPLACE TABLE statement that aggregates booking revenue per property using DATEDIFF and SUM.\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "cf691474", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "72eb4d43-cb13-4e8a-bdb4-6ea455f35ced", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE — use %sql or spark.sql()\n", - "# Create hospitality_lab.gold.property_performance\n" + "# Create hospitality_lab2.gold.property_performance\n", + "spark.sql(\"\"\"\n", + " CREATE OR REPLACE TABLE hospitality_lab2.gold.property_performance AS \n", + " SELECT \n", + " property_id,\n", + " property_name, \n", + " COUNT(*) AS confirmed_bookings, \n", + " SUM(DATEDIFF(check_out_date, check_in_date)) AS total_nights, \n", + " SUM(DATEDIFF(check_out_date, check_in_date) * rate_per_night) as total_revenue\n", + " FROM hospitality_lab2.silver.bookings_silver\n", + " WHERE booking_status = 'confirmed'\n", + " GROUP BY property_id, property_name\n", + " ORDER BY total_revenue DESC\n", + "\"\"\")\n", + "\n", + "display(spark.table(\"hospitality_lab2.gold.property_performance\"))" ] }, { "cell_type": "markdown", - "id": "fd050c0f", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "1a361574-2871-4781-8b60-fae84f2d4f76", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge 3b: Booking channel performance table\n", + "### \uD83C\uDFAF Challenge 3b: Booking channel performance table\n", "\n", - "Create a Gold table `hospitality_lab.gold.channel_performance` showing how each booking channel contributes to GlobStay revenue.\n", + "Create a Gold table `hospitality_lab2.gold.channel_performance` showing how each booking channel contributes to GlobStay revenue.\n", "\n", "**Requirements:**\n", - "- Source: `hospitality_lab.silver.bookings_silver`\n", + "- Source: `hospitality_lab2.silver.bookings_silver`\n", "- Filter to `booking_status = 'confirmed'` only\n", "- Columns: `channel`, `booking_count`, `total_revenue`, `avg_revenue_per_booking` (rounded to 2 decimal places)\n", "- Order by `total_revenue DESC`\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"How do I calculate average revenue per booking in Spark SQL using ROUND and AVG?\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"How do I calculate average revenue per booking in Spark SQL using ROUND and AVG?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "8545be2d", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "8fdb04a4-b837-4eb6-ab13-ea16fe8a4a23", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", - "# Create hospitality_lab.gold.channel_performance\n" + "# Create hospitality_lab2.gold.channel_performance\n", + "spark.sql(\"\"\"\n", + " CREATE OR REPLACE TABLE hospitality_lab2.gold.channel_performance AS\n", + " SELECT\n", + " channel,\n", + " COUNT(*) AS booking_count,\n", + " SUM(DATEDIFF(check_out_date, check_in_date) * rate_per_night) AS total_revenue,\n", + " ROUND(SUM(DATEDIFF(check_out_date, check_in_date) * rate_per_night) / COUNT(*), 2) AS avg_revenue_per_booking\n", + " FROM hospitality_lab2.silver.bookings_silver\n", + " WHERE booking_status = 'confirmed'\n", + " GROUP BY channel\n", + " ORDER BY total_revenue DESC\n", + "\"\"\")\n", + "\n", + "display(spark.table(\"hospitality_lab2.gold.channel_performance\"))" ] }, { "cell_type": "markdown", - "id": "7c2c833f", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "6bdf010e-ab40-4201-822c-305520217f15", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 4: Error Handling in Pipeline Notebooks\n", "\n", @@ -286,10 +488,18 @@ }, { "cell_type": "markdown", - "id": "7d93cc07", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "5914f84d-5451-4507-8bb4-2a5ee27db878", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge 4a: Implement safe table reading with try/except\n", + "### \uD83C\uDFAF Challenge 4a: Implement safe table reading with try/except\n", "\n", "Write a function `safe_read_table(table_name)` that:\n", "1. Tries to read the table using `spark.table(table_name)`\n", @@ -304,27 +514,72 @@ "- Use `e.getErrorClass()` to get a structured error code\n", "- Re-raise unexpected errors that you haven't handled\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"Show me how to catch a PySparkException in Python and print its error class.\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"Show me how to catch a PySparkException in Python and print its error class.\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "bbf9248b", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "8d645962-fbab-4721-9ad8-04bb2ada303d", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "# Define safe_read_table(table_name) with try/except\n", - "# Test with valid and invalid table names\n" + "# Test with valid and invalid table names\n", + "\n", + "from pyspark.errors import PySparkException\n", + "\n", + "def safe_read_table(table_name):\n", + " try:\n", + " df = spark.table(table_name)\n", + " df.count() # force evaluation to trigger errors\n", + " print(f\"Successfully read table: {table_name}\")\n", + " return df\n", + " except PySparkException as e:\n", + " error_class = e.getErrorClass()\n", + " if error_class == \"TABLE_OR_VIEW_NOT_FOUND\":\n", + " print(f\"Table not found: {table_name} (error class: {error_class})\")\n", + " return None\n", + " else:\n", + " print(f\"Unexpected Spark error: {error_class}\")\n", + " raise\n", + "\n", + "# Test with a valid table\n", + "valid_df = safe_read_table(\"hospitality_lab2.silver.bookings_silver\")\n", + "\n", + "# Test with a non-existent table\n", + "missing_df = safe_read_table(\"hospitality_lab2.silver.nonexistent_table\")\n", + "print(f\"\\nValid result: {type(valid_df)}\")\n", + "print(f\"Missing result: {missing_df}\")\n", + "\n" ] }, { "cell_type": "markdown", - "id": "ea18275d", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "88c6a6a7-0c32-4e92-b863-b9aa786fa094", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge 4b: Signal task outcome with dbutils.notebook.exit()\n", + "### \uD83C\uDFAF Challenge 4b: Signal task outcome with dbutils.notebook.exit()\n", "\n", "When a notebook runs as a Lakeflow Job task, calling `dbutils.notebook.exit()` lets downstream tasks read the outcome using `{{tasks..values.}}`.\n", "\n", @@ -335,25 +590,59 @@ "\n", "> ⚠️ **Warning:** `dbutils.notebook.exit()` halts notebook execution immediately. Define the function but **do not call it** — instead, leave the call commented out. In the job, calling it is the correct behaviour.\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"How does dbutils.notebook.exit work in Azure Databricks when a notebook runs as a job task?\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"How does dbutils.notebook.exit work in Azure Databricks when a notebook runs as a job task?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "b2dfd397", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "d7467f67-bff2-4866-9c0d-9ace4e68cd07", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "# Define report_pipeline_outcome(table_name)\n", - "# Leave the function call commented out\n" + "# Leave the function call commented out\n", + "\n", + "def report_pipeline_outcome(table_name):\n", + " try:\n", + " df = spark.table(table_name)\n", + " count = df.count()\n", + " dbutils.notebook.exit(f\"SUCCESS: Processed {count} records from {table_name}\")\n", + " except Exception as e:\n", + " dbutils.notebook.exit(f\"FAILED: {e}\")\n", + "\n", + "# Do NOT call this function interactively — it halts notebook execution.\n", + "# Uncomment the line below only when running as a Lakeflow Job task.\n", + "# report_pipeline_outcome(\"hospitality_lab2.silver.bookings_silver\")\n", + "\n", + "print(\"report_pipeline_outcome() defined (call commented out to avoid halting notebook)\")\n", + "\n" ] }, { "cell_type": "markdown", - "id": "bc282a2f", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "c5e381b4-5228-4d26-b436-13b5cd2b5c1e", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ "## Exercise 5: Parameterized Notebook Tasks\n", "\n", @@ -366,41 +655,82 @@ }, { "cell_type": "markdown", - "id": "d840a869", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "00ccb6f6-a25f-4935-926b-995897e6cf71", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge 5a: Add a runtime widget parameter\n", + "### \uD83C\uDFAF Challenge 5a: Add a runtime widget parameter\n", "\n", "Add a text widget named `booking_status` with default value `confirmed` and label `Booking Status Filter`.\n", "\n", - "Then retrieve the widget value and use it to filter `hospitality_lab.silver.bookings_silver`. Display the filtered result and print the record count.\n", + "Then retrieve the widget value and use it to filter `hospitality_lab2.silver.bookings_silver`. Display the filtered result and print the record count.\n", "\n", "**Hints:**\n", "- `dbutils.widgets.text(name, default, label)`\n", "- `dbutils.widgets.get(name)` returns the current value\n", "- In a Lakeflow Job, the task **Parameters** configuration overrides the default value at runtime\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"How do I create a text widget in Databricks and retrieve its value to filter a DataFrame?\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"How do I create a text widget in Databricks and retrieve its value to filter a DataFrame?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "bfa32172", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "20f9cd14-44ef-49f5-a6ab-a030bfebfbec", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "# Add a widget 'booking_status' (default: 'confirmed', label: 'Booking Status Filter')\n", - "# Retrieve the value and filter bookings_silver, then display results\n" + "# Retrieve the value and filter bookings_silver, then display results\n", + "\n", + "# Add a widget 'booking_status' (default: 'confirmed', label: 'Booking Status Filter')\n", + "dbutils.widgets.text(\"booking_status\", \"confirmed\", \"Booking Status Filter\")\n", + "\n", + "# Retrieve the widget value\n", + "status_filter = dbutils.widgets.get(\"booking_status\")\n", + "print(f\"Filtering by booking_status = '{status_filter}'\")\n", + "\n", + "# Filter bookings_silver and display\n", + "silver_df = spark.table(\"hospitality_lab2.silver.bookings_silver\")\n", + "filtered_df = silver_df.filter(col(\"booking_status\") == status_filter)\n", + "\n", + "display(filtered_df)\n", + "print(f\"\\nRecords with status '{status_filter}': {filtered_df.count()}\")\n" ] }, { "cell_type": "markdown", - "id": "1514f8d4", - "metadata": {}, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "inputWidgets": {}, + "nuid": "2d1c0dd4-c3f0-4d8b-b4af-b053cc914712", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "source": [ - "### 🎯 Challenge 5b: Pass values to downstream tasks\n", + "### \uD83C\uDFAF Challenge 5b: Pass values to downstream tasks\n", "\n", "After processing, a notebook task can share computed values with downstream tasks using `dbutils.jobs.taskValues.set()`.\n", "\n", @@ -412,27 +742,96 @@ "\n", "> **Note:** `dbutils.jobs.taskValues` is only available inside a Lakeflow Job run. In an interactive notebook session it prints a warning and is a no-op — this is expected.\n", "\n", - "> 🤖 **Genie Code tip:** Ask *\"How do I use dbutils.jobs.taskValues.set to pass a list value from one Databricks notebook task to another?\"*" + "> \uD83E\uDD16 **Genie Code tip:** Ask *\"How do I use dbutils.jobs.taskValues.set to pass a list value from one Databricks notebook task to another?\"*" ] }, { "cell_type": "code", - "execution_count": null, - "id": "38082ea9", - "metadata": {}, + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "inputWidgets": {}, + "nuid": "71f6f199-fa68-4a12-bf9d-02c4c318f3bc", + "showTitle": false, + "tableResultSettingsMap": {}, + "title": "" + } + }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "# Set task values: 'silver_record_count' and 'properties_processed'\n", - "# Then print the values to verify\n" + "# Then print the values to verify\n", + "\n", + "# Read Silver table\n", + "silver_df = spark.table(\"hospitality_lab2.silver.bookings_silver\")\n", + "\n", + "# Compute task values\n", + "silver_record_count = silver_df.count()\n", + "properties_processed = [row.property_id for row in silver_df.select(\"property_id\").distinct().collect()]\n", + "\n", + "# Set task values (no-op in interactive mode, functional in a Lakeflow Job)\n", + "dbutils.jobs.taskValues.set(key=\"silver_record_count\", value=silver_record_count)\n", + "dbutils.jobs.taskValues.set(key=\"properties_processed\", value=properties_processed)\n", + "\n", + "# Print to verify\n", + "print(f\"silver_record_count: {silver_record_count}\")\n", + "print(f\"properties_processed: {properties_processed}\")\n" ] } ], "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "pythonIndentUnit": 4 + }, + "notebookName": "10-design-implement-data-pipelines", + "widgets": { + "booking_status": { + "currentValue": "confirmed", + "nuid": "8ec86915-0c01-493f-b27f-c3df39d8efde", + "typedWidgetInfo": { + "autoCreated": false, + "defaultValue": "confirmed", + "label": "Booking Status Filter", + "name": "booking_status", + "options": { + "widgetDisplayType": "Text", + "validationRegex": null + }, + "parameterDataType": "String", + "dynamic": false + }, + "widgetInfo": { + "widgetType": "text", + "defaultValue": "confirmed", + "label": "Booking Status Filter", + "name": "booking_status", + "options": { + "widgetType": "text", + "autoCreated": false, + "validationRegex": null + } + } + } + } + }, "language_info": { "name": "python" } }, "nbformat": 4, - "nbformat_minor": 5 -} + "nbformat_minor": 0 +} \ No newline at end of file