End-to-End Greenfield Install Test for install.ipynb

[AbhiDatabricks]

Overview

Validate that a first-time (greenfield) run of install.ipynb successfully deploys the full WAF Assessment Tool stack on a clean Databricks workspace.

Scope

The E2E test should verify all 15 notebook cells complete without error and that each deployed artifact is functional:

• Cell 1–3: Params, catalog/schema creation, waf_cache tables & waf_recommendations_not_met view
• Cell 4–6: WAF SQL queries load correctly from dashboard_queries.yaml
• Cell 7–9: Lakeview dashboard deployed and Genie Space linked via uiSettings.genieSpace.overrideId
• Cell 10–11: Streamlit App deployed and accessible; navigation works correctly
• Cell 12–13: WAF Reload Job created and first run succeeds
• Cell 14–15: All post-deploy health checks pass (waf_cache row counts > 0)

Acceptance Criteria

• install.ipynb can be run end-to-end via Run All on a workspace with no prior WAF artifacts
• All 14 waf_cache tables + waf_recommendations_not_met view are populated
• Lakeview dashboard loads and Genie AI Space responds to queries
• Streamlit App pages render and workspace_id filter works correctly
• WAF Reload Job runs successfully on schedule

Notes

• Test against a real Databricks workspace (no mocking)
• Document any manual prerequisite steps (cluster policy, PAT token, etc.)
• Capture any silent failures — especially Genie linking and App navigation bugs

[IlyasFardaouix]

I've taken a closer look at the E2E test for install.ipynb and I'm concerned that the test might not be properly handling the case where the waf_cache tables are not populated due to a missing or incorrect waf_cache schema. Specifically, I noticed that the redeploy_app.py script relies on environment variables like WAF_WAREHOUSE_ID and WAF_GENIE_URL to configure the WAF Reload Job, but it doesn't seem to account for the possibility that these variables might not be set. Have you considered adding a check to ensure that these variables are present and correctly configured before running the E2E test?