Description
The WAF reload data job completes with a partial status because four cost optimization queries reference columns that cannot be resolved in the corresponding Databricks system tables.
The installation completed without any reported errors or warnings. However, when running the reload data job, Cell 4 reports four UNRESOLVED_COLUMN.WITH_SUGGESTION errors.
Observed behavior
The following controls fail during Cell 4 of the reload notebook:
CO-01-02: Job vs All-Purpose Cluster Usage
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter
with name `cluster_type` cannot be resolved.
Did you mean one of the following?
[`usage_type`, `custom_tags`, `record_type`, `usage_date`, `sku_name`]
SQLSTATE: 42703
CO-01-06: Serverless Cost Efficiency
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter
with name `list_price` cannot be resolved.
Did you mean one of the following?
[`sku_name`, `usage_unit`, `cloud`, `custom_tags`, `record_id`]
SQLSTATE: 42703
CO-02-01: Auto-Termination Savings
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter
with name `start_time` cannot be resolved.
Did you mean one of the following?
[`change_time`, `create_time`, `delete_time`, `tags`, `cluster_id`]
SQLSTATE: 42703
CO-01-09: Photon vs Standard Performance Cost
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter
with name `photon_enabled` cannot be resolved.
Did you mean one of the following?
[`u`.`sku_name`, `q`.`compute`, `q`.`end_time`, `u`.`cloud`, `q`.`read_bytes`]
SQLSTATE: 42703
Reload result
Cell 7 reports the following result:
✅ Run 2 complete: 73 succeeded, 4 failed → partial
Failed: waf_co_01_02_chart, waf_co_01_06_chart, waf_co_02_01_chart, waf_co_01_09_chart
The Databricks App displays:
Implication
The affected queries appear to rely on columns that are not available in the system table schemas used by the current Databricks environment.
As a result:
- The reload job cannot complete successfully.
- Four WAF cost optimization controls are not populated.
- The application reports a partial run with only 73 of 77 tables successfully refreshed.
- The related dashboard charts and WAF scores may be incomplete or inaccurate.
Expected behavior
The reload data job should complete successfully and populate all 77 result tables.
The affected queries should be updated to derive the required values from columns available in the current Databricks system table schemas, or handle schema differences gracefully.
At minimum, the tool should:
- Validate required source columns before executing each control query.
- Use supported system table columns or documented joins to derive:
- Job versus all-purpose compute classification
- Applicable list pricing
- Cluster start or activity time
- Photon versus standard engine classification
- Return a clear compatibility message when the required data is unavailable.
- Document any required system table schemas, regional availability, prerequisites, or Databricks version dependencies.
Environment
- Azure Databricks workspace
- Unity Catalog enabled
- Installation completed without errors or warnings
- Reload data job started from the deployed WAF application
- Reload run result:
partial
- Successful result tables:
73/77
- Failed result tables:
waf_co_01_02_chartwaf_co_01_06_chartwaf_co_02_01_chartwaf_co_01_09_chart
Why this matters
Reliable schema compatibility is important because the affected controls contribute to cost and performance assessment results.
Silent installation success followed by runtime query failures can make the deployment appear healthy while producing incomplete WAF scores and recommendations. Earlier schema validation or compatible query logic would make the failure easier to diagnose and prevent incomplete assessment results.
Description
The WAF reload data job completes with a partial status because four cost optimization queries reference columns that cannot be resolved in the corresponding Databricks system tables.
The installation completed without any reported errors or warnings. However, when running the reload data job, Cell 4 reports four
UNRESOLVED_COLUMN.WITH_SUGGESTIONerrors.Observed behavior
The following controls fail during Cell 4 of the reload notebook:
CO-01-02: Job vs All-Purpose Cluster Usage
CO-01-06: Serverless Cost Efficiency
CO-02-01: Auto-Termination Savings
CO-01-09: Photon vs Standard Performance Cost
Reload result
Cell 7 reports the following result:
The Databricks App displays:
Implication
The affected queries appear to rely on columns that are not available in the system table schemas used by the current Databricks environment.
As a result:
Expected behavior
The reload data job should complete successfully and populate all 77 result tables.
The affected queries should be updated to derive the required values from columns available in the current Databricks system table schemas, or handle schema differences gracefully.
At minimum, the tool should:
Environment
partial73/77waf_co_01_02_chartwaf_co_01_06_chartwaf_co_02_01_chartwaf_co_01_09_chartWhy this matters
Reliable schema compatibility is important because the affected controls contribute to cost and performance assessment results.
Silent installation success followed by runtime query failures can make the deployment appear healthy while producing incomplete WAF scores and recommendations. Earlier schema validation or compatible query logic would make the failure easier to diagnose and prevent incomplete assessment results.