Fix KQL join pitfalls and adopt lookup for dimension enrichment

[RolandKrummenacher]

📝 Changes

Follow-up to a repo-wide review of KQL join vs lookup best practices (~880 join/lookup usages examined across hub database scripts, workbooks, ARG recommendation queries, the query catalog, and docs). This PR fixes the high-severity correctness bugs, adopts lookup where it is the documented best practice, and adds a lint rule so bare joins cannot come back. One commit per surface:

1. Workbooks (SavingsPlan, AHB ×2) — correctness fixes:
   • SavingsPlan summary/details: the final resourcecontainers join had no kind and defaulted to innerunique, deduplicating results by subscription and silently dropping all but one savings plan recommendation per subscription. Now kind=inner.
   • AHB "VM Latest Change Last 7 days": joined the resourcechanges record id against resources.id, which never match, so the tile was always empty. Now joins properties.targetResourceId (lowercased both sides) with kind=inner so mv-expanded license changes are preserved.
   • Get-SQL-AHB-Disabled/Enabled: bare join on VMName dropped SQL VMs with duplicate names (innerunique) and never matched uppercase names (left original case vs right tolower). Now joins properties.virtualMachineResourceId against the VM resource id.
2. ARG recommendation queries + finops-alerts logic app — every bare | join ( made an explicit kind=inner; Recommendations-Microsoft-SQLVMsWithoutAHB gets the same virtualMachineResourceId join-key fix as above. docs/deploy/finops-alerts-*.json are generated and intentionally untouched; the next release build picks up the bicep change.
3. Hub ingestion scripts — the v1_0/v1_2 EA transforms enriched cost rows from the open-data dimension tables (PricingUnits, Regions, ResourceTypes, Services) with join kind=leftouter; converted to lookup kind=leftouter (broadcasts the small dimension, no duplicated key columns; no downstream references to the suffixed columns existed). Also guards the Services enrichment against row fan-out with summarize take_any(...) by x_ResourceType — this was an active bug: Services.csv has 30 duplicate resource-type keys (up to ×31 for microsoft.sql/locations), so cost rows for those types were being multiplied.
4. Query catalog — tagging-policy-compliance, storage-tier-distribution, macc-consumption-vs-commitment: fact-to-small-dimension joins converted to lookup.
5. Docs — compute.md AHB examples (bare joins + the same VMName case bug, published as copy-paste guidance), commitment-coverage examples now teach lookup, invalid join ... on 1 == 1 percent-of-total examples rewritten with toscalar(), and fullouter examples now coalesce their join keys so baseline-only rows keep their dimension values.
6. Lint rule — new Tests/Lint/KqlJoinKinds.Tests.ps1 scans every KQL-carrying surface (hub scripts, catalog, recommendation queries, dashboard, logic app, workbooks, optimization engine, docs examples) and fails on any bare | join without an explicit kind=. The 48 remaining pre-existing bare joins (4 workbook files, all with unique left keys today) are baselined per file as a ratchet: counts can only go down. Two more surfaces were brought to zero in this commit: the SQL DB optimization runbook (2 bare joins, behavior-preserving kind=inner) and networking.md doc examples (2 bare joins the original review sweep missed).

✅ Validation

See the validation comment for live-environment results: ARG innerunique default proven with a controlled probe, Services fan-out confirmed from open data, ingestion lookup chain executed against a hub ADX cluster, and old-vs-new catalog queries verified equivalent on 1.16M cost rows. The new lint suite passes: 161/161.

⚠️ Not addressed here (follow-ups)

• Optimization Engine query quality: the review found the same patterns at larger scale (140 leftouter+isempty anti-join emulations in recommendations.json that can inflate counts, 38 subscription-dimension joins without dedup, a StorageReplication fan-out that double-counts cost). Separately owned surface; deserves its own PR. (Its two bare joins are fixed here so the lint starts clean.)
• 48 baselined bare joins + ~75 explicit kind=innerunique tag-filter semi-joins across workbooks (style only; left keys unique today). The lint baseline ratchets these down as files are touched.
• The SavingsPlan queries still contain 4 joins, above ARG''s documented 3-join guidance (pre-existing; ARG accepted them in testing).
• open-data README region refresh query could use lookup (maintenance-only query, left as is).

🤖 Generated with Claude Code

[RolandKrummenacher]

✅ Validation results (FTK test tenant + ftk-dev ADX cluster)

ARG semantics probe — confirmed ARG bare join defaults to innerunique: a 40-row left side with one distinct key returns 1 row bare vs 40 rows with kind=inner. The bug class this PR fixes is real.

Services open-data fan-out (commit 3) — confirmed against src/open-data/Services.csv: 30 resource types have duplicate rows, e.g. microsoft.apimanagement/service ×4 and microsoft.sql/locations ×31. Before this PR, the v1_0/v1_2 EA ingestion transforms multiplied every cost row for those resource types by that factor. The take_any dedup eliminates this.

Hub ingestion lookup chain (commit 3) — the exact edited fragment (PricingUnits → Regions → ResourceTypes → Services ×2) executed against the ftk-dev Ingestion database: compiles, returns the expected schema, no duplicated key columns, non-key conflicts suffixed identically to join (so downstream projections are unaffected).

Catalog queries (commit 4) — old vs new run back-to-back on ftk-dev Hub DB (1,162,170 Costs() rows):

• macc-consumption-vs-commitment: byte-identical
• storage-tier-distribution: identical rows (order differs; query has no order by)
• tagging-policy-compliance: identical to 9 decimal places (float summation order)

Recommendation queries + AHB/SavingsPlan workbook queries (commits 1-2) — all executed via the ARG REST API with parameters substituted: parse and run cleanly, incl. the 4-join SavingsPlan queries (ARG accepted 4 joins). Old vs new SavingsPlan counts match in this tenant (1 recommendation per subscription, so innerunique happens to coincide — the probe above shows the general case). resourcechanges.properties.targetResourceId verified populated and resource-id-shaped on 376/376 rows, confirming the new VM Latest Change join key; the old key (resourcechanges record id) can never match a resource id.

Not validated: row-level semantics of the SQL VM / public IP / app gateway queries (test tenant has no IaaS resources — all returned 0 rows, syntax-only), and the logic app queries (validated indirectly via identical shapes + bicep build).

🤖 Generated with Claude Code

[RolandKrummenacher]

✅ Validation round 2 — full coverage of everything testable

Completing the earlier validation, every remaining testable change has now been run against live services (ARG REST API on the FTK test tenant; ftk-dev ADX cluster for hub KQL):

ARG (now all executed):

• finops-alerts logicApp.bicep: both edited queries extracted from the bicep, variables('resourcesTable') substituted, executed — PASS.
• compute.md Windows-AHB + SQL-AHB examples — PASS.
• networking.md backendless app gateway + idle public IP examples — the idle-IP query failed on first run and exposed a pre-existing bug: the published example referenced table resource instead of resources (DisallowedLogicalTableName), so it never ran at all. Fixed in a follow-up commit; now PASS.

Hub ADX cluster (old vs new, Hub DB, 1.16M cost rows):

• compute.md commitment-coverage queries: join→lookup versions return identical row counts (3 / 100).
• on 1 == 1 rewrites (finops-hub-database-guide.md, ftk-database-query.md): old versions fail with General_BadRequest (confirming the invalid predicate), new toscalar() versions pass with real data (6 / 44 rows).
• fullouter key-coalesce fixes: same row counts old vs new, but old had 1 (cost-spike) and 3 (service-deep-dive) rows with an empty key column; new versions have 0 — exactly the fix intent.

Transform functions (compile-checked against real schemas):

• All 6 edited ingestion transforms (ActualCosts/AmortizedCosts/Costs_transform_v1_0/v1_2) extracted from the scripts and executed with | take 0 against the Ingestion DB (raw tables present) — all compile.
• Both edited HubSetup_v1_2 functions (CommitmentDiscountUsage_v1_2, Costs_v1_2) compile against the Hub DB.

Mirrors: the three edited queries in Compute/AHB.workbook verified byte-identical to the live-tested AHB/AHB.workbook copies.

Remaining untestable in this environment: row-level semantics of the SQL VM / public IP / app gateway ARG queries (test tenant has no IaaS — all validated as executing with 0 rows).

🤖 Generated with Claude Code