purview-v1.11.15

.gitignore

@@ -32,6 +32,7 @@ AGENTS.md
 .github/agents/
 .github/instructions/
 .github/prompts/
+.github/dev-trio/
 .github/copilot-instructions.md
 .dev-trio/
 

README.md

@@ -14,7 +14,7 @@
 
 <h3 align="center">PAX PowerShell Script&nbsp;&nbsp;|&nbsp;&nbsp;PAX Cookbook&nbsp;&nbsp;|&nbsp;&nbsp;PAX Cookbook Mini-Kitchen</h3>
 
-**⬇️ Download the script:** [`PAX_Purview_Audit_Log_Processor_v1.11.14.ps1`](https://github.com/microsoft/PAX/releases/download/purview-v1.11.14/PAX_Purview_Audit_Log_Processor_v1.11.14.ps1) &nbsp;|&nbsp; Release Date: July 6, 2026
+**⬇️ Download the script:** [`PAX_Purview_Audit_Log_Processor_v1.11.15.ps1`](https://github.com/microsoft/PAX/releases/download/purview-v1.11.15/PAX_Purview_Audit_Log_Processor_v1.11.15.ps1) &nbsp;|&nbsp; Release Date: July 31, 2026
 
 **📖 Script Resources:** [Latest Documentation](https://github.com/microsoft/PAX/blob/release/release_documentation/Purview_Audit_Log_Processor/PAX_Purview_Audit_Log_Processor_Documentation_v1.11.x.md) | [Latest Release Notes](https://github.com/microsoft/PAX/blob/release/release_notes/Purview_Audit_Log_Processor/PAX_Purview_Audit_Log_Processor_Release_Note_v1.11.x.md)
 

fabric_resources/CompatibilityMatrix.md

@@ -87,13 +87,13 @@ Three trailing columns are appended at merge time to **any appended file**:
 | `Latest_Append_Date` | `YYYY-MM-DD` | Latest run timestamp that touched the file. Same value on every row; updated each append. |
 | `In_Latest_Append` | `TRUE` / `FALSE` | Whether the row appeared in the latest run's audit window or membership snapshot. `FALSE` for retained-but-departed rows. |
 
-The CopilotInteraction rollup Fact CSV additionally carries stable identity columns (always present, not just at append time) so per-run integer surrogates remain stable across appends: the two trailing raw keys **`Message_Id_Raw`** (raw audit `Messages[].Id` GUID) and **`ThreadId_Raw`** (raw `CopilotEventData.ThreadId` GUID), plus a stable normalized user-identity column — **`User_Id_Normalized`** on the AIO fact (additive; parity with the **`Audit_UserId_Normalized`** column the AIBV fact already carries). Merge keys: the rollup Fact CSV dedups on a **composite key** — the profile grain (every grain column, with per-run integer surrogates replaced by their stable forms: `UserKey` → `User_Id_Normalized` on AIO / `Audit_UserId_Normalized` on AIBV, and `ThreadId` → `ThreadId_Raw`) **plus `Message_Id_Raw`** — because one message can fan out to several rows (one per distinct grain, e.g. per accessed resource) that must each reconcile independently rather than collapse to one. The non-rollup raw audit CSV keys on `RecordId`, the EntraUsers CSV on `PersonId_Normalized`, and the Agent 365 CSV on `AgentId`.
+The CopilotInteraction rollup Fact CSV additionally carries stable identity columns (always present, not just at append time) so per-run integer surrogates remain stable across appends: the two trailing raw keys **`Message_Id_Raw`** (raw audit `Messages[].Id` GUID) and **`ThreadId_Raw`** (raw `CopilotEventData.ThreadId` GUID), plus a stable normalized user-identity column — **`User_Id_Normalized`** on the AIO fact (additive; parity with the **`Audit_UserId_Normalized`** column the ValueLens fact already carries). Merge keys: the rollup Fact CSV dedups on a **composite key** — the profile grain (every grain column, with per-run integer surrogates replaced by their stable forms: `UserKey` → `User_Id_Normalized` on AIO / `Audit_UserId_Normalized` on ValueLens, and `ThreadId` → `ThreadId_Raw`) **plus `Message_Id_Raw`** — because one message can fan out to several rows (one per distinct grain, e.g. per accessed resource) that must each reconcile independently rather than collapse to one. The non-rollup raw audit CSV keys on `RecordId`, the EntraUsers CSV on `PersonId_Normalized`, and the Agent 365 CSV on `AgentId`.
 
 **Rollup Fact CSV one-time re-baseline (parallel to the Users schema-compatibility probe below).** When an existing rollup Fact CSV `-AppendFile` target predates these keys — i.e. it is missing `Message_Id_Raw` or any composite-key column (for example an older AIO file written before `User_Id_Normalized` existed) — PAX does **not** silently overwrite or mis-merge it. It leaves the old target untouched, writes the current run to a new timestamped file, and prints re-baseline guidance. The practical effect is a one-time re-baseline with no data lost. The 0-row overwrite refusal and union-shrink guards remain in force alongside it.
 
 **Deidentification (`-Deidentify`) and merge keys.** Under `-Deidentify`, identity values are replaced with irreversible deterministic tokens before any write. The documented merge keys remain usable: `PersonId_Normalized` and the fact composite key's user component (`User_Id_Normalized` / `Audit_UserId_Normalized`) are tokenized but deterministic (so EntraUsers/Users joins, distinct counts, and cross-run fact reconciliation are preserved across runs), while `Message_Id_Raw` / `ThreadId_Raw` are message/thread GUIDs that are **not** tokenized. A run's deidentify state is detected from the target's identity columns; appending a `-Deidentify` run into a non-deidentified target (or the reverse) is hard-rejected at pre-flight to prevent token/raw identity mixing. `-Deidentify` is persisted in the checkpoint and restored on `-Resume`.
 
-**Org / manager-hierarchy columns and Delta schema evolution.** AIO / AIBV rollups add org/manager-hierarchy columns to the Users output (`OrgLevel`, `Manager_UserKey`, `TopOfChain_UserKey`, `HierarchyPath`, `IsManager`, `DirectReports`, `TotalReports`, and `Level0`…`Level14` name/key pairs; the M365 dashboard has none). On Fabric, the Users append runs a schema-compatibility probe that **tolerates added columns but rejects missing ones**: appending a current (hierarchy-bearing) run into a Delta table created by an older PAX simply adds the new columns, but appending older output that lacks the hierarchy columns into a table that already has them fails the compatibility check. Keep a given Fabric Users table on a single PAX version line, or recreate it, to avoid mixed-schema appends. The hierarchy needs no new Graph scope (manager data comes from the existing `$expand=manager` under `User.Read.All`).
+**Org / manager-hierarchy columns and Delta schema evolution.** AIO / ValueLens rollups add org/manager-hierarchy columns to the Users output (`OrgLevel`, `Manager_UserKey`, `TopOfChain_UserKey`, `HierarchyPath`, `IsManager`, `DirectReports`, `TotalReports`, and `Level0`…`Level14` name/key pairs; the M365 dashboard has none). On Fabric, the Users append runs a schema-compatibility probe that **tolerates added columns but rejects missing ones**: appending a current (hierarchy-bearing) run into a Delta table created by an older PAX simply adds the new columns, but appending older output that lacks the hierarchy columns into a table that already has them fails the compatibility check. Keep a given Fabric Users table on a single PAX version line, or recreate it, to avoid mixed-schema appends. The hierarchy needs no new Graph scope (manager data comes from the existing `$expand=manager` under `User.Read.All`).
 
 **M365 rollup append anchoring (`-IncludeM365Usage` + `-Rollup` / `-RollupPlusRaw`).** The embedded M365 Bundle Explosion Processor produces a 4-file bundle in the destination folder/Lakehouse: `<stem>_Rollup.csv`, `<stem>_UserStats.csv`, `<stem>_SessionCohort.csv`, `<stem>_SessionStats.csv`. Only `_Rollup.csv` is the merge anchor; the merge is PowerShell-driven (`Merge-M365RollupCsv` unions the Rollup, then `--rebuild-sidecars-from-rollup` regenerates the sidecars). UserStats and SessionCohort are RECOMPUTED over the merged Rollup on every run; SessionStats is union-merged in place via `Merge-M365SessionStatsCsv` (additive counter semantics keyed on `UserId`, `CreationDate`, `AppHost`). All three sidecars are anchored off the `-AppendFile` leaf stem and overwrite their derived destination URLs in-place. `-AppendFile` MUST point to a `_Rollup.csv` leaf; sidecar leaves (`_UserStats.csv`, `_SessionCohort.csv`, `_SessionStats.csv`) and CopilotInteraction (`_Interactions.csv`) / event-level (`_Exploded.csv`) leaves are rejected at pre-flight. Renamed AppendFile is fine as long as the leaf still ends in `_Rollup.csv` (e.g. `MyM365Rollup_Rollup.csv` produces sidecars `MyM365Rollup_UserStats.csv`, `MyM365Rollup_SessionCohort.csv`, `MyM365Rollup_SessionStats.csv`); the validator also accepts a timestamped `_Rollup_<YYYYMMDD_HHMMSS>.csv` anchor. Prior sidecars with different leaf names are not auto-deleted. On Fabric `Tables/<schema>`, all 4 leaves become 4 stable Delta tables under the same schema (table name = leaf stem), refreshed via `Convert-CsvToDelta -Mode overwrite` on every run.
 
@@ -123,7 +123,14 @@ major exceeds 2. Operators should never edit checkpoints by hand.
 | `parallelism`    | **1**                  | Two replicas against the same OutputPath would corrupt the checkpoint. The script holds an exclusive lock but operators should not rely on it as a primary defence. |
 | `PAX_NONINTERACTIVE` | **1**             | The image already sets this. Removing it re-enables Read-Host prompts that will silently hang on stdin in ACA. |
 | Image user       | non-root (`pax`, UID 10001) | Enforced by the Dockerfile. |
-| `OutputPath`     | Local path **or** URL (tier inferred from the path form) | Customer-visible output can be Local / SharePoint / OneLake. Checkpoint + `_PARTIAL` files always live on local container scratch internally and are mirrored to the remote tier by `Mirror-ResumeArtifactsToOneLake` so resume can re-open them. |
+| `OutputPath`     | Local path **or** URL (tier inferred from the path form) | Customer-visible output can be Local / SharePoint / OneLake. Working checkpoint and `_PARTIAL` files are created on local container scratch; Fabric runs automatically maintain a durable mirror at `Files/.pax_resume/<run timestamp>/` so a fresh host can restore and resume. |
+
+<details>
+<summary><strong>v1.11.15 hosted resume validation</strong></summary>
+
+PAX v1.11.15 corrects exact-byte transmission for checkpoint and `_PARTIAL` artifacts, and unexpected fatal top-level errors return exit code `1`. Before broad production use, run an explicit dated hosted validation and confirm both the process exit result and resume restoration from the Fabric durable mirror.
+
+</details>
 
 ## 7. Environment variables PAX honours
 
@@ -139,8 +146,7 @@ major exceeds 2. Operators should never edit checkpoints by hand.
 - `-Auth ManagedIdentity` with `-IncludeAgent365Info` / `-OnlyAgent365Info` **without** the Agent 365 application permissions granted — the app-only Agent 365 phase returns a runtime **403** until the identity holds `CopilotPackages.Read.All` + `Application.Read.All` (grant via `Grant-PAXPermissions.ps1 -IncludeAgent365`; opt-in / pre-GA). It is **not** rejected at startup.
 - Two simultaneous PAX runs against the same `OutputPath` — guarded by an exclusive
   checkpoint lock; operators should still set ACA `parallelism = 1`.
-- Remote (URL) `OutputPath` for checkpoint storage — checkpoints MUST be on the local
-  filesystem so resume can re-open them safely.
+- Fabric-hosted resume uses local working checkpoint and `_PARTIAL` files plus an automatic durable mirror at `Files/.pax_resume/<run timestamp>/`; do not manually relocate or edit these artifacts.
 - UNC paths (`\\server\share\…`) on any destination switch (`-OutputPath`, `-OutputPathUserInfo`, `-OutputPathAgent365Info`, `-OutputPathLog`, `-AppendFile`, `-AppendUserInfo`, `-AppendAgent365Info`) — rejected at parameter validation.
 - Mixed-tier destinations in a single run — every supplied `-OutputPath*` / `-Append*` (URL form) must resolve to the same tier (Local / SharePoint / Fabric).
 - `-OutputPathLog` resolving to a Fabric `Tables/*` URL — logs are not tabular; only Fabric `Files/…` is accepted.

fabric_resources/Deploy/README.md

@@ -60,9 +60,9 @@ The container image itself is **auth-agnostic** — it just bakes in PowerShell
 
 ## Deploy
 
-> **Dashboard selection.** In `-ScriptArgs`, `-Rollup` targets the **AI-in-One (AIO)** dashboard by default. Add `'-Dashboard','AIBV'` for the **AI Business Value** dashboard, or `'-Dashboard','M365'` (equivalently `'-IncludeM365Usage'`) for **M365 Usage Analytics**. AIO and AIBV are produced from the same CopilotInteraction + Entra/MAC licensing data — no other args change.
+> **Dashboard selection.** In `-ScriptArgs`, `-Rollup` targets the **AI-in-One (AIO)** dashboard by default. Add `'-Dashboard','ValueLens'` for the **ValueLens** dashboard, or `'-Dashboard','M365'` (equivalently `'-IncludeM365Usage'`) for **M365 Usage Analytics**. AIO and ValueLens use the same CopilotInteraction + Entra/MAC licensing data, output schema, and append behavior; ValueLens is a name-only change, and existing checkpoints resume without customer intervention. <!-- Add `'-Dashboard','AISID'` for the **AI Solutions Intelligence Dashboard (AISID)** — the full Purview + Entra + Microsoft Defender pipeline; its 12 fixed-name files land in a dedicated `'-OutputPathDefenderUsage','<folder>'` (or `'-AppendDefenderUsage','<folder>'`) destination and are uploaded there exactly once, never to the Purview output. Three Microsoft Defender for Cloud Apps files are compatibility tables in this release: first runs write exact headers, while append runs preserve valid prior files byte-for-byte. -->
 
-> **Anonymization & hierarchy.** Add `'-Deidentify'` to `-ScriptArgs` to write anonymized output to the destination — every identity is replaced with an irreversible token on the host before upload (off by default; works under managed identity and app registration alike; no extra Graph permission). AIO / AIBV rollups automatically include org/manager-hierarchy columns in the Users output; `'-FillerLabel','Fixed','-FillerLabelText','<text>'` only controls how empty deeper org levels are labelled (the M365 dashboard has no hierarchy).
+> **Anonymization & hierarchy.** Add `'-Deidentify'` to `-ScriptArgs` to write anonymized output to the destination — every identity is replaced with an irreversible token on the host before upload (off by default; works under managed identity and app registration alike; no extra Graph permission). AIO / ValueLens rollups automatically include org/manager-hierarchy columns in the Users output; `'-FillerLabel','Fixed','-FillerLabelText','<text>'` only controls how empty deeper org levels are labelled (the M365 dashboard has no hierarchy).
 
 ### SharePoint destination, daily 06:00 UTC
 
@@ -118,6 +118,9 @@ The two examples above route every output to the same location (Purview audit, E
 | Purview audit (raw / rollup / event-level) | `-OutputPath` | `-AppendFile` |
 | EntraUsers / MAC licensing | `-OutputPathUserInfo` | `-AppendUserInfo` (auto-enables `-IncludeUserInfo`) |
 | Agent 365 catalog | `-OutputPathAgent365Info` | `-AppendAgent365Info` (auto-enables `-IncludeAgent365Info`) |
+<!--
+| AI Solutions Intelligence Dashboard (AISID) output set (12 fixed-name files) | `-OutputPathDefenderUsage` | `-AppendDefenderUsage` (folder-only; `-Dashboard AISID` runs) |
+-->
 | Run log | `-OutputPathLog` | _(n/a)_ |
 
 Rules enforced at parameter validation: every supplied URL must resolve to the same tier (mixed Local/SP/Fabric is rejected); UNC paths are rejected on every destination switch; exactly one of each stream's pair must be bound.
@@ -155,7 +158,7 @@ Rules enforced at parameter validation: every supplied URL must resolve to the s
 
 Every appended file (rollup Fact CSV under `-AppendFile`, raw audit CSV under non-rollup `-AppendFile`, EntraUsers CSV under `-AppendUserInfo`) gains three trailing columns at merge time: `Date_Added`, `Latest_Append_Date`, `In_Latest_Append`. Rows that departed from the current run's audit window are retained in the union with `In_Latest_Append = FALSE` so historical fact-table joins continue to resolve. The CopilotInteraction rollup Fact CSV additionally carries two stable identity columns (`Message_Id_Raw`, `ThreadId_Raw`) so per-run integer surrogates remain stable across appends. Under `-AppendUserInfo`, the raw Entra membership snapshot the audit phase writes is kept **pristine** — the union lands at the `-AppendUserInfo` target, the raw file keeps its natural timestamped name (or gets a `_raw` suffix on the rare path-and-leaf collision).
 
-**Under `-Deidentify`:** `PersonId_Normalized` is tokenized but deterministic (still a stable join key), while `Message_Id_Raw` / `ThreadId_Raw` keep their real GUID values; appending a deidentified run into a non-deidentified target (or vice-versa) is hard-rejected. **Org / manager-hierarchy columns** are added to the rollup Users output on AIO / AIBV runs — the Fabric Users append adds new columns but rejects appends that are *missing* existing columns, so keep a given Users Delta table on one PAX version line (or recreate it) to avoid mixed-schema appends.
+**Under `-Deidentify`:** `PersonId_Normalized` is tokenized but deterministic (still a stable join key), while `Message_Id_Raw` / `ThreadId_Raw` keep their real GUID values; appending a deidentified run into a non-deidentified target (or vice-versa) is hard-rejected. **Org / manager-hierarchy columns** are added to the rollup Users output on AIO / ValueLens runs — the Fabric Users append adds new columns but rejects appends that are *missing* existing columns, so keep a given Users Delta table on one PAX version line (or recreate it) to avoid mixed-schema appends.
 
 ## Alternative: app registration instead of managed identity
 
@@ -327,6 +330,9 @@ Or open the file share in Azure Storage Explorer / Azure portal. The failed cont
 
 ## Notes
 
+- **Replica safety:** Keep ACA job `parallelism` at `1`, and do not run concurrent replicas against the same output target.
+- **Validation sequence:** Run an explicit one-day UTC window first, for example `-StartDate '2026-07-01' -EndDate '2026-07-02'`; then run an explicit historical range such as `-StartDate '2026-06-01' -EndDate '2026-07-01'`; then append the next explicit range using the matching `-Append*` switches. Keep the PAX version and `-Deidentify` state consistent across the seed and append runs.
+- **Hosted resume and exit behavior:** Fabric mirrors durable resume artifacts to `Files/.pax_resume/<run timestamp>/`. PAX v1.11.15 corrects exact-byte transmission for checkpoint and `_PARTIAL` artifacts, and unexpected fatal top-level errors return exit code `1`. Observe the hosted process exit and resume behavior before broad production use.
 - **Agent 365 enrichment under `-Auth ManagedIdentity` (app-only, opt-in / pre-GA).** PAX runs `-IncludeAgent365Info` / `-OnlyAgent365Info` app-only under `-Auth ManagedIdentity` (and `-Auth AppRegistration`) with no interactive sign-in, provided the identity holds the application permissions `CopilotPackages.Read.All` + `Application.Read.All` (admin-consented). Grant them with `../Prereqs/Grant-PAXPermissions.ps1 -IncludeAgent365` — they are not granted by default. A missing app-role, unlicensed tenant, or absent program enrollment surfaces as a runtime 403 (the rest of the run completes normally). This app-only path is **pre-GA** — validate it before relying on it in production. (Delegated Agent 365 instead relies on the signed-in user's AI Administrator / Global Administrator directory role, e.g. on the local-run path.)
 - **Fabric `-OutputPath` shapes the script accepts:**
   - `https://<tenant>.onelake.dfs.fabric.microsoft.com/<Workspace>/<Lakehouse>.Lakehouse` — main Delta tables go under the Lakehouse's default `Tables/` area.