fabric_resources: Agent 365 app-only docs + merge-key/schema corrections

fabric_resources/CompatibilityMatrix.md

@@ -46,7 +46,7 @@ Storage tier is inferred from each `-OutputPath*` value's form (see §4a). All f
 | DeviceCode          | ✅              | ✅          | ✅             | ✅                       |
 | AppRegistration     | ✅              | ✅          | ✅             | ✅                       |
 | ManagedIdentity     | ✅              | ✅          | ✅             | ✅                       |
-| Agent365 enrichment | Interactive / DeviceCode / AppRegistration only — **ManagedIdentity is rejected** (Graph requires a delegated context for the underlying endpoints). |
+| Agent365 enrichment | Supported under **all** auth modes. **Delegated** (Interactive / DeviceCode / Credential / Silent) requires the signed-in user to hold the **AI Administrator** or **Global Administrator** directory role. **App-only** (`AppRegistration` certificate/secret **or** `ManagedIdentity`) uses the application permissions `CopilotPackages.Read.All` + `Application.Read.All` (admin-consented) and performs no interactive sign-in, so it is valid on headless hosts (containers / ACA Jobs). App-only Agent 365 is **opt-in / pre-GA** — validate before relying on it in production; a missing app-role, unlicensed tenant, or absent program enrollment surfaces as a runtime **403**, not an up-front auth-mode rejection. |
 
 > **`-Deidentify` is auth- and tier-agnostic.** Optional output anonymization works under every auth mode and on all four destination tiers (it is applied on the host before any write/upload) and requires no additional Graph scope. See §4b for its effect on merge keys.
 
@@ -87,9 +87,11 @@ Three trailing columns are appended at merge time to **any appended file**:
 | `Latest_Append_Date` | `YYYY-MM-DD` | Latest run timestamp that touched the file. Same value on every row; updated each append. |
 | `In_Latest_Append` | `TRUE` / `FALSE` | Whether the row appeared in the latest run's audit window or membership snapshot. `FALSE` for retained-but-departed rows. |
 
-The CopilotInteraction rollup Fact CSV additionally carries two trailing identity columns (always present, not just at append time) so per-run integer surrogates remain stable across appends: **`Message_Id_Raw`** (raw audit `Messages[].Id` GUID) and **`ThreadId_Raw`** (raw `CopilotEventData.ThreadId` GUID). Merge keys: `Message_Id` for the rollup Fact CSV, `RecordId` for the non-rollup raw audit CSV, `PersonId_Normalized` for the EntraUsers CSV, `AgentId` for the Agent 365 CSV.
+The CopilotInteraction rollup Fact CSV additionally carries stable identity columns (always present, not just at append time) so per-run integer surrogates remain stable across appends: the two trailing raw keys **`Message_Id_Raw`** (raw audit `Messages[].Id` GUID) and **`ThreadId_Raw`** (raw `CopilotEventData.ThreadId` GUID), plus a stable normalized user-identity column — **`User_Id_Normalized`** on the AIO fact (additive; parity with the **`Audit_UserId_Normalized`** column the AIBV fact already carries). Merge keys: the rollup Fact CSV dedups on a **composite key** — the profile grain (every grain column, with per-run integer surrogates replaced by their stable forms: `UserKey` → `User_Id_Normalized` on AIO / `Audit_UserId_Normalized` on AIBV, and `ThreadId` → `ThreadId_Raw`) **plus `Message_Id_Raw`** — because one message can fan out to several rows (one per distinct grain, e.g. per accessed resource) that must each reconcile independently rather than collapse to one. The non-rollup raw audit CSV keys on `RecordId`, the EntraUsers CSV on `PersonId_Normalized`, and the Agent 365 CSV on `AgentId`.
 
-**Deidentification (`-Deidentify`) and merge keys.** Under `-Deidentify`, identity values are replaced with irreversible deterministic tokens before any write. The documented merge keys remain usable: `PersonId_Normalized` is tokenized but deterministic (so EntraUsers/Users joins and distinct counts are preserved across runs), while `Message_Id_Raw` / `ThreadId_Raw` are message/thread GUIDs that are **not** tokenized. A run's deidentify state is detected from the target's identity columns; appending a `-Deidentify` run into a non-deidentified target (or the reverse) is hard-rejected at pre-flight to prevent token/raw identity mixing. `-Deidentify` is persisted in the checkpoint and restored on `-Resume`.
+**Rollup Fact CSV one-time re-baseline (parallel to the Users schema-compatibility probe below).** When an existing rollup Fact CSV `-AppendFile` target predates these keys — i.e. it is missing `Message_Id_Raw` or any composite-key column (for example an older AIO file written before `User_Id_Normalized` existed) — PAX does **not** silently overwrite or mis-merge it. It leaves the old target untouched, writes the current run to a new timestamped file, and prints re-baseline guidance. The practical effect is a one-time re-baseline with no data lost. The 0-row overwrite refusal and union-shrink guards remain in force alongside it.
+
+**Deidentification (`-Deidentify`) and merge keys.** Under `-Deidentify`, identity values are replaced with irreversible deterministic tokens before any write. The documented merge keys remain usable: `PersonId_Normalized` and the fact composite key's user component (`User_Id_Normalized` / `Audit_UserId_Normalized`) are tokenized but deterministic (so EntraUsers/Users joins, distinct counts, and cross-run fact reconciliation are preserved across runs), while `Message_Id_Raw` / `ThreadId_Raw` are message/thread GUIDs that are **not** tokenized. A run's deidentify state is detected from the target's identity columns; appending a `-Deidentify` run into a non-deidentified target (or the reverse) is hard-rejected at pre-flight to prevent token/raw identity mixing. `-Deidentify` is persisted in the checkpoint and restored on `-Resume`.
 
 **Org / manager-hierarchy columns and Delta schema evolution.** AIO / AIBV rollups add org/manager-hierarchy columns to the Users output (`OrgLevel`, `Manager_UserKey`, `TopOfChain_UserKey`, `HierarchyPath`, `IsManager`, `DirectReports`, `TotalReports`, and `Level0`…`Level14` name/key pairs; the M365 dashboard has none). On Fabric, the Users append runs a schema-compatibility probe that **tolerates added columns but rejects missing ones**: appending a current (hierarchy-bearing) run into a Delta table created by an older PAX simply adds the new columns, but appending older output that lacks the hierarchy columns into a table that already has them fails the compatibility check. Keep a given Fabric Users table on a single PAX version line, or recreate it, to avoid mixed-schema appends. The hierarchy needs no new Graph scope (manager data comes from the existing `$expand=manager` under `User.Read.All`).
 
@@ -134,7 +136,7 @@ major exceeds 2. Operators should never edit checkpoints by hand.
 
 ## 8. Known incompatibilities
 
-- `-Auth ManagedIdentity` with `-IncludeAgent365Info` / `-OnlyAgent365Info` — rejected at startup.
+- `-Auth ManagedIdentity` with `-IncludeAgent365Info` / `-OnlyAgent365Info` **without** the Agent 365 application permissions granted — the app-only Agent 365 phase returns a runtime **403** until the identity holds `CopilotPackages.Read.All` + `Application.Read.All` (grant via `Grant-PAXPermissions.ps1 -IncludeAgent365`; opt-in / pre-GA). It is **not** rejected at startup.
 - Two simultaneous PAX runs against the same `OutputPath` — guarded by an exclusive
   checkpoint lock; operators should still set ACA `parallelism = 1`.
 - Remote (URL) `OutputPath` for checkpoint storage — checkpoints MUST be on the local

fabric_resources/Deploy/Deploy-PAXAcaJob.ps1

@@ -21,9 +21,11 @@
               GroupMember.Read.All (always); AuditLogsQuery-Exchange.Read.All,
               AuditLogsQuery-OneDrive.Read.All, AuditLogsQuery-SharePoint.Read.All
               (only when PAX is invoked with -IncludeM365Usage).
-              Note: Agent 365 scopes (CopilotPackages.Read.All, Application.Read.All)
-              and the legacy AuditLog.Read.All Entra-audit scope are NOT used by PAX
-              and are intentionally not granted.
+              Agent 365 scopes (CopilotPackages.Read.All, Application.Read.All) are
+              granted only when Grant-PAXPermissions.ps1 is run with -IncludeAgent365
+              (opt-in / pre-GA app-only Agent 365 enrichment); they are not granted by
+              default. The legacy AuditLog.Read.All Entra-audit scope is not used by PAX
+              and is never granted.
           * Sites.ReadWrite.All + Files.ReadWrite.All (SharePoint mode), OR
             Contributor / Storage Blob Data Contributor on the Fabric workspace (Fabric mode)
 

fabric_resources/Deploy/README.md

@@ -1,8 +1,8 @@
 # PAX Purview Audit Log Processor — ACA Job Deployment (Container path)
 
 > [!IMPORTANT]
-> **Microsoft Agent 365 enrichment is fully supported.**
-> The `-IncludeAgent365Info`, `-OnlyAgent365Info`, `-OutputPathAgent365Info`, and `-AppendAgent365Info` switches require interactive (delegated) sign-in by an AI Administrator or Global Administrator; ManagedIdentity is not supported for this stream.
+> **Microsoft Agent 365 enrichment is supported on this container path via app-only auth (opt-in / pre-GA).**
+> The `-IncludeAgent365Info`, `-OnlyAgent365Info`, `-OutputPathAgent365Info`, and `-AppendAgent365Info` switches run under `-Auth ManagedIdentity` (app-only) once the managed identity holds the application permissions `CopilotPackages.Read.All` + `Application.Read.All` (admin-consented) — no interactive sign-in is required. Grant them by running `../Prereqs/Grant-PAXPermissions.ps1` with `-IncludeAgent365` (not granted by default). This app-only path is **pre-GA** — validate it in your tenant before relying on it in production.
 
 > **Two ways to use PAX with Fabric.** This README covers the **container path** — a scheduled, unattended run hosted on Azure Container Apps Jobs using a managed identity. If you want to run PAX directly from a laptop, on-prem server, or Azure VM and still write to Fabric/OneLake, see [`../LocalRun/README.md`](../LocalRun/README.md) (no container build, no ACR, no ACA — just PowerShell + an Entra identity that has Fabric workspace access). The top-level [`../README.md`](../README.md) compares the two paths side-by-side.
 
@@ -149,6 +149,8 @@ Rules enforced at parameter validation: every supplied URL must resolve to the s
     -CronExpression '0 6 * * *'
 ```
 
+> **This example uses `-IncludeAgent365Info` under `-Auth ManagedIdentity` (app-only).** It requires the managed identity to hold `CopilotPackages.Read.All` + `Application.Read.All` — grant them with `../Prereqs/Grant-PAXPermissions.ps1 -IncludeAgent365`. App-only Agent 365 is opt-in / pre-GA; validate it in your tenant before relying on it in production. (Omit the Agent 365 switches, or the `-IncludeAgent365` grant, if you don't need this stream.)
+
 ### Provenance columns on appended files
 
 Every appended file (rollup Fact CSV under `-AppendFile`, raw audit CSV under non-rollup `-AppendFile`, EntraUsers CSV under `-AppendUserInfo`) gains three trailing columns at merge time: `Date_Added`, `Latest_Append_Date`, `In_Latest_Append`. Rows that departed from the current run's audit window are retained in the union with `In_Latest_Append = FALSE` so historical fact-table joins continue to resolve. The CopilotInteraction rollup Fact CSV additionally carries two stable identity columns (`Message_Id_Raw`, `ThreadId_Raw`) so per-run integer surrogates remain stable across appends. Under `-AppendUserInfo`, the raw Entra membership snapshot the audit phase writes is kept **pristine** — the union lands at the `-AppendUserInfo` target, the raw file keeps its natural timestamped name (or gets a `_raw` suffix on the rare path-and-leaf collision).
@@ -325,7 +327,7 @@ Or open the file share in Azure Storage Explorer / Azure portal. The failed cont
 
 ## Notes
 
-- **Agent 365 enrichment is NOT supported under `-Auth ManagedIdentity`.** PAX rejects `-IncludeAgent365Info` and `-OnlyAgent365Info` up-front when `-Auth ManagedIdentity` is in effect. The Microsoft Graph Agent Package Management API requires the AI Administrator or Global Administrator directory role, which can only be held by a signed-in user; a managed identity has no user principal and cannot satisfy the requirement, even with admin-consented application permissions. If you need Agent 365 enrichment, run PAX interactively (`-Auth WebLogin` or `DeviceCode`), or with `-Auth AppRegistration` (PAX will interactively top up a delegated context for the Agent 365 phase only). For the unattended ACA-Job pattern this README covers, simply omit those switches.
+- **Agent 365 enrichment under `-Auth ManagedIdentity` (app-only, opt-in / pre-GA).** PAX runs `-IncludeAgent365Info` / `-OnlyAgent365Info` app-only under `-Auth ManagedIdentity` (and `-Auth AppRegistration`) with no interactive sign-in, provided the identity holds the application permissions `CopilotPackages.Read.All` + `Application.Read.All` (admin-consented). Grant them with `../Prereqs/Grant-PAXPermissions.ps1 -IncludeAgent365` — they are not granted by default. A missing app-role, unlicensed tenant, or absent program enrollment surfaces as a runtime 403 (the rest of the run completes normally). This app-only path is **pre-GA** — validate it before relying on it in production. (Delegated Agent 365 instead relies on the signed-in user's AI Administrator / Global Administrator directory role, e.g. on the local-run path.)
 - **Fabric `-OutputPath` shapes the script accepts:**
   - `https://<tenant>.onelake.dfs.fabric.microsoft.com/<Workspace>/<Lakehouse>.Lakehouse` — main Delta tables go under the Lakehouse's default `Tables/` area.
   - `…/<Lakehouse>.Lakehouse/Tables` — explicit non-Schemas form.

fabric_resources/LocalRun/README.md

@@ -1,6 +1,6 @@
 # PAX + Fabric — Local / Direct run (Path A)
 
-Run PAX directly from a Windows laptop, on-prem server, or Azure VM and write outputs to a Fabric Lakehouse. **No container build, no ACR, no ACA, and no `Grant-PAXPermissions.ps1` are required.** This is the lower-setup path; pick it when you do not need a fully unattended Azure-hosted scheduled run, or when you need Microsoft Agent 365 enrichment (which is unsupported under managed identity — see the container path README).
+Run PAX directly from a Windows laptop, on-prem server, or Azure VM and write outputs to a Fabric Lakehouse. **No container build, no ACR, no ACA, and no `Grant-PAXPermissions.ps1` are required.** This is the lower-setup path; pick it when you do not need a fully unattended Azure-hosted scheduled run, or when you want the simplest route to **delegated** Microsoft Agent 365 enrichment. (Agent 365 also runs **app-only** under managed identity on the container path — opt-in / pre-GA; see the container path README.)
 
 For the side-by-side comparison of this path vs. the container path, see [`../README.md`](../README.md).
 
@@ -58,7 +58,9 @@ When you intend to pass `-IncludeM365Usage`:
 - `AuditLogsQuery-OneDrive.Read.All`
 - `AuditLogsQuery-SharePoint.Read.All`
 
-If you want Microsoft Agent 365 enrichment (`-IncludeAgent365Info` / `-OnlyAgent365Info`), the **signed-in user** also needs the **AI Administrator** or **Global Administrator** directory role. This requirement is a server-side check on the Agent 365 endpoint and cannot be satisfied by application permissions alone. (Use `-Auth WebLogin`, `DeviceCode`, `Credential`, `Silent`, or `AppRegistration`. Managed identity is rejected up-front for Agent 365.)
+> **`-UserInfoFile` can reduce the required scopes.** If you supply the user/organization directory from your own CSV via `-UserInfoFile` and every row includes a license value, PAX skips the live `/users` pull and the `/subscribedSkus` license lookup — so `User.Read.All` and `Organization.Read.All` may not be needed for that run. A single blank license value re-triggers the online license lookup (`User.Read.All` + `Organization.Read.All`).
+
+If you want Microsoft Agent 365 enrichment (`-IncludeAgent365Info` / `-OnlyAgent365Info`) under **delegated** auth (`-Auth WebLogin` / `DeviceCode` / `Credential` / `Silent`), the **signed-in user** needs the **AI Administrator** or **Global Administrator** directory role — a server-side check on the Agent 365 endpoint. Under **app-only** auth (`-Auth AppRegistration` certificate/secret, or `-Auth ManagedIdentity`) no directory role or interactive sign-in is required; instead grant the application permissions **`CopilotPackages.Read.All`** + **`Application.Read.All`** (admin-consented). App-only Agent 365 is **opt-in / pre-GA** — validate it in your tenant before relying on it in production.
 
 ---
 
@@ -171,7 +173,7 @@ If you want unattended runs but prefer not to manage a cert or secret, host PAX
        -Rollup
    ```
 
-This is functionally identical to the container path but skips the ACR + ACA layer. **Remember: Agent 365 enrichment is rejected up-front under `-Auth ManagedIdentity`.**
+This is functionally identical to the container path but skips the ACR + ACA layer. **Note: Agent 365 enrichment runs app-only under `-Auth ManagedIdentity` when the identity holds `CopilotPackages.Read.All` + `Application.Read.All` (opt-in / pre-GA — see the Microsoft Graph permissions section above).**
 
 ---
 
@@ -262,7 +264,7 @@ Every appended file gains three trailing columns at merge time:
 | `Latest_Append_Date` | `YYYY-MM-DD` | Latest run that touched the file (same on every row). |
 | `In_Latest_Append` | `TRUE` / `FALSE` | Whether the row appeared in this run's audit window / membership snapshot. `FALSE` for rows retained from prior runs that no longer surface today. |
 
-The CopilotInteraction rollup Fact CSV additionally carries two stable identity columns: **`Message_Id_Raw`** and **`ThreadId_Raw`**. These keep the per-run integer surrogates (`Message_Id`, `ThreadId`) stable across appends so multi-month threads remain a single thread in downstream models.
+The CopilotInteraction rollup Fact CSV additionally carries stable identity columns: the raw keys **`Message_Id_Raw`** and **`ThreadId_Raw`**, plus a normalized user column (**`User_Id_Normalized`** on the AIO fact; **`Audit_UserId_Normalized`** on AIBV). These keep the per-run integer surrogates (`Message_Id`, `ThreadId`, `UserKey`) stable across appends — the cross-run append merge dedups on the full grain plus `Message_Id_Raw`, so multi-month threads remain a single thread and fan-out rows (many per message) reconcile independently in downstream models.
 
 ### Pristine raw EntraUsers under `-AppendUserInfo`