ADLS Gen 2

ChangeLog.md

@@ -6,10 +6,27 @@
 
 General:
 
+- Added Data Lake Storage Gen2 (ADLS Gen2 / DFS) REST API support, served on the same port as the Blob service. Enabled by default; disable it with `--enableHierarchicalNamespace false` command-line/VS Code extension switch or the `enableHierarchicalNamespace` programmatic option. Supports filesystem and path create/delete/list/get-properties/set-properties, path read, rename, append/flush, and access control (ACL) get/set operations against the existing Blob/Loki and SQL metadata stores.
 - Fixed `EMFILE: too many open files` when reading blobs spread across many extents. `FSExtentStore.readExtents` now opens each extent's read stream lazily, one at a time, instead of opening a file handle per extent up front. (issue #1967)
-- Fixed `--oauth` startup validation and reporting. Azurite now reports supported OAuth levels consistently across command-line entrypoints and no longer crashes when `--oauth` is specified without a value. (issue #2525)
-- Fixed SharedKey and SharedKeyLite authentication when both `date` and `x-ms-date` headers are present, matching Azure Storage signing behaviour for Blob, Queue, and Table services. (issue #1385)
-- Removed npm and its transitive dependencies from the Linux Docker runtime image while retaining Node.js and existing JavaScript entrypoints, reducing the container attack surface and addressing npm-related vulnerability reports. (issue #2758)
+- Fixed `--oauth` without a value crashing during startup and unsupported OAuth levels being silently ignored. Azurite now reports the supported OAuth levels across all command-line entrypoints while treating the VS Code extension's empty default as unconfigured. (issue #2525)
+- Updated `actions/checkout` from 4.4.0 to 7.0.1 and `actions/setup-node` from 4.4.0 to 7.0.0 so the CI workflows use the actions' Node.js 24 runtimes (workflow `node-version` settings are unchanged).
+- Fixed SharedKey/SharedKeyLite authentication failing when both `date` and `x-ms-date` request headers are present. Blob and Queue now sign an empty `Date` field and Table signs the `x-ms-date` value, matching Azure Storage. (issue #1385)
+- Updated lockfile-resolved (dev-only transitive via `ajv`) `fast-uri` from 3.1.5 to 3.1.7 to remediate URI authority injection and host confusion advisories.
+- Updated lockfile-resolved `mocha` from 12.0.0-rc.6 to 12.0.0 to pick up CLI argument parsing fixes (negative numbers and quoted strings).
+- Updated the lockfile-resolved `@types/node` dev dependency from 26.2.0 to 26.4.1 for current Node.js type declarations.
+- Updated lockfile-resolved `axios` from 1.19.0 to 1.20.0 for hardened runtime option handling.
+- Updated lockfile-resolved `lint-staged` from 17.3.0 to 17.5.0; moved `.lintstagedrc` flat-format coverage into `tests/packageScripts.test.ts`.
+- Bumped `morgan` from `^1.11.0` to `^1.12.0` (lockfile resolved to 1.12.0) to remediate CVE-2026-15603 (log forging via Unicode line separators in access log tokens).
+- Updated lockfile-resolved `@typescript-eslint/eslint-plugin` and `@typescript-eslint/parser` versions from 8.67.0 through 8.69.0 for bug fixes and rule updates.
+- Updated the lockfile-resolved `picomatch` version from 4.0.5 to 4.0.7 to fix glob scanning and terminal globstars in parenthesized patterns.
+- Updated lockfile-resolved `mysql2` from 3.23.4 to 3.24.2 to correct three-byte length-coded parameter encoding and improve SQL metadata-store performance; added SQL pool regression coverage for large bound parameters.
+- Updated lockfile-resolved `eslint` from 10.9.0 to 10.10.0 for linting fixes (including the trailing-decimal `no-loss-of-precision` false-positive fix); added package-scripts coverage to confirm Azurite still runs `eslint` on `"src/**/*.ts"`.
+- Removed npm and its transitive dependencies from the Linux Docker runtime image while retaining Node.js and all existing JavaScript entrypoints, addressing the npm-related vulnerability reports in issue #2758.
+- Updated lockfile-resolved `qs` from 6.15.3 to 6.16.0 for query-string parsing fixes.
+- Updated lockfile-resolved `mysql2` from 3.24.2 to 3.24.3 for an unrelated `PoolCluster` typings fix (not used by Azurite).
+- Updated the lockfile-resolved `@types/vscode` dev dependency from 1.134.0 to 1.136.0 for current VS Code API declarations; added `tests/common/vscStatusBarItem.test.ts` covering the extension status bar transitions against the updated typings.
+- Updated the lockfile-resolved `globals` dev dependency from 17.11.0 to 17.12.0 for a refreshed list of environment global identifiers used by ESLint config.
+- Updated lockfile-resolved transitive `js-yaml` versions (4.3.1 -> 4.3.2 and 5.2.3 -> 5.4.1) for upstream parser/security fixes.
 
 Blob:
 

README.md

@@ -230,11 +230,11 @@ To view Azurite logs in Visual Studio Code:
 docker run -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite
 ```
 
-`-p 10000:10000` will expose blob service's default listening port.
+`-p 10000:10000` will expose blob service's default listening port. The DFS (ADLS Gen2) service is also available on this port.
 `-p 10001:10001` will expose queue service's default listening port.
 `-p 10002:10002` will expose table service's default listening port.
 
-Or just run blob service:
+Or just run blob service (DFS is included automatically):
 
 ```bash
 docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --blobHost 0.0.0.0
@@ -386,7 +386,7 @@ You can customize the listening address per your requirements.
 
 ### Listening Port Configuration
 
-Optional. By default, Azurite V3 will listen to 10000 as blob service port, and 10001 as queue service port, and 10002 as the table service port.
+Optional. By default, Azurite V3 will listen to 10000 as blob service port (the DFS/ADLS Gen2 service is also served on this port), 10001 as queue service port, and 10002 as the table service port.
 You can customize the listening port per your requirements.
 
 > Warning: After using a customized port, you need to update connection string or configurations correspondingly in your Storage Tools or SDKs.

docs/designs/ADLS-gen2-parity.md

@@ -0,0 +1,190 @@
+# ADLS Gen2 Parity Implementation Plan
+
+## Context
+
+Azurite previously had a **thin DFS proxy layer** on a dedicated port (10004) that translated a small subset of ADLS Gen2 DFS REST API calls to Blob REST API calls via HTTP proxying (axios). This covered only filesystem (container) create/delete/HEAD and account listing. Full ADLS Gen2 parity requires native support for path (file/directory) operations, the append-then-flush write pattern, rename/move, ACLs, and list paths — none of which can be achieved by simple query-parameter rewriting.
+
+## Architectural Decision: Hybrid (Native DFS Handlers + Shared Port)
+
+Replace the HTTP proxy with a **native Express pipeline** mounted inside `BlobRequestListenerFactory` that directly accesses `IBlobMetadataStore` and `IExtentStore` — the same store instances used by the blob handlers. DFS and Blob share a single listener on port 10000; routing is done by URL prefix inside the existing server.
+
+```
+Port 10000
+  ├─ /devstoreaccount1/<container>?resource=filesystem  →  DFS Handlers  →  IBlobMetadataStore + IExtentStore
+  ├─ /devstoreaccount1/<container>/<path>               →  DFS Handlers  →  same stores
+  └─ everything else                                    →  Blob Handlers →  same stores
+```
+
+There is no separate DFS server or dedicated DFS port. `--dfsHost` / `--dfsPort` CLI flags and the `azurite.dfsHost` / `azurite.dfsPort` VS Code settings have been removed.
+
+**Why not keep proxying?** DFS operations like List Paths, Create Directory, Rename, ACLs, and append-then-flush have no single blob API equivalent. Proxying would require multi-call orchestration, lose atomicity, and add latency.
+
+**Why shared port instead of separate listener?** The DFS and Blob APIs share the same account/container/blob namespace. A separate listener would require passing live store references across server boundaries and duplicating TLS/auth/logging configuration. Mounting DFS routing inside the existing server is simpler and keeps all requests to a single endpoint — matching how Azure itself exposes both APIs on `*.blob.core.windows.net` / `*.dfs.core.windows.net` (separate hostnames but the same backing infrastructure).
+
+### Directory Model
+
+Directories stored as **zero-length BlockBlobs with `hdi_isfolder=true` metadata** — matching Azure's real internal behavior. No separate table needed.
+
+### ACL Storage
+
+New fields on `BlobModel`: `dfsAclOwner`, `dfsAclGroup`, `dfsAclPermissions`, `dfsAcl`. LokiJS is schemaless (just add fields); SQL needs ALTER TABLE.
+
+---
+
+## Phase 0: Foundation — Shared Store Access & HNS Flag
+
+**Goal:** Wire DFS server to share stores with blob server; enable HNS mode.
+
+| File | Change |
+|------|--------|
+| `src/blob/utils/constants.ts` | Set `EMULATOR_ACCOUNT_ISHIERARCHICALNAMESPACEENABLED = true` (or make configurable) |
+| `src/blob/BlobServer.ts` | Expose `metadataStore`, `extentStore`, and `accountDataStore` via public getters |
+| `src/blob/BlobRequestListenerFactory.ts` | Mount `DfsRequestListenerFactory` as a sub-router on DFS URL patterns |
+| `src/blob/DfsRequestListenerFactory.ts` | Rewrite: replace axios proxy with native Express pipeline + DFS routing |
+| `src/blob/IBlobEnvironment.ts`, `BlobEnvironment.ts`, `src/common/Environment.ts`, `VSCEnvironment.ts` | Add `--enableHierarchicalNamespace` option; remove `--dfsHost`/`--dfsPort` |
+
+**Deliverable:** DFS requests are served on the blob port; existing filesystem tests pass via direct store access. No separate DFS listener or port.
+
+---
+
+## Phase 1: Path CRUD + List Paths
+
+**Goal:** Create/delete/read files and directories, list paths — the core operations most ADLS Gen2 SDKs depend on.
+
+### New files to create
+
+| File | Purpose |
+|------|---------|
+| `src/blob/dfs/DfsContext.ts` | DFS request context (account, filesystem, path) — analogous to `BlobStorageContext` |
+| `src/blob/dfs/DfsOperation.ts` | Enum of DFS operations for dispatch |
+| `src/blob/dfs/DfsDispatchMiddleware.ts` | Routes requests by `resource` param, `action` param, method, and headers |
+| `src/blob/dfs/DfsErrorFactory.ts` | JSON error responses (`PathNotFound`, `DirectoryNotEmpty`, etc.) |
+| `src/blob/dfs/DfsSerializer.ts` | JSON response serialization (DFS uses JSON, not XML) |
+| `src/blob/dfs/handlers/FilesystemHandler.ts` | Filesystem ops → container store operations |
+| `src/blob/dfs/handlers/PathHandler.ts` | Path create/delete/read/getProperties + listPaths |
+
+### Operations implemented
+
+- **Create Path** (`PUT ?resource=file|directory`): Creates zero-length BlockBlob; directories get `hdi_isfolder=true` metadata; auto-creates intermediate directories
+- **Delete Path** (`DELETE`): Files → `deleteBlob()`; directories with `recursive=true` → delete all blobs with prefix; `recursive=false` → 409 if non-empty
+- **Get Path Properties** (`HEAD`): Returns `x-ms-resource-type: file|directory` header
+- **Read Path** (`GET`): Streams file content via `downloadBlob()` (follows `BlobHandler.download()` pattern)
+- **List Paths** (`GET ?resource=filesystem&directory=...&recursive=true|false`): JSON response with `paths` array; uses `listBlobs()` with prefix/delimiter; supports continuation via `x-ms-continuation`
+
+### Existing files modified
+
+| File | Change |
+|------|--------|
+| `src/blob/persistence/IBlobMetadataStore.ts` | Add `dfsResourceType`, ACL fields to `BlobModel` / `IBlobAdditionalProperties` |
+| `src/blob/persistence/LokiBlobMetadataStore.ts` | No schema changes needed (schemaless) |
+| `src/blob/persistence/SqlBlobMetadataStore.ts` | Add columns: `dfsResourceType`, `dfsAclOwner`, `dfsAclGroup`, `dfsAclPermissions`, `dfsAcl` |
+
+### Tests
+
+Extend `tests/blob/dfsProxy.test.ts`:
+- Create file / directory, verify as blob
+- Delete file / empty dir / non-empty dir with recursive
+- Get properties with `x-ms-resource-type`
+- Read file content
+- List paths recursive and non-recursive
+- Cross-API: create via DFS → read via Blob API and vice versa
+
+---
+
+## Phase 2: Append-Flush Write Pattern
+
+**Goal:** Implement the DFS file write model (create empty → append chunks → flush to commit).
+
+### Key insight
+
+DFS append-then-flush maps directly to existing **BlockBlob uncommitted blocks** infrastructure: each `action=append` becomes a `stageBlock()`, and `action=flush` becomes `commitBlockList()`. No new persistence methods needed.
+
+### Changes to `src/blob/dfs/handlers/PathHandler.ts`
+
+- **`updatePath_Append(position, body)`**: Write body to `IExtentStore` as extent chunk; record as uncommitted block via `metadataStore.stageBlock()`; validate `position` matches current append offset; return 202
+- **`updatePath_Flush(position, close)`**: Commit all staged blocks via `metadataStore.commitBlockList()`; update content length to `position`; return 200 with updated ETag
+
+### Tests
+
+- Create → append 3 chunks → flush → read back, verify content
+- Append with wrong position → 400
+- Large file (multi-MB) append
+
+---
+
+## Phase 3: Rename/Move Path
+
+**Goal:** Atomic rename for files and directories.
+
+### New persistence methods
+
+| Method | Description |
+|--------|-------------|
+| `IBlobMetadataStore.renameBlob(src, dest)` | Atomic rename of single blob (metadata-only, no extent copy) |
+| `IBlobMetadataStore.renameBlobsByPrefix(srcPrefix, destPrefix)` | Atomic rename of all blobs matching prefix (for directory rename) |
+
+### PathHandler addition
+
+- **`renamePath(x-ms-rename-source)`**: Parse source header → for files: `renameBlob()`; for directories: `renameBlobsByPrefix()`. Supports cross-filesystem rename and conditional headers.
+
+### Persistence implementations
+
+- **LokiJS**: Update document `containerName` and `name` properties
+- **SQL**: `UPDATE ... SET name = REPLACE(name, oldPrefix, newPrefix) WHERE name LIKE 'prefix%'` in transaction
+
+### Tests
+
+- Rename file within filesystem / across filesystems
+- Rename directory (verify children moved)
+- Rename non-existent → 404
+- Rename with conditional headers
+
+---
+
+## Phase 4: ACL Operations
+
+**Goal:** POSIX ACL get/set for emulator parity.
+
+### PathHandler additions
+
+- **`getAccessControl()`**: Read ACL fields from blob record → return as `x-ms-owner`, `x-ms-group`, `x-ms-permissions`, `x-ms-acl` headers. Defaults: `$superuser`/`$superuser`/`rwxr-x---`
+- **`setAccessControl(owner, group, permissions, acl)`**: Validate ACL format → update blob record
+- **`setAccessControlRecursive(mode, acl)`**: `mode` = set|modify|remove; iterate blobs under prefix; support continuation; return JSON with `directoriesSuccessful`, `filesSuccessful`, `failureCount`
+
+### Tests
+
+- Set/get ACL on file and directory
+- Recursive ACL set on directory tree
+- Default ACL values on new paths
+
+---
+
+## Phase 5: Polish & Remaining Operations
+
+- **Set Filesystem Properties** (`PATCH ?resource=filesystem`) → `setContainerMetadata()`
+- **`x-ms-properties` encoding/decoding** — new `src/blob/dfs/DfsPropertyEncoding.ts` utility (base64 key=value pairs)
+- **DFS JSON error format**: `{"error":{"code":"...","message":"..."}}`
+- **Lease support** on DFS paths (reuse blob lease infrastructure)
+- **SAS validation** on DFS endpoints (reuse existing authenticators)
+- **Content-MD5/CRC64 validation** on append
+
+---
+
+## Verification Plan
+
+1. **Unit tests**: Extend `tests/blob/dfsProxy.test.ts` per phase
+2. **Cross-API tests**: Verify DFS-created data is visible via Blob API and vice versa
+3. **SDK integration**: Test with `@azure/storage-file-datalake` Node.js SDK against the emulator
+4. **Manual smoke test**: Run Azurite, use Azure Storage Explorer with DFS endpoint
+5. **Existing blob tests**: Ensure `npm test` still passes (no regression)
+
+---
+
+## Critical Reference Files
+
+- `src/blob/handlers/ContainerHandler.ts` — pattern for handler ↔ store interaction
+- `src/blob/handlers/BlockBlobHandler.ts` — `stageBlock`/`commitBlockList` for append-flush reuse
+- `src/blob/handlers/BlobHandler.ts` — `download()` pattern for Read Path
+- `src/blob/persistence/IBlobMetadataStore.ts` — store interface to extend
+- `src/blob/generated/handlers/` — handler interface patterns
+- `src/blob/middlewares/blobStorageContext.middleware.ts` — context extraction pattern for DfsContext