Skip to content

Add Azure Blob Storage as a File Sync source #1027

Closed
Closed
Add Azure Blob Storage as a File Sync source#1027
Assignees
paullizer
Labels
enhancementNew feature or request

Description

@paullizer
Paul Lizer (paullizer)
opened on Jul 14, 2026

Summary

Add Azure Blob Storage as a first-class File Sync source alongside Azure Files. Administrators and authorized workspace users should be able to configure a storage account, container, and optional blob-name prefix, then ingest new and changed blobs through the existing File Sync pipeline.

User Value

Many organizations store source documents in Blob Storage rather than an Azure Files share. Supporting Blob Storage lets those repositories participate in the same scheduled ingestion, document processing, tagging, and workspace experiences without requiring users to copy content into Azure Files first.

Proposed Behavior

  • Add an Azure Blob Storage source type to the existing File Sync source registry and admin-controlled source-type visibility settings.
  • Support the File Sync workspace scopes that are already available to other eligible source types, using the same source ownership and workspace authorization checks.
  • Collect a storage account URL or account name, container name, and optional blob-name prefix. Normalize and validate these values server-side.
  • Reuse the existing workspace identity and Key Vault-backed File Sync credential models. Prefer managed identity and least-privilege Azure RBAC; support other credential modes only where they can be stored and resolved without exposing secrets to the browser or Cosmos DB records.
  • Provide a connection test that verifies authentication, container access, and prefix validity without downloading the full source.
  • Enumerate blobs beneath the configured prefix, treating blob names as virtual paths and ignoring directory-marker records where appropriate.
  • Use stable source item identifiers plus Blob properties such as ETag, last-modified time, and content length to detect additions and changes efficiently across runs.
  • Stream blob content into the existing File Sync ingestion path so file-size limits, supported-format checks, document processing, tags, source attribution, run metrics, and retry behavior remain consistent with other connectors.
  • Apply existing File Sync deletion and rename policies when previously synchronized blobs disappear or move.
  • Redact credentials and sensitive endpoint details from API responses, logs, run errors, and frontend state.
  • Document the required Azure role assignments, recommended Storage Blob Data Reader scope, supported credential modes, network/private-endpoint considerations, and container/prefix configuration.

Acceptance Criteria

  • Azure Blob Storage can be enabled or hidden through the existing admin-managed File Sync source-type configuration.
  • Authorized users can create, edit, test, enable, disable, run, and delete a Blob Storage source through the same File Sync workflows used by other source types.
  • Source creation and update validate the account endpoint, container, prefix, authentication mode, and referenced workspace identity on the server.
  • Personal, group, and public workspace authorization boundaries remain enforced; a user cannot test or run a source outside their authorized workspace scope.
  • Managed identity authentication works with least-privilege Blob data access. Any supported client secret or connection string is stored through the existing Key Vault-backed secret path and is never returned to the frontend.
  • Connection testing distinguishes invalid credentials, missing RBAC, unreachable endpoints, missing containers, and invalid prefixes without leaking secret values.
  • A sync run recursively discovers eligible blobs under the configured prefix and downloads them without loading large files entirely into application memory.
  • New and changed blobs enter the existing ingestion pipeline with correct filename, virtual path, content type, size, last-modified value, source attribution, and configured path-tag behavior.
  • A second unchanged run does not reprocess unchanged blobs, while ETag or relevant metadata changes cause the blob to be processed again.
  • Deleted and renamed blobs follow the connector-independent File Sync reconciliation and deletion policies without affecting documents from another source.
  • Source and item identifiers remain stable and collision-safe across containers, prefixes, workspaces, and storage accounts.
  • Failures are recorded in File Sync run/item status, are retryable where appropriate, and do not expose credentials or unauthorized endpoint details.
  • Automated tests cover configuration validation, source visibility, all supported authentication paths, workspace isolation, prefix filtering, virtual paths, incremental change detection, deletion reconciliation, large-file streaming, and sanitized errors using mocked Azure SDK clients where practical.
  • Functional or integration validation covers at least one real Blob container using managed identity or an equivalent CI-safe test environment.
  • File Sync documentation explains setup, RBAC, networking, configuration, expected sync semantics, troubleshooting, and known limitations.

Notes

  • Initial scope is inbound File Sync from Blob Storage into SimpleChat. Outbound synchronization, blob uploads, and mutation of source blobs are not part of this issue.
  • Azure Data Lake Storage Gen2-specific ACL management and filesystem semantics are not required unless they work transparently through the same Blob API contract; they can be scoped separately if needed.
  • Reuse the established connector, workspace identity, Key Vault, sanitization, and run/item reconciliation abstractions rather than adding a separate synchronization subsystem.
  • Relevant implementation areas include functions_file_sync.py, route_backend_file_sync.py, workspace File Sync UI modules, source-type visibility settings, workspace identities, and Key Vault secret handling.

Activity

  1. Paul Lizer (paullizer)
    added
    enhancementNew feature or request
    on Jul 14, 2026
  2. github-project-automation
    added this to Simple Chat Roadmapon Jul 14, 2026
  3. github-project-automation
    moved this to Pending Evaluation in Simple Chat Roadmapon Jul 14, 2026
  4. Paul Lizer (paullizer)
    moved this from Pending Evaluation to Scheduled in Simple Chat Roadmapon Jul 14, 2026
  5. Paul Lizer (paullizer)
    self-assigned this
    on Jul 14, 2026
  6. Paul Lizer (paullizer)
    moved this from Scheduled to In progress in Simple Chat Roadmapon Jul 28, 2026
  7. Paul Lizer (paullizer)
    moved this from In progress to In review in Simple Chat Roadmapon Jul 28, 2026

  8. paullizer commented on Jul 28, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Jul 28, 2026
    CollaboratorAuthor

    Implementation is complete and submitted in PR #1088.

    What changed:

    • Added Azure Blob Storage as an admin-controlled File Sync source for personal, group, and public workspaces.
    • Added account/container/prefix validation, virtual-path browsing, connection testing, ETag-based change detection, and chunked downloads through the existing ingestion pipeline.
    • Reused existing workspace identities and storage authentication modes; managed identity is recommended, and saved client-secret/connection-string credentials require Azure Key Vault references.
    • Added admin/workspace UI, source badges, feature guidance, documentation, focused functional coverage, and version 0.250.067.

    Validation:

    • Azure Blob connector suite: 9 passed.
    • Azure Files and OneDrive regression suites passed.
    • Python compilation and JavaScript syntax checks passed.
    • Changed-line XSS and broken-access-control guardrails passed.
    • Whitespace and editor diagnostics passed.
    • Playwright tests were updated, but the available local worker remained on 0.250.066 and the saved authentication state redirected to Microsoft sign-in, so authenticated branch-runtime Playwright execution remains for PR/CI validation.

    Commit: a09ec55
    PR: #1088

  9. paullizer commented on Jul 28, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Jul 28, 2026
    CollaboratorAuthor

    Follow-up: the required v0.250.067 release-note entry was added in commit c0d7c20 after the initial Release Notes Check identified the omission. The updated PR head is pushed, the replacement check run is queued with no current failures, and the branch remains clean.

  10. paullizer commented on Jul 28, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Jul 28, 2026
    CollaboratorAuthor

    Security review follow-up pushed in commits 883f31a and 3feacec on PR #1088:

    • Restricted Blob URLs and connection strings to canonical HTTPS Azure Blob service endpoints for supported public-cloud suffixes.
    • Rejected arbitrary/internal hosts, IP literals, development storage, HTTP, userinfo, explicit ports, query strings, fragments, and unsupported endpoint suffixes.
    • Revalidated stored endpoint data immediately before Blob SDK client construction.
    • Replaced raw File Sync route, run-history, activity, and item exception details with fixed client-safe messages; detailed sanitized diagnostics remain in server logs only.
    • Bumped the application to v0.250.068 and added fix documentation and release notes.

    Validation: 13 Blob connector/security tests, Azure Files and OneDrive regressions, all 12 route-policy tests, Python compilation, diagnostics, XSS and access-control guardrails, and whitespace checks passed. The GHAS review thread has the detailed remediation response; replacement CodeQL analysis is running.

  11. paullizer commented on Jul 28, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Jul 28, 2026
    CollaboratorAuthor

    Container SAS follow-up pushed in commit 4026837 on PR #1088 (v0.250.069).

    What changed:

    • Accepts storage connection strings, full container SAS URLs, and standalone SAS tokens.
    • Pasted SAS URLs derive the canonical Blob service URL, selected container, and a default source name; the credential is promoted to Key Vault-backed secret storage and is not persisted in source connection metadata.
    • Container SAS requires Read and List, HTTPS-only protocol, valid signed version/signature, matching account/container, and a valid time window.
    • Extra permissions remain usable but produce named least-privilege warnings; account SAS and storage account keys are identified as broader than a single-container source needs.
    • Blob/object SAS is rejected because it cannot list a container.
    • Source rows and successful connection tests show only non-secret scope, named permissions, exact expiry, hours/days remaining, stored-policy status, IP restrictions, and warnings.
    • New credentials are validated before Key Vault writes, reusable identities ignore hidden local credentials, and identity-backed metadata refreshes after rotation.

    Validation: 19 Blob credential/security tests, Azure Files and OneDrive regressions, all 12 route-policy tests, XSS/access-control guardrails, compilation, diagnostics, and whitespace checks passed.

    The short-lived credential shared during testing was not copied into terminal commands, tests, source, logs, or issue comments. It should be rotated/revoked because it appeared in chat. A token containing Read without List must be regenerated with both Read and List.

  12. paullizer commented on Jul 28, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Jul 28, 2026
    CollaboratorAuthor

    Additional diagnostics follow-up pushed in e3e1e44:

    • Failed Azure Blob connection tests now log only non-secret exception type, Azure error code, HTTP status, request ID, credential scope, and permission letters.
    • SAS tokens, signed URLs, and raw SDK messages are excluded.
    • Users receive reviewed guidance for missing Read/List permissions, invalid signature/start/expiry, SAS IP/network restrictions, or missing/inaccessible containers; unknown SDK failures remain generic.
    • Blob credential/security suite is now 20 passed.

    For the short-lived SAS shared during testing, the required permission set is sp=rl. It must still be rotated/revoked because it appeared in chat.

  13. paullizer commented on Jul 28, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Jul 28, 2026
    CollaboratorAuthor

    Root-cause follow-up pushed in commit 10bbac4 (v0.250.070):

    1. Saving was blocked by a Blob-specific Key Vault assertion. That assertion is removed. Blob credentials now use Key Vault when enabled and existing File Sync source/identity credential persistence when disabled; browser responses remain redacted.
    2. The SAS connection test called Get Container Properties before List/Read. That probe is not required for File Sync and can return Azure AuthorizationFailure for a container SAS. The test now performs only container listing and reads one available blob.
    3. Azure enum error codes such as StorageErrorCode.AUTHORIZATION_FAILURE are normalized, and safe logs use only error code/status/request ID/auth kind/scope/permission letters.

    Validation: 21 Blob credential/security tests passed, including a fake container client that fails if Get Container Properties is called; Azure Files/OneDrive regressions, all 12 route-policy tests, security guardrails, compilation, diagnostics, and whitespace checks passed.

    Restart the Flask worker on commit 10bbac4 before retesting. The posted SAS should be rotated because it appeared in chat.

  14. paullizer commented on Aug 19, 2026

    @paullizer
    Paul Lizer (paullizer)
    on Aug 19, 2026
    CollaboratorAuthor

    Shipped in v0.250.070 via #1088.

    Release note entries:

    • v0.250.070: Azure Blob Container SAS Support and Credential Guidance
    • v0.250.068: Azure Blob File Sync Endpoint and Error Hardening
    • v0.250.067: Azure Blob Storage File Sync

    Full notes: https://microsoft.github.io/simplechat/explanation/release_notes/

    Closing as completed. This issue stayed open after the fix merged because pull requests target Development while the repository default branch is main, so GitHub never applied the Fixes # keyword automatically.

  15. Paul Lizer (paullizer)
    closed this as completedon Aug 19, 2026
  16. github-project-automation
    moved this from In review to Released in Simple Chat Roadmapon Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions