diff --git a/CLAUDE.md b/CLAUDE.md index 653502d..5284dc8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,16 +6,16 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co This is a GraphQL API server for OSWatcher that uses Neo4j as the database and provides filesystem analysis, diff computation, and search capabilities. The server includes: - GraphQL API with WebSocket subscriptions -- REST API for blob downloads (authorization check disabled by default -- see below) +- REST API for blob downloads - S3/MinIO integration for blob storage - Windows registry value filtering for sensitive data (disabled by default -- see below) -**Open-source release note (2026-08-11):** blob download authorization and -sensitive registry value redaction are commented out by default (not -deleted) in `src/rest-routes.ts` and `src/index.ts`, so a self-hosted -instance has full functionality with no configuration required. See -`docs/reference/access-restrictions.md` for what each restriction does and -how to re-enable it. +**Open-source release note:** sensitive registry value redaction is commented +out by default (not deleted) in `src/index.ts`, so a self-hosted instance has +full functionality with no configuration required. See +`docs/reference/access-restrictions.md` for what it does and how to re-enable +it. Blob download authorization (branch-based blocking of Windows binaries) was +removed entirely on 2026-09-08; it is in git history. ## Development Commands @@ -81,17 +81,12 @@ Custom resolvers for operations not auto-generated by Neo4j GraphQL: - `search` - Searches filesystem paths across commits - `traversePath` - Navigates filesystem tree structure -#### Blob Storage & Authorization +#### Blob Storage **REST Routes (`src/rest-routes.ts`)** - Express router for blob downloads at `/blob/:hash` - Uses S3 SDK to fetch blobs from MinIO with authenticated requests -- Integrates authorization check before serving blobs - -**Authorization (`src/blob-authorization.ts`)** -- `isBlobRestricted()` checks if blob belongs only to restricted branches -- Uses `CHECK_BLOB_RESTRICTED_QUERY` Cypher query from `src/queries.ts` -- Fails safe: treats unknown/error cases as restricted +- No per-blob access control: every stored blob is downloadable by hash #### Data Processing @@ -123,7 +118,6 @@ Custom resolvers for operations not auto-generated by Neo4j GraphQL: See README.md for complete `.env` setup. Key variables: - **Neo4j**: `NEO4J_URI`, `NEO4J_USER`, `NEO4J_PASSWORD` - **Storage**: `OBJECT_STORAGE_URI`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`, `MINIO_OBJECTS_BUCKET_NAME` -- **Authorization**: `RESTRICTED_BRANCH_NAME` - branch containing restricted blobs - **Optional**: `NODE_ENV`, `SENSITIVE_REGISTRY_VALUES` ## Testing Strategy @@ -143,16 +137,12 @@ The project includes comprehensive documentation in the `docs/` directory follow - [Blob Download API](docs/reference/blob-api.md) - Complete REST API spec with error codes, rate limits, examples - [Access Restrictions](docs/reference/access-restrictions.md) - Security controls and filtering mechanisms - **How-To Guides** (`docs/how-to/`) - Step-by-step task instructions - - [Configure Blob Restrictions](docs/how-to/configure-blob-restrictions.md) - Set up branch-based access control - [Integrate Blob API](docs/how-to/integrate-blob-api.md) - Frontend integration guide - **Explanation** (`docs/explanation/`) - Conceptual deep dives - - [Blob Authorization Deep Dive](docs/explanation/blob-authorization.md) - Comprehensive explanation of restriction logic, Cypher queries, design decisions, and security considerations - [Query Optimization](docs/explanation/query-optimization.md) - CALL {} subquery pattern for variable-length path queries **When working on Cypher queries or the graph data model, ALWAYS consult [docs/reference/data-model.md](docs/reference/data-model.md) first.** -**When working on blob authorization or downloads, always consult the docs/ directory first.** - ## Important Patterns ### Module System @@ -168,9 +158,9 @@ The project includes comprehensive documentation in the `docs/` directory follow ### Security Patterns - No per-user authentication. Auth0 was removed when the project went fully - open source; the access restrictions it guarded are disabled by default. -- Sensitive data filtering happens at response time, not query time -- Fail-safe approach: default to restrictive on errors + open source; the access restrictions it guarded are disabled or removed. +- Sensitive data filtering (registry redaction) happens at response time, not + query time, and fails open on error ### GraphQL Patterns - Schema-first development with `type-defs.graphql` diff --git a/README.md b/README.md index 2ca4940..d2b4733 100644 --- a/README.md +++ b/README.md @@ -19,11 +19,6 @@ MINIO_ACCESS_KEY= MINIO_SECRET_KEY= MINIO_OBJECTS_BUCKET_NAME=objects -# Optional - Blob Access Control -# Unused unless the restriction commented out in src/rest-routes.ts is -# re-enabled (see docs/reference/access-restrictions.md) -RESTRICTED_BRANCH_NAME= - # Optional - Environment NODE_ENV=development diff --git a/docs/README.md b/docs/README.md index 5996a27..1dbc866 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,7 +18,6 @@ Reference documentation provides technical specifications and API details. How-to guides show you how to solve specific problems and complete tasks. -- [Configure Blob Restrictions](./how-to/configure-blob-restrictions.md) - Set up branch-based blob access control - [Integrate Blob API (Frontend)](./how-to/integrate-blob-api.md) - Use the blob API from your frontend application ### 💡 [Explanation](./explanation/) @@ -26,7 +25,7 @@ How-to guides show you how to solve specific problems and complete tasks. Explanation documentation helps you understand the concepts and design decisions. -- [Blob Authorization Deep Dive](./explanation/blob-authorization.md) - How blob restrictions work internally +- [Query Optimization](./explanation/query-optimization.md) - CALL {} subquery pattern for variable-length path queries ### 🎓 Tutorials *Learning-oriented - Step-by-step learning paths* @@ -37,49 +36,14 @@ Explanation documentation helps you understand the concepts and design decisions ### For Developers -**Just getting started?** -- Read [Blob Authorization Explanation](./explanation/blob-authorization.md) to understand why restrictions exist -- Follow [Configure Blob Restrictions](./how-to/configure-blob-restrictions.md) to set up your environment - **Integrating the API?** - Check [Blob API Reference](./reference/blob-api.md) for complete endpoint specifications - Review [Access Restrictions](./reference/access-restrictions.md) to understand security controls - Follow [Frontend Integration Guide](./how-to/integrate-blob-api.md) for client implementation **Troubleshooting?** -- See troubleshooting section in [Configuration Guide](./how-to/configure-blob-restrictions.md#troubleshooting) - Check error codes in [API Reference](./reference/blob-api.md#error-responses) -### For System Administrators - -**Configuring restrictions:** -1. Set `OBJECT_STORAGE_URI` and `RESTRICTED_BRANCH_NAME` in `.env` -2. Verify with test requests (see [Configuration Guide](./how-to/configure-blob-restrictions.md#step-3-test-blob-restrictions)) -3. Monitor logs for authorization decisions - -**Security considerations:** -- Review [Security Considerations](./explanation/blob-authorization.md#security-considerations) -- Configure S3 bucket policies to prevent direct access -- Monitor 403 responses for unauthorized access attempts - -## Feature Overview: Blob Download Restrictions - -### Problem -Windows binaries cannot be redistributed due to Microsoft licensing restrictions, but the system needs to allow exploration and downloading of Linux files. - -### Solution -A two-tier approach: -1. **GraphQL API** - Returns blob hashes freely for exploration (browsing, searching, diffing) -2. **REST API** - Controls actual downloads with branch-based authorization - -### Key Concepts - -**Branch-Based Restrictions**: Blobs are restricted if they belong exclusively to a configured restricted branch (e.g., `windows-10`). - -**Fail-Safe Default**: If authorization check fails, access is denied (better to block accidentally than allow). - -**Streaming Architecture**: Large files are streamed directly from object storage without buffering in memory. - ## Contributing to Documentation When adding new documentation: diff --git a/docs/explanation/blob-authorization.md b/docs/explanation/blob-authorization.md deleted file mode 100644 index e8b67b2..0000000 --- a/docs/explanation/blob-authorization.md +++ /dev/null @@ -1,338 +0,0 @@ -# Blob Authorization: How It Works - -## Background - -### The Problem - -OSWatcher analyzes and stores operating system snapshots, including: -- **Linux distributions** (Ubuntu, Debian, etc.) - Open source, freely redistributable -- **Windows systems** (Windows 10, Windows 11) - Proprietary, licensing restrictions apply - -Windows binaries (`.exe`, `.dll`, `.sys` files) are stored as blobs in object storage. Microsoft's licensing terms prohibit redistribution of Windows binaries without proper licensing. - -### Why Not Just Hide the Hashes? - -The initial consideration was to remove blob hashes from GraphQL responses entirely. However, this proved problematic because: - -1. **Registry Viewer Dependency**: The Windows registry viewer feature requires blob hashes to display registry hives -2. **UX Impact**: Users need to see file hashes for exploration and verification -3. **API Complexity**: Filtering responses in GraphQL requires complex plugin logic - -### The Solution - -Instead of hiding hashes, we: -1. **Keep GraphQL unchanged** - Blob hashes remain visible for exploration -2. **Control downloads** - A separate REST API endpoint enforces restrictions -3. **Branch-based logic** - Restrictions tied to branch ancestry, not OS detection - -## Authorization Architecture - -### Two-Tier Access Control - -``` -┌─────────────────────────────────────────────────────────┐ -│ GraphQL API (Exploration) │ -│ - Returns blob hashes freely │ -│ - No restrictions on queries │ -│ - Used for: browsing, searching, diffing │ -└─────────────────────────────────────────────────────────┘ - │ - │ User wants to download - ↓ -┌─────────────────────────────────────────────────────────┐ -│ REST API (Download Control) │ -│ GET /blob/:hash │ -│ 1. Validate hash format │ -│ 2. Check authorization → Neo4j query │ -│ 3. Proxy to S3/MinIO (if allowed) │ -│ 4. Stream content to client │ -└─────────────────────────────────────────────────────────┘ -``` - -### Authorization Flow - -``` -User clicks download → REST API receives request - │ - ↓ - Validate SHA-1 hash format - │ - ↓ - Query Neo4j: isBlobRestricted(hash, branch) - │ - ┌─────────┴─────────┐ - │ │ - ↓ ↓ - true (restricted) false (allowed) - │ │ - ↓ ↓ - 403 Forbidden Proxy to S3 - │ - ↓ - Stream to client -``` - -## The Cypher Query - -### Query Logic - -```cypher -MATCH (b:Blob) -WHERE b.hash = $blob_hash -WITH b - --- Find all commits containing this blob -MATCH (b)<-[:HAS_CHILD_BLOB]-(t:Tree)<-[:HAS_CHILD_TREE|OWNS_FILESYSTEM*]-(c:Commit) -WITH collect(c) as commit_list_where_hash - --- Find all commits reachable from restricted branch -MATCH (br:Branch)-[:TRACKS_COMMIT|HAS_PREVIOUS*]-(c:Commit) -WHERE br.name = $branch_name -WITH commit_list_where_hash, collect(c) as branch_reachable_commit_list - --- Check if ALL blob commits are in restricted branch -RETURN all(c IN commit_list_where_hash WHERE c IN branch_reachable_commit_list) as is_restricted -``` - -### How It Works - -1. **Step 1**: Find the blob node by hash -2. **Step 2**: Traverse up the filesystem tree to find all commits containing this blob - - `Blob ← Tree ← Tree ← ... ← Commit` -3. **Step 3**: Find all commits reachable from the restricted branch - - `Branch → Commit → Commit → ...` (via `TRACKS_COMMIT` and `HAS_PREVIOUS`) -4. **Step 4**: Check if **every** commit containing the blob is in the restricted branch - - If yes → blob is exclusive to restricted branch → **BLOCK** - - If no → blob exists in non-restricted commits → **ALLOW** - -### Example Scenarios - -#### Scenario 1: Blob Exclusive to Windows - -``` -Branch: windows-10 - → Commit A (contains blob X) - → Commit B (contains blob X) - -Branch: ubuntu-server - → Commit C - → Commit D - -Result: isBlobRestricted(X, "windows-10") = true -Reason: All commits with blob X are in windows-10 branch -``` - -#### Scenario 2: Blob in Multiple Branches - -``` -Branch: windows-10 - → Commit A (contains blob Y) - -Branch: ubuntu-server - → Commit B (contains blob Y) ← Same blob! - -Result: isBlobRestricted(Y, "windows-10") = false -Reason: Blob Y exists in ubuntu-server (not restricted) -``` - -#### Scenario 3: Shared Library File - -``` -Branch: windows-10 - → Commit A (contains blob Z - a .dll) - -Branch: windows-11 - → Commit B (contains blob Z - same .dll) - -Branch: ubuntu-server - → Commit C (different files) - -Result: isBlobRestricted(Z, "windows-10") = false -Reason: Blob Z also exists in windows-11 (not all commits are in windows-10) - -Note: If we checked windows-11 instead: -Result: isBlobRestricted(Z, "windows-11") = false (also in windows-10) - -This means blobs shared between restricted branches are ALLOWED! -``` - -## Design Decisions - -### Why Branch-Based Instead of OS Detection? - -**Initial Idea**: Detect Windows commits by checking for Windows-specific node labels (`WinRegKey`, `Symbol`, etc.) - -**Problems**: -- Complex logic to identify Windows vs Linux -- PDB symbols (Windows debugging info) CAN be redistributed -- Registry hives' legal status is unclear -- Need to differentiate blob types (filesystem vs registry vs symbols) - -**Better Solution**: Branch-based restrictions -- Simple: "Is this blob only in the restricted branch?" -- Flexible: Works for any restricted content (not just Windows) -- Clear: Admin explicitly configures which branch is restricted - -### Why Allow Blobs in Multiple Branches? - -**Reasoning**: If a blob appears in both restricted and non-restricted branches, it's likely: -1. A shared file (e.g., common system library) -2. Accidentally duplicated data -3. Content that was re-imported into a non-restricted branch - -In all cases, the conservative approach is to **allow download** since it exists in an unrestricted context. - -**Alternative considered**: Block if blob appears in ANY restricted branch -- Rejected because it would block legitimate shared content - -### Why No Caching? - -**Current**: Each download request queries Neo4j - -**Pros**: -- Always accurate (no stale cache) -- Simple implementation -- No cache invalidation complexity - -**Cons**: -- Higher database load -- Slower response time - -**Future consideration**: Add Redis cache with short TTL (1-5 minutes) - -### Fail-Safe Principle - -**Rule**: When in doubt, block access - -**Implementation**: -```typescript -try { - const result = await session.executeRead(/* query */); - return result.records[0].get('is_restricted'); -} catch (error) { - console.error('Authorization error:', error); - return true; // Treat as restricted on error -} -``` - -**Rationale**: Better to accidentally block access than accidentally allow restricted content - -## Performance Characteristics - -### Query Complexity - -- **Time**: O(n) where n = total commits in graph -- **Space**: O(n) for collecting commit lists -- **Bottleneck**: `HAS_PREVIOUS*` traversal (can be long in commit history) - -### Optimization Opportunities - -1. **Index on Blob.hash**: Already exists (unique constraint) -2. **Index on Branch.name**: Should add if querying many branches -3. **Query plan**: Neo4j query planner should optimize path traversals -4. **Limit traversal depth**: Could add max depth to prevent runaway queries - -### Scaling Considerations - -**Current load**: Acceptable for moderate download traffic (< 100 req/min per blob) - -**High load scenarios**: -- Many concurrent downloads of same blob → add caching -- Large commit graphs (> 100k commits) → optimize query with depth limits -- Multiple restricted branches → refactor to batch check all at once - -## Security Considerations - -### Attack Vectors - -1. **Direct S3 access**: Mitigated by S3 bucket policy (only allow API server IP) -2. **Hash enumeration**: Hashes visible in GraphQL, but download is controlled -3. **Authorization bypass**: Fail-safe default blocks on errors -4. **Race conditions**: Neo4j transactions are atomic (no race condition) - -### Threat Model - -**Assumed threats**: -- Unauthorized users attempting to download Windows binaries -- Automated scripts scraping blob hashes and downloading - -**Not defended against**: -- Authenticated users with legitimate Windows licenses -- Users who already have local copies of Windows files -- Reverse engineering the authorization logic (it's open source) - -### Compliance - -**Goal**: Prevent accidental redistribution of Windows binaries - -**Non-goal**: DRM or preventing determined attackers - -**Legal disclaimer**: This system provides technical controls but does not constitute legal advice. Consult with legal counsel regarding software licensing compliance. - -## Future Enhancements - -### 1. Fine-Grained Blob Types - -Differentiate between: -- Filesystem blobs (exe, dll, sys) → BLOCK -- Registry blobs (hive data) → TBD based on legal research -- Symbol blobs (PDB data) → ALLOW - -Implementation: -```cypher -// Check blob type by relationship -MATCH (b:Blob {hash: $hash}) -OPTIONAL MATCH (b)<-[:HAS_CHILD_BLOB]-(tree:Tree) -OPTIONAL MATCH (b)-[:HAS_WINREG]->(reg:WinRegKey) -OPTIONAL MATCH (b)-[:HAS_SYMBOL]->(sym:Symbol) - -RETURN - tree IS NOT NULL as is_filesystem_blob, - reg IS NOT NULL as is_registry_blob, - sym IS NOT NULL as is_symbol_blob -``` - -### 2. Role-Based Access - -Allow Windows downloads for users with specific permissions: - -```typescript -const hasWindowsLicense = req.auth?.payload?.permissions?.includes('download:windows'); -if (!hasWindowsLicense) { - // Check restrictions -} -``` - -### 3. Audit Logging - -Track all download attempts: - -```typescript -await logDownload({ - blobHash: hash, - userId: req.auth?.payload?.sub, - timestamp: new Date(), - allowed: !restricted, - ipAddress: req.ip -}); -``` - -### 4. Rate Limiting Per User - -Currently: 100 req/min per IP -Future: Different limits for authenticated vs unauthenticated users - -### 5. Temporary Access Tokens - -Generate time-limited download URLs: - -```typescript -const token = generateToken(blobHash, expiresIn: '1h'); -const url = `/blob/${hash}?token=${token}`; -``` - -## Related Documentation - -- [Blob API Reference](../reference/blob-api.md) - API specification -- [Configuration Guide](../how-to/configure-blob-restrictions.md) - Setup instructions -- [Frontend Integration](../how-to/integrate-blob-api.md) - Client implementation diff --git a/docs/how-to/configure-blob-restrictions.md b/docs/how-to/configure-blob-restrictions.md deleted file mode 100644 index 9f4e4e9..0000000 --- a/docs/how-to/configure-blob-restrictions.md +++ /dev/null @@ -1,222 +0,0 @@ -# How to Configure Blob Download Restrictions - -## Prerequisites - -- Neo4j database with OSWatcher graph data -- S3-compatible object storage (S3, MinIO, etc.) -- Branch in Neo4j containing restricted content - -## Step 1: Configure Environment Variables - -Add the following variables to your `.env` file: - -```bash -# Object storage endpoint -OBJECT_STORAGE_URI=https://s3.amazonaws.com/your-bucket -# Or for MinIO: -# OBJECT_STORAGE_URI=https://minio.example.com/bucket-name - -# Branch name containing restricted blobs -RESTRICTED_BRANCH_NAME=windows-10 -``` - -### Environment Variable Details - -| Variable | Type | Required | Description | -|----------|------|----------|-------------| -| `OBJECT_STORAGE_URI` | URL | Yes | Full URL to object storage bucket endpoint | -| `RESTRICTED_BRANCH_NAME` | String | Yes | Name of the branch containing restricted content | - -## Step 2: Verify Configuration - -Start the server and check the startup messages: - -```bash -npm run serve -``` - -Expected output: -``` -🚀 Server ready at http://localhost:4000/graphql -🔌 WebSocket ready at ws://localhost:4000/graphql -📦 Blob API ready at http://localhost:4000/blob -``` - -## Step 3: Test Blob Restrictions - -### Test with an Unrestricted Blob - -Find a blob hash from a non-restricted branch: - -```bash -curl -X GET http://localhost:4000/blob/{hash_from_ubuntu} -``` - -Expected: 200 OK with blob content - -### Test with a Restricted Blob - -Find a blob hash exclusive to the restricted branch: - -```bash -curl -X GET http://localhost:4000/blob/{hash_from_windows} -``` - -Expected: 403 Forbidden with error message - -## Step 4: Verify Authorization Query - -Connect to your Neo4j database and test the authorization query manually: - -```cypher -// Replace $blob_hash and $branch_name with your values -MATCH (b:Blob) -WHERE b.hash = $blob_hash -WITH b -MATCH (b)<-[:HAS_CHILD_BLOB]-(t:Tree)<-[:HAS_CHILD_TREE|OWNS_FILESYSTEM*]-(c:Commit) -WITH collect(c) as commit_list_where_hash -MATCH (br:Branch)-[:TRACKS_COMMIT|HAS_PREVIOUS*]-(c:Commit) -WHERE br.name = $branch_name -WITH commit_list_where_hash, collect(c) as branch_reachable_commit_list -RETURN all(c IN commit_list_where_hash WHERE c IN branch_reachable_commit_list) as is_restricted -``` - -Results: -- `is_restricted: true` → Blob will be blocked -- `is_restricted: false` → Blob will be allowed - -## Common Configuration Scenarios - -### Scenario 1: Restrict Multiple Windows Branches - -Unfortunately, only one restricted branch is currently supported. To restrict multiple branches: - -**Option A**: Create a parent branch that tracks all restricted commits -**Option B**: Modify the code to accept comma-separated branch names (requires code change) - -### Scenario 2: Allow Windows Blobs for Authenticated Users - -The current implementation blocks restricted blobs for all users. To allow based on authentication: - -1. Modify `src/rest-routes.ts` to check `req.auth` (JWT payload) -2. Skip restriction check if user has specific permissions -3. Example: - -```typescript -// In src/rest-routes.ts -const isAuthenticated = req.auth?.payload?.sub; -const hasWindowsAccess = req.auth?.payload?.permissions?.includes('download:windows'); - -if (!hasWindowsAccess) { - const restricted = await isBlobRestricted(driver, hash, restrictedBranchName); - if (restricted) { - return res.status(403).json({ error: "Forbidden" }); - } -} -``` - -### Scenario 3: Different Object Storage URLs - -If blobs are stored in multiple buckets: - -**Current limitation**: Only one `OBJECT_STORAGE_URI` is supported - -**Workaround**: Use a reverse proxy to route to different buckets based on hash patterns - -### Scenario 4: Temporary Access to Restricted Blobs - -To temporarily allow access without changing the branch restriction: - -1. Create a new branch from the restricted commits -2. Name it differently (e.g., `windows-10-public`) -3. The blobs will now appear in multiple branches and be allowed -4. Delete the temporary branch when access should be revoked - -## Troubleshooting - -### Problem: All blobs are blocked - -**Check:** -- Is `RESTRICTED_BRANCH_NAME` set to a branch that contains all your data? -- Verify the branch exists: `MATCH (b:Branch {name: "your-branch"}) RETURN b` - -### Problem: Restricted blobs are not blocked - -**Check:** -- Is `RESTRICTED_BRANCH_NAME` correctly spelled? -- Run the authorization query manually to verify results -- Check server logs for authorization errors - -### Problem: 502 Bad Gateway errors - -**Check:** -- Is `OBJECT_STORAGE_URI` accessible from the server? -- Test: `curl {OBJECT_STORAGE_URI}/objects/{known_hash}` -- Verify network connectivity and credentials - -### Problem: Downloads are slow - -**Considerations:** -- Authorization query runs on every download (no caching) -- Network latency between server and object storage -- Consider adding Redis cache for authorization results - -## Security Considerations - -### Fail-Safe Default - -If the authorization check fails (database error, query timeout, etc.), the system defaults to **blocking access**. This ensures restricted content is not accidentally served. - -### Bypass Prevention - -- Blob hashes are still visible in GraphQL queries (for exploration) -- Direct S3 access should be blocked via bucket policies -- Only allow object storage access from the API server's IP - -### Recommended Bucket Policy (AWS S3) - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Deny", - "Principal": "*", - "Action": "s3:GetObject", - "Resource": "arn:aws:s3:::your-bucket/objects/*", - "Condition": { - "StringNotLike": { - "aws:SourceIp": ["your-api-server-ip"] - } - } - } - ] -} -``` - -## Monitoring - -### Log Analysis - -Check server logs for authorization decisions: - -```bash -# Check for restriction checks -grep "Blob download requested" logs/app.log - -# Check for authorization errors -grep "Error checking blob authorization" logs/app.log -``` - -### Metrics to Track - -- Number of 403 responses (restricted blob attempts) -- Authorization query execution time -- Object storage latency -- Rate limit hits (429 responses) - -## Next Steps - -- [Blob API Reference](../reference/blob-api.md) - Full API specification -- [Frontend Integration Guide](./integrate-blob-api.md) - Using the API from frontend -- [Authorization Deep Dive](../explanation/blob-authorization.md) - How restrictions work diff --git a/docs/how-to/integrate-blob-api.md b/docs/how-to/integrate-blob-api.md index f82da13..4329d4e 100644 --- a/docs/how-to/integrate-blob-api.md +++ b/docs/how-to/integrate-blob-api.md @@ -70,8 +70,6 @@ export class BlobService { const data: BlobDownloadError = await response.json(); switch (response.status) { - case 403: - throw new Error('This file is restricted and cannot be downloaded.'); case 404: throw new Error('File not found in storage.'); case 429: @@ -223,95 +221,7 @@ export function FileExplorer() { } ``` -## Step 4: Handle Restricted Blobs - -### Show User-Friendly Error Messages - -```typescript -// components/DownloadButton.tsx - -import { useState } from 'react'; -import { BlobService } from '../services/blobService'; - -interface DownloadButtonProps { - hash: string; - filename: string; -} - -export function DownloadButton({ hash, filename }: DownloadButtonProps) { - const [loading, setLoading] = useState(false); - const [error, setError] = useState(null); - const blobService = new BlobService(API_CONFIG.blobUrl); - - const handleDownload = async () => { - setLoading(true); - setError(null); - - try { - await blobService.downloadBlob(hash, filename); - } catch (err) { - if (err instanceof Error) { - setError(err.message); - } - } finally { - setLoading(false); - } - }; - - return ( -
- - - {error && ( -
- {error} - {error.includes('restricted') && ( -

- This file is from a restricted branch (e.g., Windows) - and cannot be downloaded due to licensing restrictions. -

- )} -
- )} -
- ); -} -``` - -### Precheck Before Download (Optional) - -You can add a HEAD request to check if a blob is accessible before attempting download: - -```typescript -export class BlobService { - /** - * Check if a blob is accessible - */ - async isAccessible(hash: string): Promise { - const url = `${this.blobBaseUrl}/${hash}`; - - try { - const response = await fetch(url, { method: 'HEAD' }); - return response.ok; - } catch { - return false; - } - } -} - -// Usage: -const accessible = await blobService.isAccessible(hash); -if (!accessible) { - showWarning('This file is restricted'); -} -``` - -## Step 5: Update Existing S3 References +## Step 4: Update Existing S3 References If you previously downloaded directly from S3, update those references: @@ -335,69 +245,32 @@ await blobService.downloadBlob(blobHash, filename); ### Test Cases -1. **Unrestricted blob download** +1. **Blob download** - Expected: Successful download -2. **Restricted blob download** - - Expected: 403 error with user-friendly message - -3. **Invalid hash format** +2. **Invalid hash format** - Expected: 400 error -4. **Non-existent blob** +3. **Non-existent blob** - Expected: 404 error -5. **Authenticated vs unauthenticated** - - Both should work the same (auth doesn't affect restrictions currently) - ### Example Test ```typescript // blobService.test.ts describe('BlobService', () => { - it('should download unrestricted blob', async () => { + it('should download a blob', async () => { const service = new BlobService('http://localhost:4000/blob'); await expect( service.downloadBlob('a94a8fe5ccb19ba61c4c0873d391e987982fbbd3') ).resolves.not.toThrow(); }); - - it('should reject restricted blob', async () => { - const service = new BlobService('http://localhost:4000/blob'); - await expect( - service.downloadBlob('restricted_hash_here') - ).rejects.toThrow('restricted'); - }); }); ``` ## Performance Considerations -### Caching - -The API doesn't cache authorization results, so each download checks the database. - -**Recommendation**: Cache the accessibility check result client-side for a short time: - -```typescript -const accessCache = new Map(); -const CACHE_TTL = 60000; // 1 minute - -async function isCachedAccessible(hash: string): Promise { - const cached = accessCache.get(hash); - const now = Date.now(); - - if (cached && now - cached.timestamp < CACHE_TTL) { - return cached.accessible; - } - - const accessible = await blobService.isAccessible(hash); - accessCache.set(hash, { accessible, timestamp: now }); - return accessible; -} -``` - ### Concurrent Downloads Implement a download queue to prevent overwhelming the server: @@ -435,5 +308,3 @@ class DownloadQueue { ## Related Documentation - [Blob API Reference](../reference/blob-api.md) - Complete API specification -- [Configure Blob Restrictions](./configure-blob-restrictions.md) - Backend configuration -- [Authorization Explanation](../explanation/blob-authorization.md) - How restrictions work diff --git a/docs/reference/access-restrictions.md b/docs/reference/access-restrictions.md index beb3ff0..6c29ed3 100644 --- a/docs/reference/access-restrictions.md +++ b/docs/reference/access-restrictions.md @@ -6,129 +6,9 @@ This document catalogs all access control and data filtering mechanisms implemen The API implements multiple layers of access control to protect sensitive data and prevent abuse: -1. **Blob Download Authorization** - Prevents redistribution of restricted content (e.g., Windows binaries) -2. **Windows Registry Value Filtering** - Redacts sensitive product keys and identifiers from responses +1. **Windows Registry Value Filtering** - Redacts sensitive product keys and identifiers from responses -## 1. Blob Download Authorization - -> **Status: disabled by default (2026-08-11).** The enforcement code in -> `src/rest-routes.ts` is commented out for the open-source release, so all -> blobs are downloadable regardless of branch. Uncomment it (and set -> `RESTRICTED_BRANCH_NAME`) to restore this restriction for a deployment -> that needs to comply with Windows redistribution licensing. The rest of -> this section describes the mechanism as designed, not current runtime -> behavior. - -### Purpose - -Prevents unauthorized downloading of blobs that belong exclusively to restricted branches, primarily to comply with Microsoft Windows licensing restrictions. - -### Scope - -- **Applies to:** REST API endpoint `GET /blob/:hash` -- **Does NOT apply to:** GraphQL queries (blob hashes remain visible for exploration) - -### Implementation - -**File:** `src/blob-authorization.ts` - -**Function:** `isBlobRestricted(driver, blobHash, restrictedBranchName)` - -**Logic:** -1. Query Neo4j to find all commits containing the blob -2. Query Neo4j to find all commits reachable from the restricted branch -3. Check if **ALL** commits containing the blob are in the restricted branch -4. If yes → blob is restricted (403 Forbidden) -5. If no → blob is allowed (download proceeds) - -**Cypher Query:** `CHECK_BLOB_RESTRICTED_QUERY` in `src/queries.ts` - -```cypher -MATCH (b:Blob) -WHERE b.hash = $blob_hash -WITH b - --- Find all commits containing this blob -MATCH (b)<-[:HAS_CHILD_BLOB]-(t:Tree)<-[:HAS_CHILD_TREE|OWNS_FILESYSTEM*]-(c:Commit) -WITH collect(c) as commit_list_where_hash - --- Find all commits reachable from restricted branch -MATCH (br:Branch)-[:TRACKS_COMMIT|HAS_PREVIOUS*]-(c:Commit) -WHERE br.name = $branch_name -WITH commit_list_where_hash, collect(c) as branch_reachable_commit_list - --- Check if ALL blob commits are in restricted branch -RETURN all(c IN commit_list_where_hash WHERE c IN branch_reachable_commit_list) as is_restricted -``` - -### Configuration - -**Environment Variable:** `RESTRICTED_BRANCH_NAME` - -**Example:** -```bash -RESTRICTED_BRANCH_NAME=windows-10 -``` - -### Behavior - -| Scenario | Result | HTTP Status | -|----------|--------|-------------| -| Blob exists only in restricted branch | Blocked | 403 Forbidden | -| Blob exists in restricted + non-restricted branches | Allowed | 200 OK | -| Blob not found in database | Allowed | (proceeds to S3 check) | -| Neo4j query error | Blocked (fail-safe) | 403 Forbidden | -| Blob not found in S3 storage | Error | 404 Not Found | - -### Fail-Safe Principle - -**Rule:** When in doubt, block access - -```typescript -try { - const result = await session.executeRead(/* query */); - return result.records[0].get('is_restricted'); -} catch (error) { - console.error('Authorization error:', error); - return true; // Treat as restricted on error -} -``` - -### Example Scenarios - -#### Scenario 1: Exclusive to Restricted Branch -``` -Branch: windows-10 - → Commit A (contains blob X) - → Commit B (contains blob X) - -Branch: ubuntu-server - → Commit C - → Commit D - -Result: isBlobRestricted(X, "windows-10") = true → 403 Forbidden -``` - -#### Scenario 2: Shared Between Branches -``` -Branch: windows-10 - → Commit A (contains blob Y) - -Branch: ubuntu-server - → Commit B (contains blob Y) ← Same blob! - -Result: isBlobRestricted(Y, "windows-10") = false → 200 OK -``` - -### Related Documentation - -- **Detailed Explanation:** [Blob Authorization Deep Dive](../explanation/blob-authorization.md) -- **Configuration Guide:** [Configure Blob Restrictions](../how-to/configure-blob-restrictions.md) -- **API Specification:** [Blob Download API](./blob-api.md) - ---- - -## 2. Windows Registry Value Filtering +## 1. Windows Registry Value Filtering > **Status: disabled by default (2026-08-11).** The Apollo Server plugin > that invokes `filterSensitiveRegistryValues` is commented out in @@ -375,7 +255,7 @@ When implementing new access restrictions, follow these established patterns: | **GraphQL Schema** | Prevent queries entirely | `@mutation(operations: [])` to disable mutations | | **Resolver Level** | Context-based access control | Check `jwt.permissions` in custom resolvers | | **Response Filter** | Data redaction/masking | Registry value filtering | -| **REST Endpoint** | Resource-level authorization | Blob download restrictions | +| **REST Endpoint** | Resource-level authorization | Per-resource checks before serving a download | ### 2. Fail-Safe Principle @@ -530,7 +410,6 @@ When adding new restrictions: | Restriction | Type | Scope | Fail-Safe | Configurable | Status | |------------|------|-------|-----------|--------------|--------| -| Blob Download Authorization | REST Endpoint | `GET /blob/:hash` | Block (403) | `RESTRICTED_BRANCH_NAME` | Disabled by default (commented out) | | Registry Value Filtering (Sensitive) | Response Filter | All GraphQL responses | Allow (fail-open) | `SENSITIVE_REGISTRY_VALUES` | Disabled by default (commented out) | | Query Complexity | GraphQL Validation | All GraphQL queries | Block (error) | Hardcoded (100 fields) | Active | | Rate Limiting | Middleware | All endpoints | Block (429) | Hardcoded (100/min) | Active | @@ -542,9 +421,7 @@ When adding new restrictions: ## Related Documentation -- [Blob Authorization Deep Dive](../explanation/blob-authorization.md) - Comprehensive explanation of blob restriction design - [Blob Download API](./blob-api.md) - Complete REST API specification -- [Configure Blob Restrictions](../how-to/configure-blob-restrictions.md) - Setup instructions - [Security and Performance](./security-and-performance.md) - Overall security architecture (if exists) --- @@ -556,3 +433,4 @@ When adding new restrictions: | 2025-12-08 | Initial documentation of blob authorization and registry filtering | Claude Code | | 2026-08-11 | Removed auth-gated feature restrictions (registry value hiding, recursive diff block, HISTORY_WITH_UPDATES block, date-based diff limit) ahead of open-source release — these gated features purely on JWT presence with no technical or legal basis, unlike blob download authorization and sensitive-value redaction, which remain | Claude Code | | 2026-08-11 | Disabled (commented out, not deleted) blob download authorization and sensitive registry value redaction by default — self-hosted OSS deployments get full functionality out of the box; both can be re-enabled by uncommenting the marked blocks in `src/rest-routes.ts` and `src/index.ts` | Claude Code | +| 2026-09-08 | Removed blob download authorization entirely (`src/blob-authorization.ts`, `CHECK_BLOB_RESTRICTED_QUERY`, `RESTRICTED_BRANCH_NAME`, and the commented-out enforcement block). It gated only Windows binaries, which are not in the public corpus; git history keeps the implementation. Sensitive registry value redaction is unaffected. | Claude Code | diff --git a/docs/reference/blob-api.md b/docs/reference/blob-api.md index bcd5ec6..b7c1a24 100644 --- a/docs/reference/blob-api.md +++ b/docs/reference/blob-api.md @@ -6,7 +6,7 @@ GET /blob/:hash ``` -Downloads a blob by its SHA-1 hash with authorization checks. +Downloads a blob by its SHA-1 hash. ## Parameters @@ -48,16 +48,6 @@ Invalid blob hash format. } ``` -#### 403 Forbidden -Blob is restricted and cannot be downloaded. - -```json -{ - "error": "Forbidden", - "message": "This blob is restricted" -} -``` - #### 404 Not Found Blob not found in object storage. @@ -100,26 +90,8 @@ Error communicating with object storage. ## Authorization -Access control is limited to blob restriction: a Neo4j query checks whether the -blob belongs exclusively to restricted branches. There is no per-user -authentication. - -### Restriction Logic - -A blob is **restricted** if: -- ALL commits containing the blob are reachable from the configured restricted branch -- Example: If `RESTRICTED_BRANCH_NAME=windows-10`, blobs exclusive to that branch are blocked - -A blob is **allowed** if: -- The blob appears in at least one commit NOT reachable from the restricted branch -- Example: If a blob exists in both `windows-10` and `ubuntu-server`, it's allowed - -### Fail-Safe Behavior - -If an error occurs during authorization checking: -- The blob is treated as **restricted** (access denied) -- Error is logged server-side -- Returns 403 Forbidden to client +The endpoint is unauthenticated and performs no per-blob access control. Every +stored blob is downloadable by hash. ## Examples @@ -130,26 +102,6 @@ curl -X GET http://localhost:4000/blob/a94a8fe5ccb19ba61c4c0873d391e987982fbbd3 -o downloaded-file.bin ``` -### Restricted Blob - -```bash -curl -X GET http://localhost:4000/blob/c3499c2729730a7f807efb8676a92dcb6f8a3f8f - -# Response: 403 Forbidden -{ - "error": "Forbidden", - "message": "This blob is restricted" -} -``` - -### With Authentication - -```bash -curl -X GET http://localhost:4000/blob/a94a8fe5ccb19ba61c4c0873d391e987982fbbd3 \ - -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..." \ - -o downloaded-file.bin -``` - ### Invalid Hash Format ```bash @@ -179,20 +131,15 @@ The endpoint supports CORS with configured allowed origins: ### Architecture ``` -Client → Express Router → Authorization Check → S3/MinIO Proxy → Stream Response - ↓ - Neo4j Query +Client → Express Router → S3/MinIO Proxy → Stream Response ``` ### Performance Characteristics - **Streaming**: Large files are streamed without buffering in memory -- **Query Complexity**: Single Neo4j Cypher query per request (O(n) where n = commits in graph) - **Caching**: No caching implemented (considers each request fresh) ### Source Code - **Endpoint Handler**: `src/rest-routes.ts` -- **Authorization Logic**: `src/blob-authorization.ts` -- **Cypher Query**: `src/queries.ts` (`CHECK_BLOB_RESTRICTED_QUERY`) - **Validation Schema**: `src/validation.ts` (`BlobHashParamSchema`) diff --git a/src/blob-authorization.ts b/src/blob-authorization.ts deleted file mode 100644 index caefefb..0000000 --- a/src/blob-authorization.ts +++ /dev/null @@ -1,44 +0,0 @@ -import { Driver } from "neo4j-driver"; -import { CHECK_BLOB_RESTRICTED_QUERY } from "./queries.js"; - -/** - * Check if a blob is restricted (only belongs to commits in restricted branches) - * - * @param driver Neo4j driver instance - * @param blobHash SHA-1 hash of the blob to check - * @param restrictedBranchName Name of the branch containing restricted commits (e.g., "windows-10") - * @returns true if blob is restricted (belongs ONLY to restricted commits), false if available for download - */ -export async function isBlobRestricted( - driver: Driver, - blobHash: string, - restrictedBranchName: string, -): Promise { - const session = driver.session(); - - try { - const result = await session.executeRead(async (tx) => { - return tx.run(CHECK_BLOB_RESTRICTED_QUERY, { - blob_hash: blobHash, - branch_name: restrictedBranchName, - }); - }); - - // Handle case where blob doesn't exist in database - if (result.records.length === 0) { - console.warn( - `Blob ${blobHash} not found in database, allowing download`, - ); - return false; // Allow download if blob not tracked in DB - } - - const isRestricted = result.records[0].get("is_restricted"); - return isRestricted === true; - } catch (error) { - console.error("Error checking blob authorization:", error); - // Fail safe: on error, treat as restricted - return true; - } finally { - await session.close(); - } -} diff --git a/src/index.ts b/src/index.ts index 7bc087b..1aa7ba3 100644 --- a/src/index.ts +++ b/src/index.ts @@ -21,8 +21,7 @@ import { expressMiddleware } from "@as-integrations/express5"; // Authorization header, nothing a browser could be tricked into attaching. An // origin allow-list therefore protects nothing (curl ignores CORS entirely) // while breaking every local frontend and third-party tool. Access control -// that does matter is enforced server-side: rate limiting and restricted-branch -// filtering. +// that does matter is enforced server-side: rate limiting. import cors from "cors"; import { createServer } from "http"; import { WebSocketServer } from "ws"; @@ -36,13 +35,6 @@ const env = cleanEnv(process.env, { NEO4J_USER: str({ desc: "Neo4j username" }), NEO4J_PASSWORD: str({ desc: "Neo4j password" }), OBJECT_STORAGE_URI: url({ desc: "S3/MinIO object storage endpoint" }), - // Blob download restriction is disabled by default for the open-source - // release (see rest-routes.ts). Left optional so the server starts - // without configuring it. - RESTRICTED_BRANCH_NAME: str({ - default: "", - desc: "Branch name for restricted blobs (unused unless the restriction in rest-routes.ts is re-enabled)", - }), MINIO_ACCESS_KEY: str({ desc: "MinIO access key for authenticated requests", }), @@ -326,9 +318,7 @@ async function main() { createRateLimit(100, 60000), // 100 requests per minute express.json({ limit: "1mb" }), createRestRouter( - driver, env.OBJECT_STORAGE_URI, - env.RESTRICTED_BRANCH_NAME, env.MINIO_ACCESS_KEY, env.MINIO_SECRET_KEY, env.MINIO_OBJECTS_BUCKET_NAME, diff --git a/src/queries.ts b/src/queries.ts index dc7885d..dcf5a9c 100644 --- a/src/queries.ts +++ b/src/queries.ts @@ -307,19 +307,6 @@ RETURN DISTINCT b.hash AS blob_hash, blob_path ORDER BY blob_path `; -// blob authorization -export const CHECK_BLOB_RESTRICTED_QUERY = ` -MATCH (b:Blob) -WHERE b.hash = $blob_hash -WITH b -MATCH (b)<-[:HAS_CHILD_BLOB]-(t:Tree)<-[:HAS_CHILD_TREE|OWNS_FILESYSTEM*]-(c:Commit) -WITH collect(c) as commit_list_where_hash -MATCH (br:Branch)-[:TRACKS_COMMIT|HAS_PREVIOUS*]-(c:Commit) -WHERE br.name = $branch_name -WITH commit_list_where_hash, collect(c) as branch_reachable_commit_list -RETURN all(c IN commit_list_where_hash WHERE c IN branch_reachable_commit_list) as is_restricted -`; - // git log - filesystem root resolution export const GET_FILESYSTEM_ROOT_QUERY = ` MATCH (c:Commit {hash: $commit_hash})-[:OWNS_FILESYSTEM]->(root:Tree) diff --git a/src/rest-routes.ts b/src/rest-routes.ts index fb370ef..9a0e214 100644 --- a/src/rest-routes.ts +++ b/src/rest-routes.ts @@ -1,18 +1,10 @@ import { Router, Request, Response } from "express"; -import { Driver } from "neo4j-driver"; import { BlobHashParamSchema } from "./validation.js"; import { ZodError } from "zod"; -// Blob download restriction is disabled by default for the open-source -// release -- see the commented-out block below and -// docs/reference/access-restrictions.md. Uncomment to re-enable -// branch-based blocking of Windows-exclusive binaries. -// import { isBlobRestricted } from "./blob-authorization.js"; import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3"; export const createRestRouter = ( - driver: Driver, objectStorageUri: string, - _restrictedBranchName: string, minioAccessKey: string, minioSecretKey: string, minioObjectsBucketName: string, @@ -38,36 +30,6 @@ export const createRestRouter = ( console.log(`Blob download requested: ${hash}`); - // Blob download restriction (branch-based blocking of - // Windows-exclusive binaries, per Microsoft redistribution - // licensing) is disabled by default for the open-source - // release. Uncomment to re-enable -- see - // docs/reference/access-restrictions.md. - // - // NOTE: re-enabling this also requires reinstating an auth - // middleware that populates req.auth. Auth0 was removed when the - // project went fully open source; see git history for the - // express-oauth2-jwt-bearer wiring. - // const permissions = (req as any).auth?.payload?.permissions as - // | string[] - // | undefined; - // const hasRestrictedAccess = - // permissions?.includes("download:restricted") ?? false; - // - // if (!hasRestrictedAccess) { - // const restricted = await isBlobRestricted( - // driver, - // hash, - // restrictedBranchName, - // ); - // if (restricted) { - // return res.status(403).json({ - // error: "Forbidden", - // message: "This blob is restricted", - // }); - // } - // } - // Fetch blob from MinIO using S3 SDK const command = new GetObjectCommand({ Bucket: minioObjectsBucketName,