feat: consolidate DataSpark web/core/cli capabilities

.documentation/copilot/audit/2026-03-30_results.md

@@ -6,7 +6,7 @@
 - **Scope**: full
 - **Auditor**: speckit.site-audit
 - **Constitution Version**: 1.0.0
-- **Repository**: markhazleton/sql2csv
+- **Repository**: markhazleton/DataSpark
 
 ---
 
@@ -42,10 +42,10 @@
 
 | Finding | Status |
 |---------|--------|
-| 10 CSRF violations — `sql2csv.web` (SEC1–SEC10) | ✅ FIXED (07:30 audit) |
-| `TreatWarningsAsErrors` missing — `sql2csv.web` (QUAL1) | ✅ FIXED (07:30 audit) |
+| 10 CSRF violations — `DataSpark.Web` (SEC1–SEC10) | ✅ FIXED (07:30 audit) |
+| `TreatWarningsAsErrors` missing — `DataSpark.Web` (QUAL1) | ✅ FIXED (07:30 audit) |
 | `MIN_COVERAGE: "0"` in CI (TEST1) | ✅ FIXED (07:30 audit) |
-| Serilog not wired — `sql2csv.web` (LOG1) | ✅ FIXED (07:30 audit) |
+| Serilog not wired — `DataSpark.Web` (LOG1) | ✅ FIXED (07:30 audit) |
 | SQL identifier concatenation SQL1–SQL4 | ✅ AUDIT CORRECTION — bracket-escaping IS present |
 | 15 CSRF violations — `DataSpark.Web` (SEC1–SEC5) | ✅ FIXED — `[ValidateAntiForgeryToken]` added to all 15 POST endpoints |
 | `Console.WriteLine` in Core (LOG1) | ✅ FIXED — replaced with silent catch in `UnivariateAnalysisExtensions.cs` |
@@ -88,10 +88,10 @@ No version findings — Spec Kit Spark is current.
 | ID | Principle | File:Line | Fix Applied |
 |----|-----------|-----------|-------------|
 | SEC1–SEC5 | IV. CSRF | `DataSpark.Web/Controllers/*` | ✅ `[ValidateAntiForgeryToken]` added to all 15 POST endpoints; commented-out attribute in `ChartController.cs:134` restored |
-| ASYNC1 | III. Async/Await | `Sql2Csv.Core/Services/*` | ✅ `.ConfigureAwait(false)` added to all fixable `await` calls across 9 files (`await using` declarations exempt — see note) |
-| LOG1 | VII. Structured Logging | `Sql2Csv.Core/Models/Analysis/UnivariateAnalysisExtensions.cs:16` | ✅ `Console.WriteLine` removed; silent catch applied (no `ILogger` available in static extension) |
-| TEST1 | II. Testing Standards | `Sql2Csv.Tests/Controllers/UnifiedDataControllerTests.cs` | ✅ All `Assert.*` replaced with FluentAssertions (`Should().Be()`, `Should().NotBeNull()`, etc.) |
-| TEST2 | II. Testing Standards | `Sql2Csv.Tests/Services/ExportFilteringTests.cs:31` | ✅ `Assert.IsTrue(...)` replaced with `.Should().OnlyContain(...)` |
+| ASYNC1 | III. Async/Await | `DataSpark.Core/Services/*` | ✅ `.ConfigureAwait(false)` added to all fixable `await` calls across 9 files (`await using` declarations exempt — see note) |
+| LOG1 | VII. Structured Logging | `DataSpark.Core/Models/Analysis/UnivariateAnalysisExtensions.cs:16` | ✅ `Console.WriteLine` removed; silent catch applied (no `ILogger` available in static extension) |
+| TEST1 | II. Testing Standards | `DataSpark.Tests/Controllers/UnifiedDataControllerTests.cs` | ✅ All `Assert.*` replaced with FluentAssertions (`Should().Be()`, `Should().NotBeNull()`, etc.) |
+| TEST2 | II. Testing Standards | `DataSpark.Tests/Services/ExportFilteringTests.cs:31` | ✅ `Assert.IsTrue(...)` replaced with `.Should().OnlyContain(...)` |
 | LOG2 | VII. Structured Logging | `DataSpark.Web/Program.cs` | ✅ `UseSerilog()` wired; TODO comment removed from `.csproj` |
 | QUAL1–QUAL3 | V. Code Quality | Large files >500 lines | ⚠️ Deferred — file size reduction is a future refactoring task |
 
@@ -116,7 +116,7 @@ No version findings — Spec Kit Spark is current.
 - [x] File uploads validate file extension
 - [x] No SQL injection via user-supplied values (`@params` used correctly)
 - [x] SQL identifiers use bracket-escaping `[{name}]` interpolation pattern
-- [x] `sql2csv.web` — all `[HttpPost]` endpoints have `[ValidateAntiForgeryToken]`
+- [x] `DataSpark.Web` — all `[HttpPost]` endpoints have `[ValidateAntiForgeryToken]`
 - [x] **`DataSpark.Web` — all 15 `[HttpPost]` endpoints now have `[ValidateAntiForgeryToken]`** ✅ FIXED
 - [ ] File uploads do not perform magic-byte (content signature) validation — extension check only (backlog item)
 - [x] Sensitive paths sanitized via `DatabaseConfiguration`
@@ -146,7 +146,7 @@ All NuGet packages are at current versions (targeting .NET 10). No known vulnera
 | Metric | Value |
 |--------|-------|
 | Approved NuGet packages (Core) | 11 |
-| npm devDependencies (`sql2csv.web`) | 11 |
+| npm devDependencies (`DataSpark.Web`) | 11 |
 | npm dependencies | 1 (alpinejs) |
 | Outdated | 0 |
 | Vulnerable (known) | 0 |
@@ -178,10 +178,10 @@ is technically approved but not yet wired — see LOG2 finding.
 | File | Lines | Recommendation |
 |------|-------|----------------|
 | `DataSpark.Web/Controllers/api/FilesController.cs` | 1,016 | Split file operations into focused controllers |
-| `Sql2Csv.Core/Services/OpenAIFileAnalysisService.cs` | 813 | Extract AI prompt-building into a helper |
-| `sql2csv.web/Controllers/HomeController.cs` | 707 | Extract upload/export/schema groups into separate controllers |
-| `Sql2Csv.Core/Services/DatabaseAnalysisService.cs` | 683 | Extract statistical analysis into a sub-service |
-| `sql2csv.web/src/js/table-components.js` | 548 | Split Alpine.js component definitions |
+| `DataSpark.Core/Services/OpenAIFileAnalysisService.cs` | 813 | Extract AI prompt-building into a helper |
+| `DataSpark.Web/Controllers/HomeController.cs` | 707 | Extract upload/export/schema groups into separate controllers |
+| `DataSpark.Core/Services/DatabaseAnalysisService.cs` | 683 | Extract statistical analysis into a sub-service |
+| `DataSpark.Web/src/js/table-components.js` | 548 | Split Alpine.js component definitions |
 
 ---
 
@@ -203,8 +203,8 @@ is technically approved but not yet wired — see LOG2 finding.
 
 | ID | File | Issue | Severity |
 |----|------|-------|----------|
-| TEST1 | `Sql2Csv.Tests/Controllers/UnifiedDataControllerTests.cs` | 10+ raw `Assert.*` calls — must use FluentAssertions | MEDIUM |
-| TEST2 | `Sql2Csv.Tests/Services/ExportFilteringTests.cs:31` | `Assert.IsTrue(...)` — must use FluentAssertions | MEDIUM |
+| TEST1 | `DataSpark.Tests/Controllers/UnifiedDataControllerTests.cs` | 10+ raw `Assert.*` calls — must use FluentAssertions | MEDIUM |
+| TEST2 | `DataSpark.Tests/Services/ExportFilteringTests.cs:31` | `Assert.IsTrue(...)` — must use FluentAssertions | MEDIUM |
 
 **Recommendation**: Replace `Assert.IsNotNull(x)` → `x.Should().NotBeNull()`,
 `Assert.AreEqual(a, b)` → `b.Should().Be(a)`, `Assert.IsTrue(cond)` → `cond.Should().BeTrue()`.
@@ -216,7 +216,7 @@ is technically approved but not yet wired — see LOG2 finding.
 ### Missing ConfigureAwait(false) in Core
 
 33 `await` calls across 9 Core service files are missing `.ConfigureAwait(false)`.
-Per Principle III: "Every `await` in `Sql2Csv.Core` MUST append `.ConfigureAwait(false)`."
+Per Principle III: "Every `await` in `DataSpark.Core` MUST append `.ConfigureAwait(false)`."
 
 | File | Missing Count | Representative Lines |
 |------|--------------|---------------------|
@@ -242,9 +242,9 @@ Per Principle III: "Every `await` in `Sql2Csv.Core` MUST append `.ConfigureAwait
 | `ILogger<T>` via constructor injection | ✅ All Core services |
 | Structured templates (no interpolation in log calls) | ✅ |
 | `LogError` with exception as first argument | ✅ |
-| `sql2csv.web` uses `UseSerilog()` in `Program.cs` | ✅ Confirmed at `Program.cs:17` |
+| `DataSpark.Web` uses `UseSerilog()` in `Program.cs` | ✅ Confirmed at `Program.cs:17` |
 | No `Console.Write*` in web/console layers | ✅ |
-| No `Console.Write*` in `Sql2Csv.Core` | ❌ `UnivariateAnalysisExtensions.cs:16` |
+| No `Console.Write*` in `DataSpark.Core` | ❌ `UnivariateAnalysisExtensions.cs:16` |
 | `DataSpark.Web` Serilog wired | ❌ Package present, `UseSerilog()` not called |
 
 ---
@@ -264,7 +264,7 @@ Per Principle III: "Every `await` in `Sql2Csv.Core` MUST append `.ConfigureAwait
 ### Medium Priority (Next Sprint)
 
 3. **LOG1** — Replace `Console.WriteLine(...)` at
-   `Sql2Csv.Core/Models/Analysis/UnivariateAnalysisExtensions.cs:16` with `ILogger<T>` logging.
+   `DataSpark.Core/Models/Analysis/UnivariateAnalysisExtensions.cs:16` with `ILogger<T>` logging.
 
 4. **TEST1/TEST2** — Replace raw `Assert.*` calls in `UnifiedDataControllerTests.cs` and
    `ExportFilteringTests.cs` with FluentAssertions equivalents.
@@ -283,12 +283,12 @@ Per Principle III: "Every `await` in `Sql2Csv.Core` MUST append `.ConfigureAwait
 
 | Finding | 07:30 UTC Audit | 08:05 UTC Audit | Trend |
 |---------|----------------|----------------|-------|
-| CSRF violations — `sql2csv.web` | 10 ❌ | 0 ✅ | ↓ FIXED |
+| CSRF violations — `DataSpark.Web` | 10 ❌ | 0 ✅ | ↓ FIXED |
 | CSRF violations — `DataSpark.Web` | Not audited | 15 ❌ | 🆕 NEW |
 | Async `ConfigureAwait` missing | ~237 ❌ | 33 ❌ | ↓ Improved |
-| `TreatWarningsAsErrors` missing | `sql2csv.web` ❌ | 0 (all fixed) ✅ | ↓ FIXED |
+| `TreatWarningsAsErrors` missing | `DataSpark.Web` ❌ | 0 (all fixed) ✅ | ↓ FIXED |
 | `MIN_COVERAGE = "0"` | ❌ | ✅ (`"80"`) | ↓ FIXED |
-| Serilog not wired — `sql2csv.web` | ❌ | ✅ | ↓ FIXED |
+| Serilog not wired — `DataSpark.Web` | ❌ | ✅ | ↓ FIXED |
 | SQL identifier concatenation (SQL1–SQL4) | ❌ Flagged | ✅ Cleared (audit correction) | — |
 | `Console.WriteLine` in Core | Not flagged | ❌ | 🆕 NEW |
 | Raw `Assert.*` in tests | Not flagged | ❌ | 🆕 NEW |
@@ -309,6 +309,6 @@ Per Principle III: "Every `await` in `Sql2Csv.Core` MUST append `.ConfigureAwait
 ---
 
 *Audit generated by speckit.site-audit v1.0*
-*Constitution-driven codebase audit for sql2csv*
+*Constitution-driven codebase audit for DataSpark*
 *Next audit recommended: 2026-04-06*
 *To re-run: `/speckit.site-audit` or `/speckit.site-audit --scope=constitution`*

.documentation/memory/constitution.md

@@ -6,21 +6,21 @@
 
 ### I. Clean Architecture — Core-First (MANDATORY)
 
-All business logic, database access, and data transformation MUST reside in `Sql2Csv.Core`.
-The `sql2csv.web` and `sql2csv.console` projects are presentation layers only — they wire up DI,
+All business logic, database access, and data transformation MUST reside in `DataSpark.Core`.
+The `DataSpark.Web` and `DataSpark.Console` projects are presentation layers only — they wire up DI,
 handle HTTP/CLI concerns, and delegate all work to Core services.
 
 - (MUST) No database queries in Web controllers or Console commands
 - (MUST) No business logic in ViewModels, Views, or Presentation classes
-- (MUST) `Sql2Csv.Core` has zero dependencies on `sql2csv.web` or `sql2csv.console`
+- (MUST) `DataSpark.Core` has zero dependencies on `DataSpark.Web` or `DataSpark.Console`
 - (MUST) New features start as Core service interfaces before any presentation wiring
 
-**Evidence**: `Sql2Csv.Core/Services/` contains all 18 service implementations; controllers
+**Evidence**: `DataSpark.Core/Services/` contains all 18 service implementations; controllers
 contain only thin delegation to Core services.
 
 ### II. Testing Standards (MANDATORY)
 
-All production code in `Sql2Csv.Core` MUST have corresponding tests in `Sql2Csv.Tests`.
+All production code in `DataSpark.Core` MUST have corresponding tests in `DataSpark.Tests`.
 The CI pipeline MUST enforce an 80% minimum coverage floor.
 
 - (MUST) Test framework: MSTest with `[TestMethod]`, `[TestClass]`, `[TestInitialize]`, `[TestCleanup]`
@@ -29,19 +29,19 @@ The CI pipeline MUST enforce an 80% minimum coverage floor.
 - (MUST) Test naming: `Method_Scenario_ExpectedBehavior` (e.g. `Constructor_WithNullLogger_ShouldThrowArgumentNullException`)
 - (MUST) SQLite-dependent tests MUST extend `DatabaseTestBase` for consistent setup/teardown
 - (MUST) CI `MIN_COVERAGE` must be set to `80` or higher in `ci.yml`
-- (SHOULD) Integration tests live in `Sql2Csv.Tests/Integration/`; unit tests in subdirectories mirroring `Sql2Csv.Core`
+- (SHOULD) Integration tests live in `DataSpark.Tests/Integration/`; unit tests in subdirectories mirroring `DataSpark.Core`
 
 **Evidence**: 126 test methods across 21 test files; `DatabaseTestBase` used by all DB-touching tests.
 
 ### III. Async/Await Discipline (MANDATORY)
 
-All I/O operations MUST be async. Library code in `Sql2Csv.Core` MUST use `ConfigureAwait(false)`
+All I/O operations MUST be async. Library code in `DataSpark.Core` MUST use `ConfigureAwait(false)`
 on every `await` expression to prevent deadlocks when consumed from synchronous contexts.
 
 - (MUST) All database, file, and network I/O must use `async`/`await` — never `.Result` or `.Wait()`
-- (MUST) Every `await` in `Sql2Csv.Core` MUST append `.ConfigureAwait(false)`
+- (MUST) Every `await` in `DataSpark.Core` MUST append `.ConfigureAwait(false)`
 - (MUST) All public async methods MUST accept `CancellationToken cancellationToken = default`
-- (MAY) `ConfigureAwait(false)` is optional in `sql2csv.web` and `sql2csv.console` (ASP.NET Core and hosted services have no sync context)
+- (MAY) `ConfigureAwait(false)` is optional in `DataSpark.Web` and `DataSpark.Console` (ASP.NET Core and hosted services have no sync context)
 
 **Evidence**: All 18+ Core services use `await using`; cancellation tokens present throughout.
 
@@ -71,7 +71,7 @@ null-reference exceptions and silent quality degradation.
 - (SHOULD) Prefer `sealed` on service implementation classes that are not designed for inheritance
 
 **Evidence**: All 4 projects have `<Nullable>enable</Nullable>`; `TreatWarningsAsErrors` currently
-missing from `sql2csv.web` and `sql2csv.console`.
+missing from `DataSpark.Web` and `DataSpark.Console`.
 
 ### VI. Database Access — SQL Safety (MANDATORY)
 
@@ -94,7 +94,7 @@ MUST use Serilog as the logging provider.
 - (MUST) All classes that log MUST receive `ILogger<T>` via constructor injection — never `LoggerFactory.Create` or static loggers
 - (MUST) Log messages MUST use structured templates: `_logger.LogInformation("Found {Count} tables", count)` — never string interpolation in log calls
 - (MUST) Log levels: `LogInformation` for normal flow, `LogWarning` for recoverable issues, `LogError` for exceptions with the exception as first argument
-- (MUST) `sql2csv.web` MUST configure Serilog via `UseSerilog()` in `Program.cs`
+- (MUST) `DataSpark.Web` MUST configure Serilog via `UseSerilog()` in `Program.cs`
 - (MUST NOT) `Console.Write*` or `Debug.Write*` in production code — use `ILogger` exclusively
 
 **Evidence**: All 18+ Core services use `ILogger<T>` with structured templates; `Serilog.AspNetCore`

.documentation/specs/001-dataspark-consolidation/checklists/requirements.md

@@ -0,0 +1,50 @@
+# Specification Quality Checklist: DataSpark Platform Consolidation
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-03-30
+**Updated**: 2026-03-30 (post-clarification)
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+**Notes**: The spec references "C# DTOs" and "SQLite" by name — these are acceptable because they are the *product features themselves* (the tool generates C# code and works with SQLite databases), not implementation choices. The spec does not prescribe HOW to build these features internally.
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Clarification Results (5 questions asked, 5 answered)
+
+- [x] Q1: DataSpark.Web removal → Absorbed into DataSpark.Web (FR-041 updated)
+- [x] Q2: API authentication → API key auth required (FR-050 added)
+- [x] Q3: Sample dataset mutability → Read-only (FR-006 updated)
+- [x] Q4: Repository rename scope → Full rename including GitHub repo (FR-041 updated)
+- [x] Q5: ML features scope → Deferred to future release (Assumptions updated)
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+- [x] All clarifications integrated into relevant spec sections
+
+## Notes
+
+- All items pass. The specification is ready for `/speckit.plan`.
+- The spec covers 11 user stories across 3 priority tiers (P1: 2, P2: 3, P3: 6), 50 functional requirements, 7 key entities, 14 success criteria, 7 edge cases, and 5 clarifications.
+- The Assumptions section clearly documents all technical decisions, scope boundaries, and deferrals (ML features).
+- SC-001 through SC-014 are all measurable with specific numeric targets or observable conditions.
+- No [NEEDS CLARIFICATION] markers were needed — all decisions resolved via clarification questions or reasonable defaults.

.documentation/specs/001-dataspark-consolidation/contracts/README.md

@@ -0,0 +1,18 @@
+# API Contracts: DataSpark Platform
+
+**Branch**: `001-dataspark-consolidation` | **Date**: 2026-03-30
+
+This directory defines public interface contracts for the DataSpark platform.
+
+## Contract Types
+
+| File | Interface Type | Consumers |
+|------|---------------|-----------|
+| [web-api.md](web-api.md) | RESTful HTTP API | External clients, scripts, dashboards |
+| [cli.md](cli.md) | Command-line interface | Developers, CI/CD pipelines |
+
+## Authentication
+
+All REST API endpoints require API key authentication via the `X-Api-Key` header.
+Web UI routes use standard cookie-based session + CSRF tokens.
+CLI operates locally with no authentication.