codefrydev · PrashantUnity · Sep 2, 2026 · Sep 2, 2026 · Sep 2, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -115,5 +115,5 @@ jobs:
           dotnet tool install -g NSwag.ConsoleCore
           export PATH="$PATH:$HOME/.dotnet/tools"
           (cd services/Bff && nswag run nswag.json)
-          git diff --exit-code services/Bff/src/Bff.Application/Generated/FastApiClient.g.cs \
+          git diff --exit-code services/Bff/src/Bff.Api/Application/Generated/FastApiClient.g.cs \
             || (echo "::error::FastApiClient.g.cs is stale — run services/Bff/generate-client.sh and commit." && exit 1)
diff --git a/AGENT.md b/AGENT.md
@@ -13,14 +13,13 @@ Developer reference for agents and contributors. User-facing overview: [README.m
 **Key paths**
 
 - `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `analysis/`, `ai_service_client.py`, `tools/`
-- `services/Bff/` -- .NET BFF (auth, CORS, `/api/*` proxy to FastAPI + AiService + Data)
+- `services/Bff/` -- .NET BFF (auth, CORS, `/api/*` proxy to FastAPI + CoreService + AiService; port 8090)
+- `services/CoreService/` -- .NET core service (report build & orchestration, data queries & PDF/Excel export, Google/Bing integrations; port 8094). See `services/CoreService/README.md`
 - `services/AiService/` -- .NET AI (chat, secrets, LLM config, MCP, enrichment; port 8092)
-- `services/Data/` -- .NET data service (report payloads, portfolio, issue status, typed config, PDF/Excel/CSV/JSON/sitemap export; port 8091)
-- `services/ReportService/` -- .NET report build + pipeline orchestration (port 8094). Worker runs crawl+Lighthouse; report via `REPORT_SERVICE_URL`
 - `web/src/` -- React SPA (`AppRoutes.tsx`, `views/`, `components/`); pipeline UI: `PipelineRunnerFab`, `PipelineContext`
 - `services/Schema/` -- EF Core schema migrations (schema owner)
 
-**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Data `:8091`, AiService `:8092`, ReportService `:8094`, FastAPI `:8096`, BFF `:8090`, Vite `:3000`; default `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling`). See `scripts/local-run.sh`. **Local tests:** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%) plus web and .NET checks — mirrors CI; Docker CI is separate (see `.github/workflows/ci.yml`). `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
+**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, CoreService `:8094`, AiService `:8092`, FastAPI `:8096`, BFF `:8090`, Vite `:3000`; default `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling`). See `scripts/local-run.sh`. **Local tests:** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%) plus web and .NET checks — mirrors CI; Docker CI is separate (see `.github/workflows/ci.yml`). `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
 
 **JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: Playwright from `requirements.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`.
 
@@ -32,20 +31,20 @@ Developer reference for agents and contributors. User-facing overview: [README.m
 - **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: local artifacts (Docker: `/data`); settings and API keys live in Postgres.
 - **Pipeline storage** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) lives in **PostgreSQL only**. Deliverables use the Export view, `GET /api/report/export`, or MCP `export_*` tools — not files written by the main pipeline step.
 - **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch. Per-URL raw HTML: `crawl_page_html` table (migration `015`); API `GET/POST /api/crawl/page-html`.
-- **Browser API (BFF):** All `/api/*` routes are served by `services/Bff/`. **FastAPI:** `/api/run`, `/api/jobs/*`, `/api/pipeline-config`, crawl, integrations (OAuth reads), properties, content drafts, etc. **ReportService:** report build + full-audit orchestration (internal; worker uses `REPORT_SERVICE_URL`). **IntegrationsService:** Google/Bing OAuth and fetch (browser + internal report enrichment). **AiService:** `/api/chat` (SSE), `/api/llm-settings`, `/api/secrets`, `/api/ollama/status`, etc. **Data:** report payload reads, portfolio, issue status, saved filters, `/api/pipeline-settings`, `/api/ui-preferences`, `/api/client-preferences` (see `DATA_ROUTES`), and PDF/workbook export. `PipelineRunnerFab` saves pipeline config (FastAPI) and LLM state (`PUT /api/llm-settings` → AiService) before each run.
+- **Browser API (BFF):** All `/api/*` routes are served by `services/Bff/`. **FastAPI:** crawl, pipeline worker CLI, audit-tool CLI bridge. **CoreService:** report build + full-audit orchestration, data reads (portfolio, payload, issue status, saved filters, typed config), PDF/workbook exports, Google/Bing OAuth and data fetching. **AiService:** `/api/chat` (SSE), `/api/llm-settings`, `/api/secrets`, `/api/ollama/status`, MCP catalog. `PipelineRunnerFab` saves pipeline config (FastAPI) and LLM state (`PUT /api/llm-settings` → AiService) before each run.
 - **MCP:** AiService (.NET) — stdio host or HTTP at `/mcp` when `WP_MCP_HTTP=1` on `:8092`. Configure at **`/mcp`** in the web UI. See `docs/MCP.md` and [services/AiService/README.md](services/AiService/README.md).
 - **AI Chat UI:** `/chat` — property-scoped chat with saved sessions (`chat_sessions`, `chat_messages`; migration `012_chat_sessions`).
 - **Job store:** PostgreSQL `pipeline_jobs` (FastAPI); live job status via `/api/jobs/*` through the BFF.
-- **Docker:** Root `Dockerfile` (Python backend); `web/Dockerfile` (Vite SPA + nginx); `docker-compose.yml` (postgres + fastapi + worker + report + integrations + ai + data + bff + web); **`docker-compose.prod.yml`** (production + optional MCP profile mapping host `:8000` → AiService `:8092`); **`docker-compose.pull.yml`** for pre-built images (`BACKEND_IMAGE`, `WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`**
+- **Docker:** Root `Dockerfile` (Python backend); `web/Dockerfile` (Vite SPA + nginx); `docker-compose.yml` (postgres + fastapi + core + ai + bff + web); **`docker-compose.prod.yml`** (production + optional MCP profile mapping host `:8000` → AiService `:8092`); **`docker-compose.pull.yml`** for pre-built images (`BACKEND_IMAGE`, `WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`**
 
 **Where to edit**
 
 | Task | Where |
 |------|--------|
 | Crawl | `crawl/crawler.py`, `crawl/fetchers/`, `parsing/links.py` (`normalize_link` preserves trailing slashes from `href`) |
-| Report (native build) | `services/ReportService/src/ReportService.Application/Build/` |
-| Report (Python bridge) | `reporting/builder.py`, `reporting/categories/` |
-| PDF / workbook export | `services/Data/src/Data.Rendering/` (rendering), `Data.Api/Controllers/ReportExportController.cs`; BFF routes `/api/report/export` and `/api/report/export-workbook` to Data |
+| Report (native build) | `services/CoreService/src/CoreService.Api/Application/Build/` |
+| PDF / workbook export | `services/CoreService/src/CoreService.Api/Rendering/`, `Controllers/ReportExportController.cs` |
+| Integrations (Google/Bing) | `services/CoreService/src/CoreService.Api/Providers/Google/`, `IntegrationsApplication/` |
 | DB schema | `services/Schema/src/Schema.Model/` |
 | Local analysis | `analysis/local.py`, `requirements.txt` |
 | AI insights (LLM) | `services/AiService/` (browser-facing + MCP + native audit tools), `ai_service_client.py` (worker), `llm_config.py` (typed loader) |

diff --git a/AGENTS.md b/AGENTS.md
@@ -13,11 +13,9 @@ This file is the canonical entry point for agents. For full detail see [AGENT.md
 - `src/website_profiling/` — Python crawl/Lighthouse engine
   - `crawl/`, `lighthouse/`, `worker/`, `api/` (jobs, crawl, internal bridges), `reporting/` (bridge-only until C# port completes)
 - `web/` — Vite + React SPA (static nginx in prod); browser calls `services/Bff/` for all `/api/*`
-- `services/Bff/` — .NET BFF (auth, CORS, proxy to FastAPI + ReportService + IntegrationsService + Data + AiService)
-- `services/Data/` — .NET data service (report payloads, portfolio, issue status, filters, typed config, PDF/Excel/CSV/JSON/sitemap export; port 8091). Export profiles: `executive|standard|full|premium`. Env: `REPORT_API_URL` (branding lookup).
-- `services/ReportService/` — .NET report build + pipeline orchestration (port 8094). See [services/ReportService/README.md](services/ReportService/README.md)
+- `services/Bff/` — .NET BFF (auth, CORS, proxy to FastAPI + CoreService + AiService; port 8090)
+- `services/CoreService/` — .NET core service (report build & orchestration, data queries & PDF/Excel exports, Google/Bing integrations; port 8094). See [services/CoreService/README.md](services/CoreService/README.md)
 - `services/AiService/` — .NET AI service (Microsoft.Extensions.AI, chat, enrichment, MCP, **secrets/llm-settings writes**; port 8092). See [services/AiService/README.md](services/AiService/README.md)
-- `services/IntegrationsService/` — .NET Google/Bing integrations (GSC/GA4 fetch, OAuth, page-live, keyword reads; port 8093). See [services/IntegrationsService/README.md](services/IntegrationsService/README.md)
 - `services/Schema/` — EF Core DB migrations (schema owner)
 - `services/WebsiteProfiling.slnx` — unified .NET solution (all services + shared libs)
 - `docs/` — documentation index
@@ -26,7 +24,7 @@ This file is the canonical entry point for agents. For full detail see [AGENT.md
 **Run / dev**
 
 ```bash
-./local-run          # Start Postgres + Data + AiService + ReportService + IntegrationsService + worker + FastAPI + BFF + Vite
+./local-run          # Start Postgres + CoreService + AiService + worker + FastAPI + BFF + Vite
 ./local-test         # Python + web + .NET tests (CI parity; dotnet: services/WebsiteProfiling.slnx)
 python -m src        # Run audit pipeline
 # MCP: AiService stdio/HTTP — see services/AiService/README.md and docs/MCP.md
@@ -41,7 +39,9 @@ python -m src        # Run audit pipeline
 | Task | Where |
 |------|-------|
 | Crawl | `src/website_profiling/crawl/` |
-| Report (native C#) | `services/ReportService/src/ReportService.Application/Build/` |
+| Report (native C#) | `services/CoreService/src/CoreService.Api/Application/Build/` |
+| Data reads & exports | `services/CoreService/src/CoreService.Api/Rendering/`, `DataApplication/` |
+| Integrations (Google/Bing) | `services/CoreService/src/CoreService.Api/Providers/Google/`, `IntegrationsApplication/` |
 | Report (Python bridge) | `src/website_profiling/reporting/` |
 | Typed settings / config | `config/typed_config_manifest.json`, `src/website_profiling/db/typed_config/` |
 | GEO / AEO / Agent readiness | `src/website_profiling/tools/audit_tools/geo/geo_tools.py`, `geo/agent_readiness.py` |

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -39,7 +39,7 @@ CI also runs a **Docker** job (image build, browser pytest in container, compose
 
 When adding tools coverage tests, register new files in `scripts/local-test.sh`, `scripts/local-test.ps1`, and `.github/workflows/ci.yml` (see [AGENT.md](AGENT.md)).
 
-**.NET DI validation:** Every API service (`AiService`, `Bff`, `Data`, `IntegrationsService`, `ReportService`) enables `ValidateOnBuild` / `ValidateScopes` in `Program.cs` and includes `ServiceRegistrationValidationTests` — a `WebApplicationFactory` smoke test that builds the real host graph. Add the same when introducing a new .NET API project.
+**.NET DI validation:** Every API service (`AiService`, `Bff`, `CoreService`) enables `ValidateOnBuild` / `ValidateScopes` in `Program.cs` and includes `ServiceRegistrationValidationTests` — a `WebApplicationFactory` smoke test that builds the real host graph. Add the same when introducing a new .NET API project.
 
 ## How to contribute