Skip to content

Phase 3 Profiling and RUM

Jump to bottom
Ivan P edited this page Aug 28, 2026 · 7 revisions

Each service is an independent unit of work. Pyroscope applies to the 8 deployed services. Faro applies to the 5 with browser-facing frontends, plus credo-ts through the services that consume it.

Each service has separate issues for code changes and Helm chart changes.

Total estimated effort: ~27h (includes infrastructure validation) Prerequisite: Phase 2 complete; Pyroscope and Alloy faro.receiver deployed to dev

Issue: Deploy Grafana Pyroscope

Labels: phase-3 infrastructure profiling Estimated Effort: ~1.5 days

https://github.com/bcgov/DITP-DevOps/issues/339

Pyroscope Continuous Profiling

acapy-agent

Issue: acapy-agent — Pyroscope Profiling (Code)

Labels: phase-3 profiling python acapy Estimated Effort: ~1h

Description: Add pyroscope-io to continuously profile CPU and memory usage in acapy-agent.

Requirements:

  • Add pyroscope-io to dependencies
  • Call pyroscope.configure(application_name="acapy-agent", server_address=..., tags={"version": ...}) in acapy_agent/__main__.py inside the main entry loop

Acceptance Criteria:

  • Profiling data appears in Grafana Pyroscope for acapy-agent
  • CPU flame graphs visible and navigable
  • No measurable performance regression

Issue: acapy-agent — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling acapy Estimated Effort: ~15min

Description: Add PYROSCOPE_SERVER_ADDRESS to the acapy-agent deployment.

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS env var present in deployed pod
  • Profiling data visible in Pyroscope

acapy-vc-authn-oidc

Issue: acapy-vc-authn-oidc — Pyroscope Profiling (Code)

Labels: phase-3 profiling python vc-authn Estimated Effort: ~1h

Requirements: Add pyroscope-io; configure in FastAPI startup before the app begins serving requests.

Acceptance Criteria:

  • Profiling data for vc-authn-oidc-controller visible in Pyroscope
  • No measurable performance regression

Issue: acapy-vc-authn-oidc — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling vc-authn Estimated Effort: ~15min

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS env var present in deployed pod

traction

Issue: traction — Pyroscope Profiling (Code)

Labels: phase-3 profiling nodejs python traction Estimated Effort: ~2h

Requirements:

  • tenant-ui: Add @pyroscope/nodejs; initialize via Pyroscope.init() and Pyroscope.start() in tracing.ts
  • ACA-Py plugin: Add pyroscope-io; call pyroscope.configure() at startup

Acceptance Criteria:

  • Profiling data for traction-tenant-ui and traction-acapy visible in Pyroscope
  • No measurable performance regression in either component

Issue: traction — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling traction Estimated Effort: ~15min

Requirements: Add PYROSCOPE_SERVER_ADDRESS to both tenant-ui and ACA-Py deployments.

Acceptance Criteria:

  • Env var present in both pods

acapy-endorser-service

Issue: acapy-endorser-service — Pyroscope Profiling (Code)

Labels: phase-3 profiling python endorser Estimated Effort: ~1h

Requirements: Add pyroscope-io; configure via pyroscope.configure() early in main.py using env vars.

Acceptance Criteria:

  • Profiling data for acapy-endorser-service visible in Pyroscope

Issue: acapy-endorser-service — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling endorser Estimated Effort: ~15min

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS present in deployed pod

didwebvh-server-py

Issue: didwebvh-server-py — Pyroscope Profiling (Code)

Labels: phase-3 profiling python didwebvh Estimated Effort: ~1h

Requirements: Add pyroscope-io to server/pyproject.toml; configure in server/main.py before uvicorn.run.

Acceptance Criteria:

  • Profiling data for didwebvh-server visible in Pyroscope

Issue: didwebvh-server-py — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling didwebvh Estimated Effort: ~15min

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS present in deployed pod

credo-ts

Issue: credo-ts — Pyroscope Profiling (Code)

Labels: phase-3 profiling typescript credo Estimated Effort: ~1h

Requirements: Add @pyroscope/nodejs; initialize early in the SDK wrapper before the OTel SDK setup using env vars.

Acceptance Criteria:

  • Profiling data for credo-agent visible in Pyroscope

Issue: credo-ts consuming service — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling credo Estimated Effort: ~15min

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS present in consuming service pod

didcomm-mediator-credo

Issue: didcomm-mediator-credo — Pyroscope Profiling (Code)

Labels: phase-3 profiling typescript mediator Estimated Effort: ~1h

Requirements: Add @grafana/pyroscope-nodejs; call Pyroscope.init() and Pyroscope.start() in apps/mediator/instrumentation.js before OTel SDK initialization.

Acceptance Criteria:

  • Profiling data for didcomm-mediator-credo visible in Pyroscope

Issue: didcomm-mediator-credo — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling mediator Estimated Effort: ~15min

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS present in deployed pod

bc-wallet-demo

Issue: bc-wallet-demo — Pyroscope Profiling (Code)

Labels: phase-3 profiling typescript bc-wallet Estimated Effort: ~2h

Requirements: Add @pyroscope/nodejs to server workspace; initialize via Pyroscope.init({ serverAddress, appName }) in server/src/index.ts at startup.

Acceptance Criteria:

  • Profiling data for bc-wallet-demo-server visible in Pyroscope

Issue: bc-wallet-demo — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling bc-wallet Estimated Effort: ~15min

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS present in deployed pod

digital-trust-common-service

Issue: digital-trust-common-service — Pyroscope Profiling (Code)

Labels: phase-3 profiling typescript nestjs common-service Estimated Effort: ~2h

Requirements: Add @pyroscope/nodejs; initialize with Pyroscope.init({ serverAddress, appName }) and Pyroscope.start() in the same telemetry module as the OTel SDK (libs/common/src/telemetry/), gated on PYROSCOPE_ENABLED. The Worker takes its own appName once dist/worker.js exists.

Acceptance Criteria:

  • Profiling data for digital-trust-common-service visible in Pyroscope
  • Worker profiles reported separately from the API once the Worker is enabled

Issue: digital-trust-common-service — Pyroscope Helm Chart Updates

Labels: phase-3 helm profiling common-service Estimated Effort: ~30min

Requirements: Add the shared pyroscope: block to values.yaml; render PYROSCOPE_SERVER_ADDRESS on both the API and Worker Deployments; extend the OTLP egress NetworkPolicy to cover port 12347, since egress on this chart is implicit-deny.

Acceptance Criteria:

  • PYROSCOPE_SERVER_ADDRESS present in both deployed pods
  • Egress to Alloy on 12347 allowed by NetworkPolicy

Faro Real User Monitoring

Applies to: traction, acapy-vc-authn-oidc, didwebvh-server-py, credo-ts, bc-wallet-demo, digital-trust-common-service.

Each service requires a backend proxy route (code) and opt-in Helm chart values. See Architecture & Infrastructure for the Faro collection pattern and Helm values design.

traction

Issue: traction — Faro RUM (Code)

Labels: phase-3 faro rum nodejs traction Estimated Effort: ~3h

Requirements:

  • Add @grafana/faro-web-sdk, @grafana/faro-web-tracing to the frontend workspace
  • Add Express proxy route (POST /faro) in tenant-ui that forwards browser payloads to http://monitoring-collector-alloy:12347
  • Initialize Faro in Vue's main.ts inside loadApp(), pointing collectorUrl at the app's own /faro path

Acceptance Criteria:

  • Browser errors and Web Vitals appear in Loki as structured log entries
  • Browser-side traces (XHR/Fetch spans) appear in Tempo linked to backend spans
  • Proxy route forwards correctly without CORS errors in browser console

Issue: traction — Faro Helm Chart Updates

Labels: phase-3 helm faro traction Estimated Effort: ~30min

Requirements: Add faro.enabled, faro.proxy.enabled, faro.proxy.upstreamUrl, faro.collectorUrl to chart values for both tenant-ui and ACA-Py deployments.

Acceptance Criteria:

  • faro.enabled: false by default — no change to existing deployments
  • When enabled, faro.collectorUrl is injected into browser bundle
  • Proxy route activated when faro.proxy.enabled: true

acapy-vc-authn-oidc

Issue: acapy-vc-authn-oidc — Faro RUM (Code)

Labels: phase-3 faro rum python vc-authn Estimated Effort: ~4h

Requirements:

  • Add FastAPI proxy route (POST /faro) forwarding browser payloads to http://monitoring-collector-alloy:12347
  • Create Nuxt 3 client plugin faro.client.ts pointing collectorUrl at the app's own /faro path
  • Inject Faro CDN script tag into Jinja2 templates (e.g. verified_credentials.html)

Acceptance Criteria:

  • Browser errors and Web Vitals from both Nuxt 3 and Jinja2 surfaces appear in Loki
  • Browser traces appear in Tempo
  • No CORS errors in browser console

Issue: acapy-vc-authn-oidc — Faro Helm Chart Updates

Labels: phase-3 helm faro vc-authn Estimated Effort: ~30min

Requirements: Add faro.enabled, faro.proxy.enabled, faro.proxy.upstreamUrl, faro.collectorUrl to chart values.

Acceptance Criteria:

  • faro.enabled: false by default
  • Faro collector URL injected when enabled

didwebvh-server-py

Issue: didwebvh-server-py — Faro RUM (Code)

Labels: phase-3 faro rum python didwebvh Estimated Effort: ~2h

Requirements:

  • Add FastAPI proxy route (POST /faro) forwarding browser payloads to http://monitoring-collector-alloy:12347
  • Inject Faro Web SDK initialization script tag into server/app/templates/components/head.jinja
  • Pass collectorUrl via server/config.py from environment

Acceptance Criteria:

  • Web Explorer browser events appear in Loki
  • Proxy route forwards without CORS errors

Issue: didwebvh-server-py — Faro Helm Chart Updates

Labels: phase-3 helm faro didwebvh Estimated Effort: ~30min

Requirements: Add faro.enabled, faro.proxy.enabled, faro.proxy.upstreamUrl, faro.collectorUrl to chart values.

Acceptance Criteria:

  • faro.enabled: false by default

credo-ts

Issue: credo-ts — Faro RUM (Code)

Labels: phase-3 faro rum typescript credo Estimated Effort: ~5h

Requirements:

  • Standard Faro Web SDK configuration inside React or Vue consumer roots
  • instrumentations: [...getWebInstrumentations(), new TracingInstrumentation()]

Note: Proxy route responsibility falls on the consuming application's backend, not credo-ts itself.

Acceptance Criteria:

  • Browser events from consumer app appear in Loki
  • Browser traces appear in Tempo

Issue: credo-ts consuming service — Faro Helm Chart Updates

Labels: phase-3 helm faro credo Estimated Effort: ~30min

Requirements: Add faro.* values to the consuming service's Helm chart.

Acceptance Criteria:

  • faro.enabled: false by default

bc-wallet-demo

Issue: bc-wallet-demo — Faro RUM (Code)

Labels: phase-3 faro rum typescript bc-wallet Estimated Effort: ~4h

Requirements:

  • Add Express proxy route (POST /faro) on backend server forwarding to http://monitoring-collector-alloy:12347
  • Add @grafana/faro-web-sdk, @grafana/faro-web-tracing to frontend workspace
  • Initialize Faro in frontend/src/index.tsx before createRoot, pointing collectorUrl at the app's own /faro path

Acceptance Criteria:

  • React app browser errors and Web Vitals appear in Loki
  • Browser XHR/Fetch traces appear in Tempo linked to backend spans
  • No CORS errors in browser console

Issue: bc-wallet-demo — Faro Helm Chart Updates

Labels: phase-3 helm faro bc-wallet Estimated Effort: ~30min

Requirements: Add faro.enabled, faro.proxy.enabled, faro.proxy.upstreamUrl, faro.collectorUrl (injected as build-time args or public/config.json) for the frontend build.

Acceptance Criteria:

  • faro.enabled: false by default
  • Faro collector URL injected into React bundle when enabled

digital-trust-common-service

Issue: digital-trust-common-service — Faro RUM (Code)

Labels: phase-3 faro rum react common-service Estimated Effort: ~3h

Context: The admin UI is a Vite + React SPA at apps/ui (entry apps/ui/src/main.tsx), talking to the API through axios. In local development Caddy serves the SPA and the API on one origin (app.localhost), mirroring the intended production topology (IN-15 / #160). The SPA is not deployed today — the production image builds only the NestJS app and the chart has no UI Deployment — so this work can be written and validated locally, but end-to-end validation in OpenShift is blocked until the SPA is served.

Requirements:

  • Add @grafana/faro-web-sdk and @grafana/faro-web-tracing to apps/ui/package.json
  • Initialize Faro in apps/ui/src/main.tsx before createRoot(...).render(...), reading the collector URL from a build-time VITE_FARO_COLLECTOR_URL and no-opping when it is empty
  • Add a NestJS proxy route (POST /faro) in the API that forwards browser payloads to http://monitoring-collector-alloy:12347, so no Alloy endpoint is ever exposed publicly — the same-origin topology means the SPA posts to its own /faro path
  • Exclude the proxy route from the auth guards and from the audit interceptor
  • Extend the chart's OTLP egress NetworkPolicy to allow the API pod to reach Alloy on 12347

Acceptance Criteria:

  • Browser errors and Web Vitals appear in Loki as structured log entries
  • Browser-side traces (XHR/fetch spans) appear in Tempo, linked to backend API spans
  • Proxy route forwards without CORS errors in the browser console
  • With VITE_FARO_COLLECTOR_URL unset the SPA loads and makes no Faro network calls

Issue: digital-trust-common-service — Faro Helm Chart Updates

Labels: phase-3 helm faro common-service Estimated Effort: ~30min

Blocked on: the SPA being built into the image and served in OpenShift (IN-15 / #160). Until then there is no deployed frontend to configure.

Requirements: Add the shared faro: block (enabled, collectorUrl, proxy.enabled, proxy.path, proxy.upstreamUrl) to values.yaml; inject the collector URL into the SPA build; activate the proxy route when faro.proxy.enabled is true.

Acceptance Criteria:

  • faro.enabled: false by default — no change to existing deployments
  • Faro collector URL injected into the React bundle when enabled
  • Proxy route activated when faro.proxy.enabled: true

Issue: Phase 3 Infrastructure Validation (dev only)

Estimated Effort: ~½ day

Description

Validate that Pyroscope and Alloy's Faro receiver are correctly deployed and accepting data in ca7f8f-dev before beginning per-service Phase 3 onboarding. This is the go/no-go gate for Phase 3 service work.

Unlike Phase 1, there is no synthetic profiling data generator equivalent to telemetrygen. Pyroscope is validated via health check and API reachability. The Faro receiver is validated by sending a test payload directly to Alloy — no browser or instrumented service required. Per-service data flow validation is covered by each service's acceptance criteria.

Requirements

  • Confirm Pyroscope is running and healthy in ca7f8f-dev (query the /ready endpoint)
  • Confirm Pyroscope is reachable from within the namespace (e.g. from a debug pod)
  • Send a minimal test payload to Alloy's Faro receiver endpoint (http://monitoring-collector-alloy:12347/collect) and confirm it is accepted (HTTP 200)
  • Confirm the test Faro payload appears as a log entry in Loki (dev)
  • Resolve any RBAC, network policy, or Alloy pipeline issues before proceeding

Expected Outcome

Pyroscope is running and reachable from within ca7f8f-dev. Alloy's Faro receiver is accepting payloads and routing them to Loki. The infrastructure is confirmed ready for per-service Phase 3 onboarding.

Acceptance Criteria

  • Pyroscope /ready endpoint returns healthy in ca7f8f-dev
  • Pyroscope reachable from within the namespace on its configured port
  • Alloy faro.receiver accepts a test POST to /collect without error
  • Test Faro payload visible as a log entry in Grafana → Loki (dev)
  • Any network policy or RBAC issues identified and resolved

Clone this wiki locally