Sonda

Your agent's eyes into production.

Sonda is a self-hosted, read-only gateway that gives AI agents one credential and one service-centric interface to production telemetry and, where granted, to application data. It stores neither: it queries the sources on demand, restricts every query to the services an agent may see, and returns compact responses with links back to the source.

Source	Capabilities
Sentry	errors, releases, traces
Loki	logs
Prometheus (and Mimir, Thanos, VictoriaMetrics, Grafana Cloud)	metrics, alerts
Cloudflare Workers Observability and GraphQL Analytics	logs, metrics, traces
AWS CloudWatch Logs Insights and metrics	logs, metrics
Grafana Tempo	traces
Kubernetes pods, ReplicaSets and events	workloads
Grafana managed alerting	alerts
Vercel runtime logs	logs
Contember Content API	data (read-only GraphQL)

agent ──sonda CLI──► sonda-server ──► Sentry / Loki / Prometheus / Cloudflare / CloudWatch / Tempo / Kubernetes / Grafana / Vercel / Contember
          one token     authz, query scoping, grouping, limits, audit

• One token per agent. Provider credentials stay on the server.
• Service-centric. Agents ask for engine-api in production; Sonda knows the Sentry project, the Loki selector, the Worker script, the log groups and the Tempo service name.
• Enforced scoping. Native queries are allowed in each backend's own dialect (LogQL, PromQL, TraceQL, Logs Insights, Cloudflare query JSON, Vercel KQL). Sonda adds the service scope to every query, so a query cannot reach another service. describe tells the agent which dialect a capability takes.
• Read-only data. The data capability runs GraphQL against a Contember project with a read-only API key. Sonda refuses a key that can write there, and checks every query: no mutations, a limit on every list, denied fields hidden also from filters and ordering.
• Compact answers. Log search returns an exact count (an estimate for Cloudflare, capped at 10,000 for Vercel) and a sample grouped into patterns; metrics return summaries; a trace is a span tree. Large results stay on the server behind result handles.
• Audited and limited. Every request is logged; time range, lines, response size, timeouts, rate and concurrency are limited per identity.

Sonda does not reason about incidents. The agent does; Sonda supplies the evidence.

Quick start

go build -o . ./cmd/...            # builds sonda-server and sonda

cp sonda.example.yaml sonda.yaml   # configure sources and services
export SONDA_ADMIN_TOKEN=$(openssl rand -hex 32)
./sonda-server check -c sonda.yaml
./sonda-server serve -c sonda.yaml

# Open http://localhost:8080/admin, sign in with the admin token, create an identity
# and copy its token.
./sonda auth login --url http://localhost:8080   # paste the token; saved to ~/.config/sonda/config.yaml (0600)
./sonda services
./sonda overview engine-api -e production

Requires Go 1.26.5 or newer. A container image builds from the Dockerfile.

Credentials on a local machine

Sonda reads credentials from the environment and does not load .env files itself. Keep them out of sonda.yaml and out of git (.env is ignored):

# .env
SONDA_ADMIN_TOKEN=...
SENTRY_TOKEN=...

set -a; source .env; set +a
./sonda-server serve -c sonda.yaml

With a password manager the values do not touch the disk at all, e.g. 1Password: op run --env-file=.env -- ./sonda-server serve -c sonda.yaml with op:// references in .env. AWS sources use your local profile from ~/.aws.

For agents

skills/sonda/SKILL.md teaches an agent the investigation flow and the commands. Install it as a skill for Claude Code, or paste it into the agent's instructions.

Command	Purpose
sonda services, sonda describe <svc>	What the token can access; backend and dialect per capability
sonda overview <svc>	Releases, top errors, signal summaries, pod health, active alerts
sonda errors <svc>, sonda error <svc> <id>, sonda releases <svc>	Sentry
sonda logs search/query/fields <svc>, sonda result <id>	Logs
sonda metrics list/query <svc>, sonda signal <svc> <name>	Metrics
sonda traces search/query <svc>, sonda trace <svc> <id>	Traces
sonda workloads <svc>	Kubernetes pods, rollouts and events
sonda alerts <svc>	Active alerts of Prometheus or Grafana alerting rules
sonda data schema/query <svc>	Application data (read-only GraphQL)

The CLI prints JSON when its output is piped and compact text in a terminal.

Configuration

Sources, projects with their services, and server limits are in sonda.yaml (see sonda.example.yaml). A project groups the services of one product, across platforms. The server reads it at startup. Agent identities are in the database and are managed in the admin UI. The parts that matter for security:

• Bindings are the scoping boundary. Every query of the service is restricted to them:
  • Loki and Prometheus selector: must contain an = or =~ matcher that rejects the empty value, and the labels must really identify the service in your data.
  • Cloudflare script: the Worker script name. Metrics bindings can also map the service's Durable Object namespaces, D1 databases, queues and R2 buckets by name to their IDs (durable_object_namespaces, d1_databases, queues, r2_buckets).
  • CloudWatch log_groups: 1 to 50 exact log group names, no patterns or ARNs.
  • CloudWatch metrics namespace and dimensions: 1 to 10 dimensions.
  • Tempo service: the resource.service.name of the service's spans.
  • Sentry errors project and environment, and optionally query: Sentry search terms ANDed with every issue and event search, for services that share a project, e.g. server_name:engine-e0-*. Issue counts then cover only matching events in the range; the totals of a single issue and the releases stay project-wide.
  • Sentry traces project and environment, as for errors. Sentry has no project-scoped span search, so Sonda checks the project and environment of every span it returns.
  • Kubernetes namespace and labels: exact labels the service's pods and ReplicaSets carry.
  • Vercel project and environment: the project ID (prj_…, not the name) and the Vercel environment, e.g. production or preview.
  • Alerts selector: a Prometheus selector over the alert labels, with the same rule as above. The alerts API returns the alerts of every service, so this filter in Sonda is the scope.
  • Contember project and stage, with an optional entities allow list and deny_fields. The API key's role is the main boundary (see below).
• Grants give an identity capabilities (errors, logs, metrics, traces, workloads, alerts, data) in project environments, e.g. logs in production of project engine. A grant covers every service of the project that configures the capability in that environment, including services added to the project later. data is the exception: it is granted per service, e.g. data in production of service shop-api, and opens only that service's data binding; it is never implied by the others. Anything else is not found. A traces grant also decides which spans of a trace an identity sees in full; spans of other services are reduced to service, timing and status.
• Provider credentials come from environment variables or, for AWS, the default credential chain or Sonda's own identity (below). ${NAME} reads the variable NAME; when it is unset or empty and NAME_FILE is set, the value is read from that file (Docker and Kubernetes secrets). A token, password, admin token, credential header (Authorization, Cookie, X-Api-Key…), URL password or password in server.database written directly into the config file works, but check and serve warn about it. grafana.url must not contain credentials or a query, because it is copied into links returned to agents. Use read-only tokens and roles; the Security section of docs/design.md lists the Cloudflare permission and the AWS IAM actions.
• Admin token (admin.token) can create tokens for every project. Keep it secret and expose /admin only on a trusted network.

Sources and bindings for Cloudflare, AWS, Tempo, Kubernetes, Grafana alerting and Vercel:

sources:
  cloudflare:
    type: cloudflare
    account_id: "${CF_ACCOUNT_ID}"
    auth: { type: bearer, token: "${CF_API_TOKEN}" }  # url defaults to https://api.cloudflare.com/client/v4
  aws-prod:
    type: aws
    region: eu-west-1
    role_arn: "${SONDA_AWS_ROLE_ARN}"  # optional; the default credential chain otherwise
    max_scan_bytes: 10GB               # Logs Insights is billed by scanned bytes
  tempo-prod:
    type: tempo
    url: http://tempo.monitoring.svc:3200
    headers: { X-Scope-OrgID: prod }   # multi-tenant Tempo only
    grafana_datasource_uid: tempo-prod
  k8s-prod:
    type: kubernetes                   # without url: the pod's service account (in-cluster)
    url: https://ABC123.gr7.eu-west-1.eks.amazonaws.com
    auth: { type: bearer, token: "${K8S_TOKEN}" }  # a read-only service account token
    ca_file: /etc/sonda/k8s-prod-ca.crt
  grafana:
    type: grafana                      # Grafana-managed alerts; a prometheus source serves its own rules' alerts
    url: https://grafana.example.com
    auth: { type: bearer, token: "${GRAFANA_TOKEN}" }  # a Viewer service account token
  vercel:
    type: vercel
    team_id: team_abc123               # the team ID, not the slug
    auth: { type: bearer, token: "${VERCEL_TOKEN}" }  # url defaults to https://api.vercel.com

projects:
  web:
    services:
      web-worker:
        environments:
          production:
            logs:    { source: cloudflare, script: web-worker }
            metrics: { source: cloudflare, script: web-worker, d1_databases: { main: 6f1c2b9e-3d4a-4e5f-8a7b-0c1d2e3f4a5b } }
            traces:  { source: cloudflare, script: web-worker }
        signals:
          errors: { metric: errors, by: [status] }  # Cloudflare signals are mappings
          slow_queries: { metric: d1_query_time_p99 }
  engine:
    description: Content engine
    services:
      engine-db:
        environments:
          production:
            logs:    { source: aws-prod, log_groups: [/aws/rds/cluster/engine/postgresql] }
            metrics: { source: aws-prod, namespace: AWS/RDS, dimensions: { DBClusterIdentifier: engine } }
        signals:
          cpu: { metric: CPUUtilization, stat: Average }
      engine-api:
        environments:
          production:
            traces: { source: tempo-prod, service: engine-api }  # resource.service.name
            workloads: { source: k8s-prod, namespace: engine, labels: { app.kubernetes.io/name: engine-api } }
            alerts: { source: grafana, selector: '{service="engine-api"}' }
  site:
    services:
      site-frontend:
        environments:
          production:
            logs: { source: vercel, project: prj_abc123, environment: production }
          preview:
            logs: { source: vercel, project: prj_abc123, environment: preview }

A signal is PromQL for Prometheus and a mapping for Cloudflare or CloudWatch. All metrics environments of a service with signals must use the same source type.

AWS without access keys, and backends in a private EKS cluster

Sonda can be an OIDC issuer of its own identity, so AWS trusts it wherever it runs, without a long-lived access key and without the hosting platform's workload identity. The same AWS credentials sign EKS tokens, which reach a Loki, Prometheus or Tempo inside a cluster through the API server's service proxy; nothing new runs in the private network.

identity:
  issuer: https://sonda.example.com     # Sonda's public URL; serves /.well-known/openid-configuration

sources:
  aws-prod:
    type: aws
    region: eu-west-1
    credentials: identity               # AssumeRoleWithWebIdentity with Sonda's own token
    role_arn: arn:aws:iam::123456789012:role/sonda
  loki-prod:
    type: loki
    url: https://ABC123.gr7.eu-west-1.eks.amazonaws.com/api/v1/namespaces/logging/services/http:loki:3100/proxy
    ca_file: /etc/sonda/eks-prod-ca.crt # certificateAuthority.data of the cluster, decoded
    auth: { type: eks, cluster: prod, aws_source: aws-prod }

• In AWS: an IAM OIDC identity provider for the issuer URL with audience sts.amazonaws.com, and a role that trusts it with conditions on aud (sts.amazonaws.com) and sub (sonda).
• In the cluster: an access entry for the role in a Kubernetes group, and a Role in the backend's namespace with get on services/proxy for resourceNames: ["http:loki:3100"] (get and create for Prometheus).
• The signing key lives in Sonda's database and the issuer URL is the trust anchor; see Security in docs/design.md.

Application data (Contember)

sources:
  shop-content:
    type: contember
    url: https://api.example.com        # engine base URL; Sonda calls /content/{project}/{stage}
    auth: { type: bearer, token: "${SHOP_CONTEMBER_READER_TOKEN}" }

projects:
  shop:
    services:
      shop-api:
        environments:
          production:
            data:
              source: shop-content
              project: shop             # Contember project slug
              stage: live               # default
              entities: [Order, Product] # optional allow list; empty = everything the role reads
              deny_fields: ["*.email", Customer.phone] # a denied relation hides both its sides

limits:
  data_max_rows: 500                    # row budget: list limits, nested lists multiplied

Give the API key its own Contember role with read permissions only, and use its ACL predicates to hide rows agents must not see. Contember then generates no mutations for the key. Create the key with this one membership: Sonda can only see what the key may do in the bound project and stage. It introspects the schema at startup and on every schema refresh (10 minutes), and refuses a key that can write there: serve does not start, and requests fail. Allow-listed entities and deny_fields that do not exist are refused the same way, because a typo would leave a field readable. sonda-server check --online reports both and warns about readable fields that look like personal data. Query results and the audited query text can contain personal data; treat the audit log accordingly.

data grants are stored in their own table, identity_service_grants. An older Sonda ignores that table, so a rollback needs no cleanup: identities keep their other grants, and the data grants apply again after the upgrade.

Admin UI

Set admin.token to enable the UI at /admin. Without it, the UI is not served.

• Identities: create an identity, select its grants in a project environment × capability matrix, and data in a separate service environment matrix, override limits, rotate or revoke its token. Changes apply to the next request; no restart. A token is shown once; Sonda stores only its SHA-256.
• Services: a read-only view of the configuration: projects, their services and environments, what each capability reads, and the sources. It shows no credentials.
• Audit log: every agent request and every admin action, filterable by identity.

The UI is server-rendered with htmx and embedded in sonda-server.

HTTP API

The CLI is a thin client over the API in api/. All endpoints need Authorization: Bearer <token>.

Endpoint	Capability
GET /v1/services, GET /v1/services/{s}	any grant
GET /v1/services/{s}/overview	any granted
GET /v1/services/{s}/errors, /errors/{id}, /releases	errors
POST /v1/services/{s}/logs/search, /logs/query; GET /logs/fields	logs
GET /v1/services/{s}/metrics, /signals/{name}; POST /metrics/query	metrics
POST /v1/services/{s}/traces/search, /traces/query; GET /traces/{id}	traces
GET /v1/services/{s}/data/schema; POST /data/query	data
GET /v1/results/{id}	the identity that created it

Time ranges: env, since (30m, 6h, 2d) or from/to (RFC 3339) as query parameters, or the same fields in POST bodies.

Development

go test ./...
docker compose -f dev/docker-compose.yml up -d   # Loki :13100, Prometheus :19090, Tempo :13200 (OTLP :14318), Postgres :15432, Contember :14000
SONDA_TEST_LOKI_URL=http://localhost:13100 go test ./internal/logql   # LogQL scoping against a real Loki
dev/seed-traces.sh                               # traces of two services in the local Tempo
SONDA_TEST_TEMPO_URL=http://localhost:13200 go test ./internal/traceql   # TraceQL scoping against a real Tempo
SONDA_TEST_POSTGRES_URL=postgres://sonda:sonda@localhost:15432/sonda go test ./internal/store   # store against Postgres
eval "$(dev/seed-contember.sh)"                  # Contember engine :14000 with a read-only and a write key
go test ./internal/query -run Contember          # data queries against a real Contember

The design and its trade-offs are in docs/design.md.

License

MIT