on-call

CLAUDE.md

@@ -263,6 +263,8 @@ INGEST_ADMISSION_WAIT_SECONDS=5       # how long a request may wait for a slot b
 
 # Notifications
 NOTIFICATION_POLL_SECONDS=60          # polled rule evaluation interval; minimum 5, invalid values fall back to 60
+ONCALL_POLL_SECONDS=30                # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
+OUTBOX_POLL_SECONDS=15                # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).
 
 # Retention (see "Data Retention" section below)
 SQLITE_RETENTION_DAYS=30              # 0 to disable; only applies in SQLite mode
@@ -704,6 +706,37 @@ Templates are DB rows seeded by migrations (`traceway-otel-agent` for the OTel h
 | POST | `/api/invitations/:token/accept` | None | Accept (new user) |
 | POST | `/api/invitations/:token/accept-existing` | App | Accept (existing user) |
 
+**On-Call** (teams, schedules, escalation policies, pages)
+
+Org-scoped entities in the main DB. Teams (`teams`/`team_members`) own projects one-to-one (`project_teams`, unique on project_id). Schedules (`oncall_schedules`) store PagerDuty-style calendar layers as a JSON `definition` document (rotations daily/weekly/custom, handoff time/day, time-of-day and day-of-week restrictions); one-off overrides are normalized rows (`oncall_overrides`). The pure resolution engine lives in `backend/app/oncall/` (`ResolveRange`/`ResolveAt`, tz-aware calendar math, later layer wins, overrides trump all). Both apply the same stacking, so a schedule puts exactly one person on call at any instant: `ResolveAt` is `ResolveRange` over a single instant, and paging a whole schedule stack (waking the person an override was meant to relieve) is the bug it exists to prevent. Escalation policies (`escalation_policies`) hold JSON steps (`targets` schedule/user/team/channel + `delayMinutes`, `repeatCount`); rules page on-call via the `escalation` notification channel type (config `{"policyId"}`), special-cased in `notifications/dispatch.go` through `RegisterPageOpener` (wired in `cmd/run.go`). A fired rule opens a `pages` row (dedup key `ruleId|dedupToken` with a partial unique index while unresolved; rules without a dedup token dedup at rule level; refires bump `event_count` and never reset the escalation clock). The escalator worker (`oncall/escalator.go`, `ONCALL_POLL_SECONDS`) claims due pages in a transaction (pg advisory lock 824737002 for multi-instance), inserts `page_notifications` rows, resolves targets to users, delivers via each user's `user_contact_methods` (email/slack/pushover/telegram adapter configs; account-email fallback always), then escalates level by level until ack/resolve or exhaustion. `RequireOrganizationAccess` middleware (any org role) gates member-level reads; mutations are org-admin; page acknowledge deliberately requires no write access.
+
+| Method | Endpoint | Auth | Purpose |
+|--------|----------|------|---------|
+| GET/POST | `/api/organizations/:organizationId/teams` | Member / Admin | List (with members+projects) / create |
+| PUT/DELETE | `/api/organizations/:organizationId/teams/:teamId` | Admin | Update / delete |
+| PUT | `.../teams/:teamId/members`, `.../teams/:teamId/projects` | Admin | Replace ordered members / owned projects |
+| GET/POST | `/api/organizations/:organizationId/schedules` | Member / Admin | List / create |
+| GET/PUT/DELETE | `.../schedules/:scheduleId` | Member / Admin / Admin | Detail+overrides / whole-document update / delete |
+| GET | `.../schedules/:scheduleId/timeline?from=&to=` | Member | Rendered per-layer + final shifts (max 62 days) |
+| POST/DELETE | `.../schedules/:scheduleId/overrides(/:overrideId)` | Member | Create (any member, max 30d) / delete (creator, covered user, or admin) |
+| GET | `/api/organizations/:organizationId/oncall/now` | Member | Overview: per team/schedule current + next on-call |
+| GET | `/api/oncall/current?projectId=` | App | Owning team + current on-call for a project (issue page) |
+| GET | `/api/escalation-policies` | App | Policies of the project's org (channel dialog picker) |
+| GET/POST | `/api/organizations/:organizationId/escalation-policies` | Member / Admin | List / create |
+| PUT/DELETE | `.../escalation-policies/:id` | Admin | Update / delete (422 while referenced by a channel) |
+| POST | `/api/pages` | App | List (POST-body: status open/acknowledged/resolved/active + pagination) |
+| GET | `/api/pages/:id` | App | Detail + delivery log |
+| POST | `/api/pages/:id/acknowledge` | App (no write gate) | open -> acknowledged, stops escalation; 409 if not open |
+| POST | `/api/pages/:id/resolve` | App | open/acknowledged -> resolved; 409 if already resolved |
+| GET | `/api/pages/open-count` | App | Sidebar badge count |
+| GET/POST | `/api/contact-methods` | App | Own contact methods (self-scoped). Types: email, slack, pushover, telegram, sms. SMS requires Twilio (`TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` + one of `TWILIO_FROM_NUMBER` / `TWILIO_MESSAGING_SERVICE_SID`); without them SMS is not offered at all — the list response carries `smsEnabled: false` so the type picker hides it; create, re-point, resend-code and test answer 422; the escalator drops existing sms methods before its "no methods left" check so those users fall back to the account email; and the adapter errors instead of reporting a delivery nobody received. Disabling and deleting a leftover sms method stay available. Creating/re-pointing an sms method starts code verification; unverified numbers are never paged |
+| PUT/DELETE | `/api/contact-methods/:id` | App | Update (incl. enabled toggle) / delete |
+| POST | `/api/contact-methods/:id/test` | App | Send a canned test through one method (422 for unverified sms) |
+| POST | `/api/contact-methods/:id/verify` | App (rate-limited) | Confirm the 6-digit SMS code (hashed at rest, 10-min expiry, 5-attempt cap). Deliberately **not** under `middleware.Transactional`: a wrong code answers 422, which would roll the consumed attempt back, so the handler manages its own transactions (nesting one under the middleware would also deadlock the single-connection SQLite main DB) |
+| POST | `/api/contact-methods/:id/resend-code` | App (rate-limited) | Re-issue the verification code |
+| GET/PUT | `/api/user-notification-rules` | App | Per-user notification-rule chains `{high: [{contactMethodId, delayMinutes}], low: [...]}` (PagerDuty-style: the page's urgency picks the chain; steps are enqueued at claim time as scheduled outbox deliveries and cancelled on ack; no chain = all enabled+verified methods immediately). Escalation policies carry `urgency: auto\|high\|low` in their definition (auto: critical -> high); pages store the resolved urgency |
+| GET/POST | `/api/ack/:token` | None (rate-limited) | Tokenized no-login acknowledge: per-delivery `twk_` tokens (SHA-256-hashed on page_notifications), GET = read-only summary (scanner-safe), POST = idempotent ack recorded as `acknowledged_via='link'` attributed to the delivery's recipient; 404 after resolve. Frontend page: `/ack/[token]` |
+
 **Logs**
 | Method | Endpoint | Auth | Purpose |
 |--------|----------|------|---------|
@@ -885,6 +918,8 @@ The DB rows in `session_recordings` are pruned by the SQLite retention worker (a
 
 The `profiles` DB rows (and their `storage_key`) are pruned by the SQLite retention worker / ClickHouse TTL above — not coupled to this disk cleanup, mirroring the session-recording split.
 
+**5. Main-DB outbox prune — `retention.Start` worker** (`backend/app/retention/outbox.go`, same `startDBPruneWorker` scaffolding as item 0). Runs in **all modes** against `db.DB`: once at startup, then every 24h, deleting terminal `notification_outbox` rows — `sent`/`cancelled` older than 7 days, `failed` older than 30 days. `pending` and `sending` rows are never pruned, so nothing undelivered is dropped. No env var — always on. The durable record of what was notified lives in `fired_notifications` and `page_notifications`, which this does not touch. Note that `pages` and `page_notifications` themselves are currently retained indefinitely.
+
 #### Session Recording Uploader
 
 Session recording segments arriving on `/api/report` are not uploaded inline. The handler enqueues each segment onto a bounded worker pool (`backend/app/recordings/uploader.go`, started from `cmd/run.go` next to `retention.Start`). Workers drain the queue and write the body via `storage.Store.Write` (S3 or local disk); successful writes are handed to a single batcher goroutine that calls `SessionRecordingRepository.InsertAsync` once per ~1000 rows or every 2 s, whichever comes first — single-row inserts are an anti-pattern for ClickHouse. Enqueue is non-blocking: when the queue is full the segment is dropped (newest-first) so a burst of `/api/report` traffic cannot spawn unbounded goroutines or saturate S3.
@@ -950,11 +985,10 @@ tags Map(String, String),  -- contextual tags from scope
 ### Database Migrations
 
 **CRITICAL RULES:**
-1. Each migration file must contain **exactly ONE SQL statement**
-2. Only create `.up.sql` files (no down migrations)
-3. Use sequential numbering: `NNNN_description.up.sql`
-
-**Why one statement per file?** ClickHouse migration runner executes each file as a single statement. Multiple statements will fail.
+1. `migrations/ch/` and `migrations/pg/` files must contain **exactly ONE SQL statement**. Both run through `golang-migrate`, and the ClickHouse driver is constructed with `MultiStatementEnabled: false` (`migrations_telemetry_ch.go`), so a second statement fails. `pg/` follows the same rule for symmetry.
+2. `migrations/sqlite/`, `migrations/sqlite_telemetry/` and `migrations/duckdb_telemetry/` run through `runMigrationsOn` in `migrations.go`, which splits on semicolons outside string literals (`splitStatements`, covered by `split_test.go`). A file there may hold a `CREATE TABLE` plus its indexes — that is the existing convention, e.g. `sqlite/0043_create_pages.up.sql`.
+3. Only create `.up.sql` files (no down migrations)
+4. Use sequential numbering: `NNNN_description.up.sql`
 
 **Example - Adding two columns requires TWO files:**
 ```

backend/app/config/config.go

@@ -1,6 +1,10 @@
 package config
 
-import "os"
+import (
+	"os"
+	"strconv"
+	"time"
+)
 
 type Cfg struct {
 	JWTSecret string
@@ -9,7 +13,7 @@ type Cfg struct {
 	PostgresHost     string
 	PostgresPort     string
 	PostgresDatabase string
-	PostgresUsername  string
+	PostgresUsername string
 	PostgresPassword string
 	PostgresSSLMode  string
 	SQLitePath       string
@@ -48,6 +52,15 @@ type Cfg struct {
 	SymbolicatorParser       string
 
 	NotificationPollSeconds string
+	OncallPollSeconds       string
+	OutboxPollSeconds       string
+
+	AllowPrivateNotificationTargets string
+
+	TwilioAccountSID          string
+	TwilioAuthToken           string
+	TwilioFromNumber          string
+	TwilioMessagingServiceSID string
 
 	SMTPEnabled  string
 	SMTPHost     string
@@ -89,6 +102,25 @@ var Config *Cfg
 
 func Init(c *Cfg) { Config = c }
 
+// PollSeconds parses a poll-interval value with a 5-second floor; empty or
+// invalid values fall back to defaultSeconds.
+func PollSeconds(value string, defaultSeconds int) time.Duration {
+	seconds := defaultSeconds
+	if value != "" {
+		if parsed, err := strconv.Atoi(value); err == nil && parsed >= 5 {
+			seconds = parsed
+		}
+	}
+	return time.Duration(seconds) * time.Second
+}
+
+// TwilioEnabled reports whether SMS sending is configured: account credentials
+// plus at least one sender (a from-number or a messaging service).
+func (c *Cfg) TwilioEnabled() bool {
+	return c.TwilioAccountSID != "" && c.TwilioAuthToken != "" &&
+		(c.TwilioFromNumber != "" || c.TwilioMessagingServiceSID != "")
+}
+
 func LoadFromEnv() *Cfg {
 	return &Cfg{
 		JWTSecret: os.Getenv("JWT_SECRET"),
@@ -97,7 +129,7 @@ func LoadFromEnv() *Cfg {
 		PostgresHost:     os.Getenv("POSTGRES_HOST"),
 		PostgresPort:     os.Getenv("POSTGRES_PORT"),
 		PostgresDatabase: os.Getenv("POSTGRES_DATABASE"),
-		PostgresUsername:  os.Getenv("POSTGRES_USERNAME"),
+		PostgresUsername: os.Getenv("POSTGRES_USERNAME"),
 		PostgresPassword: os.Getenv("POSTGRES_PASSWORD"),
 		PostgresSSLMode:  os.Getenv("POSTGRES_SSLMODE"),
 		SQLitePath:       os.Getenv("SQLITE_PATH"),
@@ -136,6 +168,15 @@ func LoadFromEnv() *Cfg {
 		SymbolicatorParser:       os.Getenv("SYMBOLICATOR_PARSER"),
 
 		NotificationPollSeconds: os.Getenv("NOTIFICATION_POLL_SECONDS"),
+		OncallPollSeconds:       os.Getenv("ONCALL_POLL_SECONDS"),
+		OutboxPollSeconds:       os.Getenv("OUTBOX_POLL_SECONDS"),
+
+		AllowPrivateNotificationTargets: os.Getenv("ALLOW_PRIVATE_NOTIFICATION_TARGETS"),
+
+		TwilioAccountSID:          os.Getenv("TWILIO_ACCOUNT_SID"),
+		TwilioAuthToken:           os.Getenv("TWILIO_AUTH_TOKEN"),
+		TwilioFromNumber:          os.Getenv("TWILIO_FROM_NUMBER"),
+		TwilioMessagingServiceSID: os.Getenv("TWILIO_MESSAGING_SERVICE_SID"),
 
 		SMTPEnabled:  os.Getenv("SMTP_ENABLED"),
 		SMTPHost:     os.Getenv("SMTP_HOST"),

backend/app/controllers/ack.controller.go

@@ -0,0 +1,146 @@
+package controllers
+
+import (
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/tracewayapp/traceway/backend/app/db"
+	"github.com/tracewayapp/traceway/backend/app/models"
+	"github.com/tracewayapp/traceway/backend/app/oncall"
+	"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+	"github.com/tracewayapp/traceway/backend/app/repositories/transactional/shared"
+
+	"github.com/gin-gonic/gin"
+	traceway "go.tracewayapp.com"
+)
+
+// ackController serves the tokenized no-login acknowledge flow.
+//
+// Security properties:
+//   - GET is strictly read-only: email scanners follow links, so the ack side
+//     effect lives on POST only.
+//   - Tokens are 256-bit random, stored as SHA-256 hashes, unique-indexed, and
+//     scoped to acknowledging exactly one page (no dashboard access, no
+//     resolve, no other pages).
+//   - Resolved pages 404, which is the token invalidation.
+//   - Both verbs are rate-limited per IP; a cheap shape pre-check rejects
+//     garbage before hashing.
+type ackController struct{}
+
+var AckController = ackController{}
+
+const ackTokenPrefix = "twk_"
+
+func ackTokenHashFromParam(param string) (string, bool) {
+	if !strings.HasPrefix(param, ackTokenPrefix) || len(param) < 40 || len(param) > 64 {
+		return "", false
+	}
+	return shared.HashAuthToken(param), true
+}
+
+type ackPageView struct {
+	Subject            string     `json:"subject"`
+	Body               string     `json:"body"`
+	Severity           string     `json:"severity"`
+	Urgency            string     `json:"urgency"`
+	Status             string     `json:"status"`
+	RuleName           string     `json:"ruleName"`
+	ProjectName        string     `json:"projectName"`
+	EventCount         int        `json:"eventCount"`
+	LastEventAt        time.Time  `json:"lastEventAt"`
+	CreatedAt          time.Time  `json:"createdAt"`
+	AcknowledgedAt     *time.Time `json:"acknowledgedAt"`
+	AcknowledgedByName *string    `json:"acknowledgedByName"`
+}
+
+// loadPageForToken resolves token -> delivery row -> page, returning nil when
+// the token is unknown or the page is resolved.
+func (c *ackController) loadPageForToken(ctx *gin.Context) (*models.Page, *models.PageNotification, bool) {
+	hash, ok := ackTokenHashFromParam(ctx.Param("token"))
+	if !ok {
+		ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+		return nil, nil, false
+	}
+	tx := db.GetTx(ctx)
+	notification, err := transactional.PageNotificationRepository.FindByAckTokenHash(tx, hash)
+	if err != nil {
+		ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to look up ack token: %w", err))
+		return nil, nil, false
+	}
+	if notification == nil {
+		ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+		return nil, nil, false
+	}
+	page, err := transactional.PageRepository.FindById(tx, notification.PageId)
+	if err != nil {
+		ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load page for ack token: %w", err))
+		return nil, nil, false
+	}
+	if page == nil || page.Status == models.PageStatusResolved {
+		ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+		return nil, nil, false
+	}
+	return page, notification, true
+}
+
+func (c *ackController) Get(ctx *gin.Context) {
+	tx := db.GetTx(ctx)
+	page, _, ok := c.loadPageForToken(ctx)
+	if !ok {
+		return
+	}
+
+	view := ackPageView{
+		Subject:        page.Subject,
+		Body:           page.Body,
+		Severity:       page.Severity,
+		Urgency:        page.Urgency,
+		Status:         page.Status,
+		RuleName:       page.RuleName,
+		EventCount:     page.EventCount,
+		LastEventAt:    page.LastEventAt,
+		CreatedAt:      page.CreatedAt,
+		AcknowledgedAt: page.AcknowledgedAt,
+	}
+	// Name lookups are best-effort decoration: the ack summary still renders
+	// without them, but a lookup error is reported rather than swallowed.
+	if project, err := transactional.ProjectRepository.FindById(tx, page.ProjectId); err != nil {
+		traceway.CaptureException(traceway.NewStackTraceErrorf("failed to load project name for ack view (page=%d): %w", page.Id, err))
+	} else if project != nil {
+		view.ProjectName = project.Name
+	}
+	if page.AcknowledgedBy != nil {
+		if user, err := transactional.UserRepository.FindById(tx, *page.AcknowledgedBy); err != nil {
+			traceway.CaptureException(traceway.NewStackTraceErrorf("failed to load acknowledger name for ack view (page=%d): %w", page.Id, err))
+		} else if user != nil {
+			view.AcknowledgedByName = &user.Name
+		}
+	}
+	ctx.JSON(http.StatusOK, view)
+}
+
+// Acknowledge acks the token's page, attributed to the delivery row's
+// recipient. Idempotent: re-taps on an already acknowledged page return 200.
+func (c *ackController) Acknowledge(ctx *gin.Context) {
+	tx := db.GetTx(ctx)
+	page, notification, ok := c.loadPageForToken(ctx)
+	if !ok {
+		return
+	}
+	if page.Status == models.PageStatusAcknowledged {
+		ctx.JSON(http.StatusOK, gin.H{"status": models.PageStatusAcknowledged})
+		return
+	}
+	acknowledged, err := oncall.AcknowledgePage(tx, page.Id, notification.UserId, oncall.AckViaLink, time.Now().UTC())
+	if err != nil {
+		ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to acknowledge page %d via link: %w", page.Id, err))
+		return
+	}
+	if !acknowledged {
+		// Lost a race with another ack; still a success for the tapper.
+		ctx.JSON(http.StatusOK, gin.H{"status": models.PageStatusAcknowledged})
+		return
+	}
+	ctx.JSON(http.StatusOK, gin.H{"status": models.PageStatusAcknowledged, "acknowledgedBy": notification.UserId})
+}