diff --git a/CLAUDE.md b/CLAUDE.md
index 0b1bf9b4..81b30066 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -263,6 +263,8 @@ INGEST_ADMISSION_WAIT_SECONDS=5 # how long a request may wait for a slot b
# Notifications
NOTIFICATION_POLL_SECONDS=60 # polled rule evaluation interval; minimum 5, invalid values fall back to 60
+ONCALL_POLL_SECONDS=30 # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
+OUTBOX_POLL_SECONDS=15 # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).
# Retention (see "Data Retention" section below)
SQLITE_RETENTION_DAYS=30 # 0 to disable; only applies in SQLite mode
@@ -704,6 +706,37 @@ Templates are DB rows seeded by migrations (`traceway-otel-agent` for the OTel h
| POST | `/api/invitations/:token/accept` | None | Accept (new user) |
| POST | `/api/invitations/:token/accept-existing` | App | Accept (existing user) |
+**On-Call** (teams, schedules, escalation policies, pages)
+
+Org-scoped entities in the main DB. Teams (`teams`/`team_members`) own projects one-to-one (`project_teams`, unique on project_id). Schedules (`oncall_schedules`) store PagerDuty-style calendar layers as a JSON `definition` document (rotations daily/weekly/custom, handoff time/day, time-of-day and day-of-week restrictions); one-off overrides are normalized rows (`oncall_overrides`). The pure resolution engine lives in `backend/app/oncall/` (`ResolveRange`/`ResolveAt`, tz-aware calendar math, later layer wins, overrides trump all). Both apply the same stacking, so a schedule puts exactly one person on call at any instant: `ResolveAt` is `ResolveRange` over a single instant, and paging a whole schedule stack (waking the person an override was meant to relieve) is the bug it exists to prevent. Escalation policies (`escalation_policies`) hold JSON steps (`targets` schedule/user/team/channel + `delayMinutes`, `repeatCount`); rules page on-call via the `escalation` notification channel type (config `{"policyId"}`), special-cased in `notifications/dispatch.go` through `RegisterPageOpener` (wired in `cmd/run.go`). A fired rule opens a `pages` row (dedup key `ruleId|dedupToken` with a partial unique index while unresolved; rules without a dedup token dedup at rule level; refires bump `event_count` and never reset the escalation clock). The escalator worker (`oncall/escalator.go`, `ONCALL_POLL_SECONDS`) claims due pages in a transaction (pg advisory lock 824737002 for multi-instance), inserts `page_notifications` rows, resolves targets to users, delivers via each user's `user_contact_methods` (email/slack/pushover/telegram adapter configs; account-email fallback always), then escalates level by level until ack/resolve or exhaustion. `RequireOrganizationAccess` middleware (any org role) gates member-level reads; mutations are org-admin; page acknowledge deliberately requires no write access.
+
+| Method | Endpoint | Auth | Purpose |
+|--------|----------|------|---------|
+| GET/POST | `/api/organizations/:organizationId/teams` | Member / Admin | List (with members+projects) / create |
+| PUT/DELETE | `/api/organizations/:organizationId/teams/:teamId` | Admin | Update / delete |
+| PUT | `.../teams/:teamId/members`, `.../teams/:teamId/projects` | Admin | Replace ordered members / owned projects |
+| GET/POST | `/api/organizations/:organizationId/schedules` | Member / Admin | List / create |
+| GET/PUT/DELETE | `.../schedules/:scheduleId` | Member / Admin / Admin | Detail+overrides / whole-document update / delete |
+| GET | `.../schedules/:scheduleId/timeline?from=&to=` | Member | Rendered per-layer + final shifts (max 62 days) |
+| POST/DELETE | `.../schedules/:scheduleId/overrides(/:overrideId)` | Member | Create (any member, max 30d) / delete (creator, covered user, or admin) |
+| GET | `/api/organizations/:organizationId/oncall/now` | Member | Overview: per team/schedule current + next on-call |
+| GET | `/api/oncall/current?projectId=` | App | Owning team + current on-call for a project (issue page) |
+| GET | `/api/escalation-policies` | App | Policies of the project's org (channel dialog picker) |
+| GET/POST | `/api/organizations/:organizationId/escalation-policies` | Member / Admin | List / create |
+| PUT/DELETE | `.../escalation-policies/:id` | Admin | Update / delete (422 while referenced by a channel) |
+| POST | `/api/pages` | App | List (POST-body: status open/acknowledged/resolved/active + pagination) |
+| GET | `/api/pages/:id` | App | Detail + delivery log |
+| POST | `/api/pages/:id/acknowledge` | App (no write gate) | open -> acknowledged, stops escalation; 409 if not open |
+| POST | `/api/pages/:id/resolve` | App | open/acknowledged -> resolved; 409 if already resolved |
+| GET | `/api/pages/open-count` | App | Sidebar badge count |
+| GET/POST | `/api/contact-methods` | App | Own contact methods (self-scoped). Types: email, slack, pushover, telegram, sms. SMS requires Twilio (`TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` + one of `TWILIO_FROM_NUMBER` / `TWILIO_MESSAGING_SERVICE_SID`); without them SMS is not offered at all — the list response carries `smsEnabled: false` so the type picker hides it; create, re-point, resend-code and test answer 422; the escalator drops existing sms methods before its "no methods left" check so those users fall back to the account email; and the adapter errors instead of reporting a delivery nobody received. Disabling and deleting a leftover sms method stay available. Creating/re-pointing an sms method starts code verification; unverified numbers are never paged |
+| PUT/DELETE | `/api/contact-methods/:id` | App | Update (incl. enabled toggle) / delete |
+| POST | `/api/contact-methods/:id/test` | App | Send a canned test through one method (422 for unverified sms) |
+| POST | `/api/contact-methods/:id/verify` | App (rate-limited) | Confirm the 6-digit SMS code (hashed at rest, 10-min expiry, 5-attempt cap). Deliberately **not** under `middleware.Transactional`: a wrong code answers 422, which would roll the consumed attempt back, so the handler manages its own transactions (nesting one under the middleware would also deadlock the single-connection SQLite main DB) |
+| POST | `/api/contact-methods/:id/resend-code` | App (rate-limited) | Re-issue the verification code |
+| GET/PUT | `/api/user-notification-rules` | App | Per-user notification-rule chains `{high: [{contactMethodId, delayMinutes}], low: [...]}` (PagerDuty-style: the page's urgency picks the chain; steps are enqueued at claim time as scheduled outbox deliveries and cancelled on ack; no chain = all enabled+verified methods immediately). Escalation policies carry `urgency: auto\|high\|low` in their definition (auto: critical -> high); pages store the resolved urgency |
+| GET/POST | `/api/ack/:token` | None (rate-limited) | Tokenized no-login acknowledge: per-delivery `twk_` tokens (SHA-256-hashed on page_notifications), GET = read-only summary (scanner-safe), POST = idempotent ack recorded as `acknowledged_via='link'` attributed to the delivery's recipient; 404 after resolve. Frontend page: `/ack/[token]` |
+
**Logs**
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
@@ -885,6 +918,8 @@ The DB rows in `session_recordings` are pruned by the SQLite retention worker (a
The `profiles` DB rows (and their `storage_key`) are pruned by the SQLite retention worker / ClickHouse TTL above — not coupled to this disk cleanup, mirroring the session-recording split.
+**5. Main-DB outbox prune — `retention.Start` worker** (`backend/app/retention/outbox.go`, same `startDBPruneWorker` scaffolding as item 0). Runs in **all modes** against `db.DB`: once at startup, then every 24h, deleting terminal `notification_outbox` rows — `sent`/`cancelled` older than 7 days, `failed` older than 30 days. `pending` and `sending` rows are never pruned, so nothing undelivered is dropped. No env var — always on. The durable record of what was notified lives in `fired_notifications` and `page_notifications`, which this does not touch. Note that `pages` and `page_notifications` themselves are currently retained indefinitely.
+
#### Session Recording Uploader
Session recording segments arriving on `/api/report` are not uploaded inline. The handler enqueues each segment onto a bounded worker pool (`backend/app/recordings/uploader.go`, started from `cmd/run.go` next to `retention.Start`). Workers drain the queue and write the body via `storage.Store.Write` (S3 or local disk); successful writes are handed to a single batcher goroutine that calls `SessionRecordingRepository.InsertAsync` once per ~1000 rows or every 2 s, whichever comes first — single-row inserts are an anti-pattern for ClickHouse. Enqueue is non-blocking: when the queue is full the segment is dropped (newest-first) so a burst of `/api/report` traffic cannot spawn unbounded goroutines or saturate S3.
@@ -950,11 +985,10 @@ tags Map(String, String), -- contextual tags from scope
### Database Migrations
**CRITICAL RULES:**
-1. Each migration file must contain **exactly ONE SQL statement**
-2. Only create `.up.sql` files (no down migrations)
-3. Use sequential numbering: `NNNN_description.up.sql`
-
-**Why one statement per file?** ClickHouse migration runner executes each file as a single statement. Multiple statements will fail.
+1. `migrations/ch/` and `migrations/pg/` files must contain **exactly ONE SQL statement**. Both run through `golang-migrate`, and the ClickHouse driver is constructed with `MultiStatementEnabled: false` (`migrations_telemetry_ch.go`), so a second statement fails. `pg/` follows the same rule for symmetry.
+2. `migrations/sqlite/`, `migrations/sqlite_telemetry/` and `migrations/duckdb_telemetry/` run through `runMigrationsOn` in `migrations.go`, which splits on semicolons outside string literals (`splitStatements`, covered by `split_test.go`). A file there may hold a `CREATE TABLE` plus its indexes — that is the existing convention, e.g. `sqlite/0043_create_pages.up.sql`.
+3. Only create `.up.sql` files (no down migrations)
+4. Use sequential numbering: `NNNN_description.up.sql`
**Example - Adding two columns requires TWO files:**
```
diff --git a/backend/app/config/config.go b/backend/app/config/config.go
index b9df860d..15cb5054 100644
--- a/backend/app/config/config.go
+++ b/backend/app/config/config.go
@@ -1,6 +1,10 @@
package config
-import "os"
+import (
+ "os"
+ "strconv"
+ "time"
+)
type Cfg struct {
JWTSecret string
@@ -9,7 +13,7 @@ type Cfg struct {
PostgresHost string
PostgresPort string
PostgresDatabase string
- PostgresUsername string
+ PostgresUsername string
PostgresPassword string
PostgresSSLMode string
SQLitePath string
@@ -48,6 +52,15 @@ type Cfg struct {
SymbolicatorParser string
NotificationPollSeconds string
+ OncallPollSeconds string
+ OutboxPollSeconds string
+
+ AllowPrivateNotificationTargets string
+
+ TwilioAccountSID string
+ TwilioAuthToken string
+ TwilioFromNumber string
+ TwilioMessagingServiceSID string
SMTPEnabled string
SMTPHost string
@@ -89,6 +102,25 @@ var Config *Cfg
func Init(c *Cfg) { Config = c }
+// PollSeconds parses a poll-interval value with a 5-second floor; empty or
+// invalid values fall back to defaultSeconds.
+func PollSeconds(value string, defaultSeconds int) time.Duration {
+ seconds := defaultSeconds
+ if value != "" {
+ if parsed, err := strconv.Atoi(value); err == nil && parsed >= 5 {
+ seconds = parsed
+ }
+ }
+ return time.Duration(seconds) * time.Second
+}
+
+// TwilioEnabled reports whether SMS sending is configured: account credentials
+// plus at least one sender (a from-number or a messaging service).
+func (c *Cfg) TwilioEnabled() bool {
+ return c.TwilioAccountSID != "" && c.TwilioAuthToken != "" &&
+ (c.TwilioFromNumber != "" || c.TwilioMessagingServiceSID != "")
+}
+
func LoadFromEnv() *Cfg {
return &Cfg{
JWTSecret: os.Getenv("JWT_SECRET"),
@@ -97,7 +129,7 @@ func LoadFromEnv() *Cfg {
PostgresHost: os.Getenv("POSTGRES_HOST"),
PostgresPort: os.Getenv("POSTGRES_PORT"),
PostgresDatabase: os.Getenv("POSTGRES_DATABASE"),
- PostgresUsername: os.Getenv("POSTGRES_USERNAME"),
+ PostgresUsername: os.Getenv("POSTGRES_USERNAME"),
PostgresPassword: os.Getenv("POSTGRES_PASSWORD"),
PostgresSSLMode: os.Getenv("POSTGRES_SSLMODE"),
SQLitePath: os.Getenv("SQLITE_PATH"),
@@ -136,6 +168,15 @@ func LoadFromEnv() *Cfg {
SymbolicatorParser: os.Getenv("SYMBOLICATOR_PARSER"),
NotificationPollSeconds: os.Getenv("NOTIFICATION_POLL_SECONDS"),
+ OncallPollSeconds: os.Getenv("ONCALL_POLL_SECONDS"),
+ OutboxPollSeconds: os.Getenv("OUTBOX_POLL_SECONDS"),
+
+ AllowPrivateNotificationTargets: os.Getenv("ALLOW_PRIVATE_NOTIFICATION_TARGETS"),
+
+ TwilioAccountSID: os.Getenv("TWILIO_ACCOUNT_SID"),
+ TwilioAuthToken: os.Getenv("TWILIO_AUTH_TOKEN"),
+ TwilioFromNumber: os.Getenv("TWILIO_FROM_NUMBER"),
+ TwilioMessagingServiceSID: os.Getenv("TWILIO_MESSAGING_SERVICE_SID"),
SMTPEnabled: os.Getenv("SMTP_ENABLED"),
SMTPHost: os.Getenv("SMTP_HOST"),
diff --git a/backend/app/controllers/ack.controller.go b/backend/app/controllers/ack.controller.go
new file mode 100644
index 00000000..04485c94
--- /dev/null
+++ b/backend/app/controllers/ack.controller.go
@@ -0,0 +1,146 @@
+package controllers
+
+import (
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional/shared"
+
+ "github.com/gin-gonic/gin"
+ traceway "go.tracewayapp.com"
+)
+
+// ackController serves the tokenized no-login acknowledge flow.
+//
+// Security properties:
+// - GET is strictly read-only: email scanners follow links, so the ack side
+// effect lives on POST only.
+// - Tokens are 256-bit random, stored as SHA-256 hashes, unique-indexed, and
+// scoped to acknowledging exactly one page (no dashboard access, no
+// resolve, no other pages).
+// - Resolved pages 404, which is the token invalidation.
+// - Both verbs are rate-limited per IP; a cheap shape pre-check rejects
+// garbage before hashing.
+type ackController struct{}
+
+var AckController = ackController{}
+
+const ackTokenPrefix = "twk_"
+
+func ackTokenHashFromParam(param string) (string, bool) {
+ if !strings.HasPrefix(param, ackTokenPrefix) || len(param) < 40 || len(param) > 64 {
+ return "", false
+ }
+ return shared.HashAuthToken(param), true
+}
+
+type ackPageView struct {
+ Subject string `json:"subject"`
+ Body string `json:"body"`
+ Severity string `json:"severity"`
+ Urgency string `json:"urgency"`
+ Status string `json:"status"`
+ RuleName string `json:"ruleName"`
+ ProjectName string `json:"projectName"`
+ EventCount int `json:"eventCount"`
+ LastEventAt time.Time `json:"lastEventAt"`
+ CreatedAt time.Time `json:"createdAt"`
+ AcknowledgedAt *time.Time `json:"acknowledgedAt"`
+ AcknowledgedByName *string `json:"acknowledgedByName"`
+}
+
+// loadPageForToken resolves token -> delivery row -> page, returning nil when
+// the token is unknown or the page is resolved.
+func (c *ackController) loadPageForToken(ctx *gin.Context) (*models.Page, *models.PageNotification, bool) {
+ hash, ok := ackTokenHashFromParam(ctx.Param("token"))
+ if !ok {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+ return nil, nil, false
+ }
+ tx := db.GetTx(ctx)
+ notification, err := transactional.PageNotificationRepository.FindByAckTokenHash(tx, hash)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to look up ack token: %w", err))
+ return nil, nil, false
+ }
+ if notification == nil {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+ return nil, nil, false
+ }
+ page, err := transactional.PageRepository.FindById(tx, notification.PageId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load page for ack token: %w", err))
+ return nil, nil, false
+ }
+ if page == nil || page.Status == models.PageStatusResolved {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+ return nil, nil, false
+ }
+ return page, notification, true
+}
+
+func (c *ackController) Get(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ page, _, ok := c.loadPageForToken(ctx)
+ if !ok {
+ return
+ }
+
+ view := ackPageView{
+ Subject: page.Subject,
+ Body: page.Body,
+ Severity: page.Severity,
+ Urgency: page.Urgency,
+ Status: page.Status,
+ RuleName: page.RuleName,
+ EventCount: page.EventCount,
+ LastEventAt: page.LastEventAt,
+ CreatedAt: page.CreatedAt,
+ AcknowledgedAt: page.AcknowledgedAt,
+ }
+ // Name lookups are best-effort decoration: the ack summary still renders
+ // without them, but a lookup error is reported rather than swallowed.
+ if project, err := transactional.ProjectRepository.FindById(tx, page.ProjectId); err != nil {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("failed to load project name for ack view (page=%d): %w", page.Id, err))
+ } else if project != nil {
+ view.ProjectName = project.Name
+ }
+ if page.AcknowledgedBy != nil {
+ if user, err := transactional.UserRepository.FindById(tx, *page.AcknowledgedBy); err != nil {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("failed to load acknowledger name for ack view (page=%d): %w", page.Id, err))
+ } else if user != nil {
+ view.AcknowledgedByName = &user.Name
+ }
+ }
+ ctx.JSON(http.StatusOK, view)
+}
+
+// Acknowledge acks the token's page, attributed to the delivery row's
+// recipient. Idempotent: re-taps on an already acknowledged page return 200.
+func (c *ackController) Acknowledge(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ page, notification, ok := c.loadPageForToken(ctx)
+ if !ok {
+ return
+ }
+ if page.Status == models.PageStatusAcknowledged {
+ ctx.JSON(http.StatusOK, gin.H{"status": models.PageStatusAcknowledged})
+ return
+ }
+ acknowledged, err := oncall.AcknowledgePage(tx, page.Id, notification.UserId, oncall.AckViaLink, time.Now().UTC())
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to acknowledge page %d via link: %w", page.Id, err))
+ return
+ }
+ if !acknowledged {
+ // Lost a race with another ack; still a success for the tapper.
+ ctx.JSON(http.StatusOK, gin.H{"status": models.PageStatusAcknowledged})
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"status": models.PageStatusAcknowledged, "acknowledgedBy": notification.UserId})
+}
diff --git a/backend/app/controllers/contact_method.controller.go b/backend/app/controllers/contact_method.controller.go
new file mode 100644
index 00000000..a76aefa1
--- /dev/null
+++ b/backend/app/controllers/contact_method.controller.go
@@ -0,0 +1,514 @@
+package controllers
+
+import (
+ "crypto/rand"
+ "database/sql"
+ "encoding/binary"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional/shared"
+
+ "github.com/gin-gonic/gin"
+ traceway "go.tracewayapp.com"
+)
+
+type contactMethodController struct{}
+
+var ContactMethodController = contactMethodController{}
+
+type contactMethodRequest struct {
+ MethodType string `json:"methodType"`
+ Config json.RawMessage `json:"config"`
+ Enabled *bool `json:"enabled"`
+}
+
+// Personal-delivery adapter shapes only; webhook/github are not contact
+// methods.
+var validContactMethodTypes = map[string]bool{
+ "email": true, "slack": true, "pushover": true, "telegram": true, "sms": true,
+}
+
+const (
+ verificationCodeTTL = 10 * time.Minute
+ maxVerificationAttempts = 5
+ maxContactEmailLength = 254
+ verificationSubjectTemplate = "Your Traceway verification code is %s"
+)
+
+// Per-phone-number cap on verification SMS sends, across all users and
+// methods: the route rate limits alone would still let one account (or many)
+// repoint methods in a loop and flood a number with codes.
+var (
+ verificationSendLimiter = middleware.NewFixedWindowLimiter(3, time.Hour)
+ errVerificationSendLimited = errors.New("verification send limit reached for this number")
+)
+
+// newVerificationCode returns a 6-digit code from crypto/rand
+// (rejection-sampled so every code is equally likely).
+func newVerificationCode() (string, error) {
+ for {
+ var raw [4]byte
+ if _, err := rand.Read(raw[:]); err != nil {
+ return "", err
+ }
+ value := binary.BigEndian.Uint32(raw[:])
+ if value >= 4_000_000_000 {
+ continue
+ }
+ return fmt.Sprintf("%06d", value%1_000_000), nil
+ }
+}
+
+const smsUnavailableMessage = "SMS is not available on this Traceway instance because no Twilio credentials are configured."
+
+// smsUnavailable reports whether an SMS message could not be delivered at all.
+// It gates the paths that would start a phone number down the SMS road; it
+// deliberately does not gate plain edits or deletes, so a method left behind by
+// a removed Twilio config can still be turned off and cleaned up.
+func smsUnavailable(methodType string) bool {
+ return methodType == "sms" && !config.Config.TwilioEnabled()
+}
+
+// beginVerification stores a fresh hashed code on the method and enqueues the
+// verification SMS through the outbox (retries for free, Twilio off the
+// request path). Returns errVerificationSendLimited when the number's send
+// budget is exhausted.
+func beginVerification(ctx *gin.Context, tx *sql.Tx, method *models.UserContactMethod) error {
+ if !verificationSendLimiter.Allow(oncall.SMSPhoneNumber(method.Config)) {
+ return errVerificationSendLimited
+ }
+ code, err := newVerificationCode()
+ if err != nil {
+ return err
+ }
+ if err := outbox.CancelByKey(tx, outbox.VerificationCancelKey(method.Id)); err != nil {
+ return err
+ }
+ expiresAt := time.Now().UTC().Add(verificationCodeTTL)
+ if err := transactional.UserContactMethodRepository.SetVerification(tx, method.Id, shared.HashAuthToken(code), expiresAt); err != nil {
+ return err
+ }
+ if _, err := outbox.Enqueue(tx, outbox.Delivery{
+ Kind: models.OutboxKindVerification,
+ AdapterType: "sms",
+ AdapterConfig: json.RawMessage(method.Config),
+ CancelKey: outbox.VerificationCancelKey(method.Id),
+ Message: notifications.Message{
+ Subject: fmt.Sprintf(verificationSubjectTemplate, code),
+ Severity: notifications.SeverityInfo,
+ },
+ }); err != nil {
+ return err
+ }
+ // Waking inline would race the drain worker against the still-open
+ // request transaction: it would poll, see nothing, and sleep out the
+ // full interval.
+ middleware.OnCommit(ctx, outbox.Wake)
+ return nil
+}
+
+// respondBeginVerificationError maps a beginVerification failure to the right
+// response: 429 for a throttled number, 500 otherwise.
+func respondBeginVerificationError(ctx *gin.Context, err error, reason string) {
+ if errors.Is(err, errVerificationSendLimited) {
+ ctx.JSON(http.StatusTooManyRequests, gin.H{"error": "Too many verification codes were sent to this number recently. Try again in an hour."})
+ return
+ }
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf(reason+": %w", err))
+}
+
+func (c *contactMethodController) List(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ userId := middleware.GetUserId(ctx)
+ methods, err := transactional.UserContactMethodRepository.FindByUser(tx, userId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list contact methods: %w", err))
+ return
+ }
+ if methods == nil {
+ methods = []*models.UserContactMethod{}
+ }
+ // smsEnabled drives the type picker: without Twilio credentials the
+ // instance cannot send a single message, so SMS is never offered.
+ ctx.JSON(http.StatusOK, gin.H{"methods": methods, "smsEnabled": config.Config.TwilioEnabled()})
+}
+
+func (c *contactMethodController) Create(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ userId := middleware.GetUserId(ctx)
+
+ var request contactMethodRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if smsUnavailable(request.MethodType) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": smsUnavailableMessage})
+ return
+ }
+ if message := validateContactMethod(request.MethodType, request.Config); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ existing, err := transactional.UserContactMethodRepository.FindByUser(tx, userId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check existing contact methods: %w", err))
+ return
+ }
+ for _, method := range existing {
+ if method.MethodType == request.MethodType && sameContactConfig(json.RawMessage(method.Config), request.Config) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "You already have this contact method."})
+ return
+ }
+ }
+
+ method := &models.UserContactMethod{
+ UserId: userId,
+ MethodType: request.MethodType,
+ Config: models.JSONText(request.Config),
+ Enabled: true,
+ // Only phone numbers need proving; every other method type is
+ // usable immediately.
+ Verified: request.MethodType != "sms",
+ CreatedAt: time.Now().UTC(),
+ }
+ id, err := transactional.UserContactMethodRepository.Create(tx, method)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to create contact method: %w", err))
+ return
+ }
+ method.Id = id
+ if method.MethodType == "sms" {
+ if err := beginVerification(ctx, tx, method); err != nil {
+ respondBeginVerificationError(ctx, err, "failed to start phone verification")
+ return
+ }
+ }
+ ctx.JSON(http.StatusCreated, method)
+}
+
+func (c *contactMethodController) Update(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ method, ok := c.loadOwnMethod(ctx)
+ if !ok {
+ return
+ }
+ var request contactMethodRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if request.MethodType == "" {
+ request.MethodType = method.MethodType
+ }
+ if request.Config == nil {
+ request.Config = json.RawMessage(method.Config)
+ }
+ if message := validateContactMethod(request.MethodType, request.Config); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+
+ // Re-pointing an sms method at a different number voids the verification.
+ configChanged := string(method.Config) != string(request.Config)
+ if request.MethodType == "sms" {
+ // JSONB round-trips reformat the stored config, so a byte comparison
+ // would void verification (and send a code) on a no-op save; the
+ // number is the only thing verification proves.
+ configChanged = oncall.SMSPhoneNumber(method.Config) != oncall.SMSPhoneNumber(request.Config)
+ }
+ typeChanged := method.MethodType != request.MethodType
+ reverify := request.MethodType == "sms" && (configChanged || typeChanged)
+ // Only a change that would need a new code is blocked: disabling or
+ // renaming a leftover sms method must stay possible after Twilio is gone.
+ if reverify && smsUnavailable(request.MethodType) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": smsUnavailableMessage})
+ return
+ }
+
+ method.MethodType = request.MethodType
+ method.Config = models.JSONText(request.Config)
+ if request.Enabled != nil {
+ method.Enabled = *request.Enabled
+ }
+ if reverify {
+ method.Verified = false
+ }
+ if request.MethodType != "sms" {
+ method.Verified = true
+ }
+ if err := transactional.UserContactMethodRepository.Update(tx, method); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to update contact method: %w", err))
+ return
+ }
+ if reverify {
+ if err := beginVerification(ctx, tx, method); err != nil {
+ respondBeginVerificationError(ctx, err, "failed to restart phone verification")
+ return
+ }
+ }
+ ctx.JSON(http.StatusOK, method)
+}
+
+// Verify runs outside middleware.Transactional (see routes.go): each step gets
+// its own transaction so consuming an attempt survives the 422 a wrong code
+// answers with.
+func (c *contactMethodController) Verify(ctx *gin.Context) {
+ method, ok := c.loadOwnMethodInOwnTx(ctx)
+ if !ok {
+ return
+ }
+ var request struct {
+ Code string `json:"code"`
+ }
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if method.Verified {
+ ctx.JSON(http.StatusOK, gin.H{"verified": true})
+ return
+ }
+ if method.VerificationExpiresAt == nil || time.Now().UTC().After(*method.VerificationExpiresAt) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "The code has expired. Request a new one."})
+ return
+ }
+ // The attempt is consumed atomically in SQL, before the hash compare: a
+ // check-then-increment would let concurrent requests exceed the cap.
+ attemptAllowed, err := db.ExecuteTransaction(func(attemptTx *sql.Tx) (bool, error) {
+ return transactional.UserContactMethodRepository.IncrementVerificationAttempts(attemptTx, method.Id, maxVerificationAttempts)
+ })
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to record verification attempt: %w", err))
+ return
+ }
+ if !attemptAllowed {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Too many attempts. Request a new code."})
+ return
+ }
+ if shared.HashAuthToken(strings.TrimSpace(request.Code)) != method.VerificationCodeHash {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "The code is not correct."})
+ return
+ }
+ if _, err := db.ExecuteTransaction(func(verifyTx *sql.Tx) (struct{}, error) {
+ if err := transactional.UserContactMethodRepository.MarkVerified(verifyTx, method.Id); err != nil {
+ return struct{}{}, err
+ }
+ return struct{}{}, outbox.CancelByKey(verifyTx, outbox.VerificationCancelKey(method.Id))
+ }); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to mark contact method verified: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"verified": true})
+}
+
+func (c *contactMethodController) ResendCode(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ method, ok := c.loadOwnMethod(ctx)
+ if !ok {
+ return
+ }
+ if method.MethodType != "sms" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Only phone numbers need verification."})
+ return
+ }
+ if method.Verified {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "This phone number is already verified."})
+ return
+ }
+ if smsUnavailable(method.MethodType) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": smsUnavailableMessage})
+ return
+ }
+ if err := beginVerification(ctx, tx, method); err != nil {
+ respondBeginVerificationError(ctx, err, "failed to resend verification code")
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Verification code sent"})
+}
+
+func (c *contactMethodController) Delete(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ method, ok := c.loadOwnMethod(ctx)
+ if !ok {
+ return
+ }
+ if err := transactional.UserContactMethodRepository.Delete(tx, method.Id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete contact method: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Contact method deleted"})
+}
+
+// Test sends a canned message through one method, mirroring the channel test
+// endpoint (no Transactional middleware; short-lived reads only).
+func (c *contactMethodController) Test(ctx *gin.Context) {
+ userId := middleware.GetUserId(ctx)
+ methodId, err := strconv.Atoi(ctx.Param("id"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid contact method ID"})
+ return
+ }
+
+ type methodAndUser struct {
+ Method *models.UserContactMethod
+ User *models.User
+ }
+ loaded, err := db.ExecuteTransaction(func(tx *sql.Tx) (methodAndUser, error) {
+ method, err := transactional.UserContactMethodRepository.FindById(tx, methodId)
+ if err != nil {
+ return methodAndUser{}, err
+ }
+ user, err := transactional.UserRepository.FindById(tx, userId)
+ if err != nil {
+ return methodAndUser{}, err
+ }
+ return methodAndUser{Method: method, User: user}, nil
+ })
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load contact method: %w", err))
+ return
+ }
+ if loaded.Method == nil || loaded.Method.UserId != userId || loaded.User == nil {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Contact method not found"})
+ return
+ }
+ if smsUnavailable(loaded.Method.MethodType) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": smsUnavailableMessage})
+ return
+ }
+ if loaded.Method.MethodType == "sms" && !loaded.Method.Verified {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Verify this phone number before testing it."})
+ return
+ }
+
+ adapterConfig := json.RawMessage(loaded.Method.Config)
+ if loaded.Method.MethodType == "email" {
+ adapterConfig, _ = oncall.EmailDeliveryFor(loaded.User.Email, oncall.ParseEmailOverride(loaded.Method.Config))
+ }
+ adapter, err := notifications.NewAdapter(loaded.Method.MethodType, adapterConfig)
+ if err != nil {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ return
+ }
+ testMsg := notifications.Message{
+ Subject: "Traceway on-call test",
+ Body: "This is a test of your on-call contact method. If you received this, pages will reach you here.",
+ Severity: notifications.SeverityInfo,
+ RuleType: "test",
+ RuleName: "Test",
+ }
+ if err := adapter.Send(ctx.Request.Context(), testMsg); err != nil {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Test notification failed: " + err.Error()})
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"success": true})
+}
+
+func (c *contactMethodController) loadOwnMethod(ctx *gin.Context) (*models.UserContactMethod, bool) {
+ return c.resolveOwnMethod(ctx, func(methodId int) (*models.UserContactMethod, error) {
+ return transactional.UserContactMethodRepository.FindById(db.GetTx(ctx), methodId)
+ })
+}
+
+// loadOwnMethodInOwnTx is the loader for handlers that run without the
+// Transactional middleware. Nesting a transaction under one is not an option:
+// the main SQLite DB has a single connection, so the inner Begin would wait on
+// the outer transaction forever.
+func (c *contactMethodController) loadOwnMethodInOwnTx(ctx *gin.Context) (*models.UserContactMethod, bool) {
+ return c.resolveOwnMethod(ctx, func(methodId int) (*models.UserContactMethod, error) {
+ return db.ExecuteTransaction(func(tx *sql.Tx) (*models.UserContactMethod, error) {
+ return transactional.UserContactMethodRepository.FindById(tx, methodId)
+ })
+ })
+}
+
+func (c *contactMethodController) resolveOwnMethod(ctx *gin.Context, find func(int) (*models.UserContactMethod, error)) (*models.UserContactMethod, bool) {
+ userId := middleware.GetUserId(ctx)
+ methodId, err := strconv.Atoi(ctx.Param("id"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid contact method ID"})
+ return nil, false
+ }
+ method, err := find(methodId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load contact method: %w", err))
+ return nil, false
+ }
+ if method == nil || method.UserId != userId {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Contact method not found"})
+ return nil, false
+ }
+ return method, true
+}
+
+func sameContactConfig(a, b json.RawMessage) bool {
+ var left, right any
+ if json.Unmarshal(a, &left) != nil || json.Unmarshal(b, &right) != nil {
+ return false
+ }
+ leftJSON, err := json.Marshal(left)
+ if err != nil {
+ return false
+ }
+ rightJSON, err := json.Marshal(right)
+ if err != nil {
+ return false
+ }
+ return string(leftJSON) == string(rightJSON)
+}
+
+func validateContactMethod(methodType string, methodConfig json.RawMessage) string {
+ if !validContactMethodTypes[methodType] {
+ return "Method type must be one of: email, slack, pushover, telegram, sms."
+ }
+ if methodType == "email" {
+ var parsed struct {
+ Email string `json:"email"`
+ }
+ if len(methodConfig) > 0 {
+ if err := json.Unmarshal(methodConfig, &parsed); err != nil {
+ return "Invalid email configuration."
+ }
+ }
+ if parsed.Email != "" && !strings.Contains(parsed.Email, "@") {
+ return "The email address is not valid."
+ }
+ if len(parsed.Email) > maxContactEmailLength {
+ return "The email address is too long."
+ }
+ return ""
+ }
+ adapter, err := notifications.NewAdapter(methodType, methodConfig)
+ if err != nil {
+ return err.Error()
+ }
+ if err := adapter.Validate(); err != nil {
+ return err.Error()
+ }
+ if methodType == "slack" {
+ var parsed struct {
+ WebhookURL string `json:"webhookUrl"`
+ }
+ if err := json.Unmarshal(methodConfig, &parsed); err == nil {
+ if err := notifications.ValidateOutboundURL(parsed.WebhookURL); err != nil {
+ return err.Error()
+ }
+ }
+ }
+ return ""
+}
diff --git a/backend/app/controllers/escalation_policy.controller.go b/backend/app/controllers/escalation_policy.controller.go
new file mode 100644
index 00000000..f35e449f
--- /dev/null
+++ b/backend/app/controllers/escalation_policy.controller.go
@@ -0,0 +1,243 @@
+package controllers
+
+import (
+ "database/sql"
+ "encoding/json"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+
+ "github.com/gin-gonic/gin"
+ traceway "go.tracewayapp.com"
+)
+
+type escalationPolicyController struct{}
+
+var EscalationPolicyController = escalationPolicyController{}
+
+type escalationPolicyRequest struct {
+ Name string `json:"name"`
+ Definition json.RawMessage `json:"definition"`
+}
+
+// ListForProject serves the channel-dialog picker and the on-call page: all
+// policies of the project's organization, readable by any project member.
+func (c *escalationPolicyController) ListForProject(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ project, err := transactional.ProjectRepository.FindById(tx, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load project: %w", err))
+ return
+ }
+ if project == nil || project.OrganizationId == nil {
+ ctx.JSON(http.StatusOK, gin.H{"policies": []*models.EscalationPolicy{}})
+ return
+ }
+ policies, err := transactional.EscalationPolicyRepository.FindByOrganization(tx, *project.OrganizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list escalation policies: %w", err))
+ return
+ }
+ if policies == nil {
+ policies = []*models.EscalationPolicy{}
+ }
+ ctx.JSON(http.StatusOK, gin.H{"policies": policies})
+}
+
+func (c *escalationPolicyController) ListForOrganization(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+ policies, err := transactional.EscalationPolicyRepository.FindByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list escalation policies: %w", err))
+ return
+ }
+ if policies == nil {
+ policies = []*models.EscalationPolicy{}
+ }
+ ctx.JSON(http.StatusOK, gin.H{"policies": policies})
+}
+
+func (c *escalationPolicyController) Create(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+ userId := middleware.GetUserId(ctx)
+
+ var request escalationPolicyRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ request.Name = strings.TrimSpace(request.Name)
+ if message := validatePolicyName(request.Name); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ existing, err := transactional.EscalationPolicyRepository.FindByOrganizationAndName(tx, organizationId, request.Name)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check policy name: %w", err))
+ return
+ }
+ if existing != nil {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "An escalation policy with this name already exists."})
+ return
+ }
+ definition, err := oncall.ValidatePolicyDefinition(tx, organizationId, request.Definition)
+ if err != nil {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ return
+ }
+ normalized, err := oncall.MarshalPolicyDefinition(definition)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to marshal policy definition: %w", err))
+ return
+ }
+
+ now := time.Now().UTC()
+ policy := &models.EscalationPolicy{
+ OrganizationId: organizationId,
+ Name: request.Name,
+ Definition: models.JSONText(normalized),
+ CreatedBy: &userId,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+ id, err := transactional.EscalationPolicyRepository.Create(tx, policy)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to create escalation policy: %w", err))
+ return
+ }
+ policy.Id = id
+ ctx.JSON(http.StatusCreated, policy)
+}
+
+func (c *escalationPolicyController) Update(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ policy, ok := c.loadPolicy(ctx, organizationId)
+ if !ok {
+ return
+ }
+ var request escalationPolicyRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ request.Name = strings.TrimSpace(request.Name)
+ if message := validatePolicyName(request.Name); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ existing, err := transactional.EscalationPolicyRepository.FindByOrganizationAndName(tx, organizationId, request.Name)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check policy name: %w", err))
+ return
+ }
+ if existing != nil && existing.Id != policy.Id {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "An escalation policy with this name already exists."})
+ return
+ }
+ definition, err := oncall.ValidatePolicyDefinition(tx, organizationId, request.Definition)
+ if err != nil {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ return
+ }
+ normalized, err := oncall.MarshalPolicyDefinition(definition)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to marshal policy definition: %w", err))
+ return
+ }
+
+ policy.Name = request.Name
+ policy.Definition = models.JSONText(normalized)
+ policy.UpdatedAt = time.Now().UTC()
+ if err := transactional.EscalationPolicyRepository.Update(tx, policy); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to update escalation policy: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, policy)
+}
+
+func (c *escalationPolicyController) Delete(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ policy, ok := c.loadPolicy(ctx, organizationId)
+ if !ok {
+ return
+ }
+
+ // Block deletion while an escalation channel references the policy, so a
+ // rule cannot silently start failing to page.
+ referencing, err := c.findReferencingChannels(tx, organizationId, policy.Id)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check referencing channels: %w", err))
+ return
+ }
+ if len(referencing) > 0 {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "This policy is used by notification channel(s): " + strings.Join(referencing, ", ") + ". Remove those channels first."})
+ return
+ }
+
+ if err := transactional.EscalationPolicyRepository.Delete(tx, policy.Id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete escalation policy: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Escalation policy deleted"})
+}
+
+func (c *escalationPolicyController) findReferencingChannels(tx *sql.Tx, organizationId int, policyId int) ([]string, error) {
+ channels, err := transactional.NotificationChannelRepository.FindEscalationByOrganization(tx, organizationId)
+ if err != nil {
+ return nil, err
+ }
+ var names []string
+ for _, channel := range channels {
+ if oncall.EscalationChannelPolicyId(json.RawMessage(channel.Config)) == policyId {
+ names = append(names, channel.Name)
+ }
+ }
+ return names, nil
+}
+
+func (c *escalationPolicyController) loadPolicy(ctx *gin.Context, organizationId int) (*models.EscalationPolicy, bool) {
+ tx := db.GetTx(ctx)
+ policyId, err := strconv.Atoi(ctx.Param("id"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid policy ID"})
+ return nil, false
+ }
+ policy, err := transactional.EscalationPolicyRepository.FindById(tx, policyId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load escalation policy: %w", err))
+ return nil, false
+ }
+ if policy == nil || policy.OrganizationId != organizationId {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Escalation policy not found"})
+ return nil, false
+ }
+ return policy, true
+}
+
+func validatePolicyName(name string) string {
+ if name == "" {
+ return "A policy name is required."
+ }
+ if len(name) > 100 {
+ return "The policy name can be at most 100 characters."
+ }
+ return ""
+}
diff --git a/backend/app/controllers/health_deep.go b/backend/app/controllers/health_deep.go
index 999cc5ad..658a6574 100644
--- a/backend/app/controllers/health_deep.go
+++ b/backend/app/controllers/health_deep.go
@@ -1,10 +1,13 @@
package controllers
import (
+ "fmt"
"net/http"
"github.com/gin-gonic/gin"
"github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+ traceway "go.tracewayapp.com"
)
type healthDeepController struct{}
@@ -37,6 +40,7 @@ type HealthDeepResponse struct {
InsertFailures uint64 `json:"insertFailures"`
IngestRejected uint64 `json:"ingestRejected"`
Engine *db.TelemetryEngineStats `json:"engine,omitempty"`
+ Outbox *outbox.HealthStats `json:"outbox,omitempty"`
}
// 503 means the configured telemetry backend is down. On the embedded
@@ -54,6 +58,12 @@ func (h healthDeepController) Get(c *gin.Context) {
if engine, ok := db.GetTelemetryEngineStats(c.Request.Context()); ok {
resp.Engine = &engine
}
+ // Best-effort: health must not 500 because the main DB hiccuped.
+ if outboxStats, err := outbox.HealthSnapshot(); err == nil {
+ resp.Outbox = outboxStats
+ } else {
+ traceway.CaptureException(fmt.Errorf("failed to load outbox health: %w", err))
+ }
if resp.TelemetryBackend == "clickhouse" && !resp.CHReachable {
c.JSON(http.StatusServiceUnavailable, resp)
diff --git a/backend/app/controllers/member.controller.go b/backend/app/controllers/member.controller.go
index 1bf8a581..695b2937 100644
--- a/backend/app/controllers/member.controller.go
+++ b/backend/app/controllers/member.controller.go
@@ -4,6 +4,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
"net/http"
"strconv"
@@ -110,6 +111,18 @@ func (c *memberController) RemoveMember(ctx *gin.Context) {
return
}
+ err = transactional.TeamRepository.RemoveUserFromOrgTeams(tx, organizationId, targetUserId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("Failed to remove member from teams: %w", err))
+ return
+ }
+
+ err = oncall.RemoveUserFromOrgSchedules(tx, organizationId, targetUserId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("Failed to remove member from on-call schedules: %w", err))
+ return
+ }
+
ctx.JSON(http.StatusOK, gin.H{"message": "Member removed"})
}
diff --git a/backend/app/controllers/notification_channel.controller.go b/backend/app/controllers/notification_channel.controller.go
index be1f8d6e..d5650ee8 100644
--- a/backend/app/controllers/notification_channel.controller.go
+++ b/backend/app/controllers/notification_channel.controller.go
@@ -13,7 +13,10 @@ import (
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+
+ "github.com/google/uuid"
traceway "go.tracewayapp.com"
)
@@ -43,7 +46,45 @@ type createChannelRequest struct {
}
var validChannelTypes = map[string]bool{
- "email": true, "webhook": true, "slack": true, "github": true, "pushover": true, "telegram": true,
+ "email": true, "webhook": true, "slack": true, "github": true, "pushover": true, "telegram": true, "escalation": true,
+}
+
+// validateEscalationChannelConfig checks the {policyId} config against the
+// project's organization. Escalation channels have no adapter, so this
+// replaces the NewAdapter validation path. Returns a 422 message.
+func validateEscalationChannelConfig(tx *sql.Tx, projectId uuid.UUID, config json.RawMessage) (string, error) {
+ policyId := oncall.EscalationChannelPolicyId(config)
+ if policyId == 0 {
+ return "An escalation policy is required.", nil
+ }
+ policy, err := transactional.EscalationPolicyRepository.FindById(tx, policyId)
+ if err != nil {
+ return "", err
+ }
+ project, err := transactional.ProjectRepository.FindById(tx, projectId)
+ if err != nil {
+ return "", err
+ }
+ if policy == nil || project == nil || project.OrganizationId == nil || *project.OrganizationId != policy.OrganizationId {
+ return "Escalation policy not found in this project's organization.", nil
+ }
+ return "", nil
+}
+
+// validateChannelConfig routes to the right validation for the channel type.
+// Returns a 422 message, or an empty string when the config is valid.
+func validateChannelConfig(tx *sql.Tx, projectId uuid.UUID, channelType string, config json.RawMessage) (string, error) {
+ if channelType == "escalation" {
+ return validateEscalationChannelConfig(tx, projectId, config)
+ }
+ adapter, err := notifications.NewAdapter(channelType, config)
+ if err != nil {
+ return err.Error(), nil
+ }
+ if err := adapter.Validate(); err != nil {
+ return err.Error(), nil
+ }
+ return "", nil
}
func (ctrl *notificationChannelController) Create(ctx *gin.Context) {
@@ -69,17 +110,15 @@ func (ctrl *notificationChannelController) Create(ctx *gin.Context) {
return
}
if !validChannelTypes[req.ChannelType] {
- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Channel type must be one of: email, webhook, slack, github, pushover, telegram."})
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Channel type must be one of: email, webhook, slack, github, pushover, telegram, escalation."})
return
}
- adapter, err := notifications.NewAdapter(req.ChannelType, req.Config)
- if err != nil {
- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ if message, err := validateChannelConfig(db.GetTx(ctx), projectId, req.ChannelType, req.Config); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate channel config: %w", err))
return
- }
- if err := adapter.Validate(); err != nil {
- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
return
}
@@ -142,17 +181,15 @@ func (ctrl *notificationChannelController) Update(ctx *gin.Context) {
return
}
if !validChannelTypes[req.ChannelType] {
- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Channel type must be one of: email, webhook, slack, github, pushover, telegram."})
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Channel type must be one of: email, webhook, slack, github, pushover, telegram, escalation."})
return
}
- adapter, err := notifications.NewAdapter(req.ChannelType, req.Config)
- if err != nil {
- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ if message, err := validateChannelConfig(db.GetTx(ctx), projectId, req.ChannelType, req.Config); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate channel config: %w", err))
return
- }
- if err := adapter.Validate(); err != nil {
- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
return
}
@@ -239,6 +276,27 @@ func (ctrl *notificationChannelController) Test(ctx *gin.Context) {
return
}
+ // Testing an escalation channel opens a real page so the whole loop is
+ // exercised; the dialog warns that it pages the on-call responder.
+ if channel.ChannelType == "escalation" {
+ policyId := oncall.EscalationChannelPolicyId(json.RawMessage(channel.Config))
+ if policyId == 0 {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "An escalation policy is required."})
+ return
+ }
+ opened, err := oncall.OpenTestPage(policyId, projectId, channel.Id, channel.Name)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("test page failed: %w", err))
+ return
+ }
+ if !opened {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "A test page for this channel is still open. Resolve it before testing again."})
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"success": true})
+ return
+ }
+
adapter, err := notifications.NewAdapter(channel.ChannelType, channel.Config)
if err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})
@@ -254,7 +312,7 @@ func (ctrl *notificationChannelController) Test(ctx *gin.Context) {
}
if err := adapter.Send(ctx.Request.Context(), testMsg); err != nil {
- ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("test notification failed: %w", err))
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Test notification failed: " + err.Error()})
return
}
diff --git a/backend/app/controllers/oncall.controller.go b/backend/app/controllers/oncall.controller.go
new file mode 100644
index 00000000..a238ed1f
--- /dev/null
+++ b/backend/app/controllers/oncall.controller.go
@@ -0,0 +1,583 @@
+package controllers
+
+import (
+ "database/sql"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+
+ "github.com/gin-gonic/gin"
+ traceway "go.tracewayapp.com"
+)
+
+type oncallController struct{}
+
+var OncallController = oncallController{}
+
+type scheduleRequest struct {
+ TeamId int `json:"teamId"`
+ Name string `json:"name"`
+ Description string `json:"description"`
+ Timezone string `json:"timezone"`
+ Definition models.JSONText `json:"definition"`
+}
+
+type createOverrideRequest struct {
+ UserId int `json:"userId"`
+ StartAt time.Time `json:"startAt"`
+ EndAt time.Time `json:"endAt"`
+}
+
+type scheduleUserInfo struct {
+ Name string `json:"name"`
+ Email string `json:"email"`
+}
+
+func (c *oncallController) ListSchedules(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ schedules, err := transactional.OncallScheduleRepository.ListByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list schedules: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"schedules": schedules})
+}
+
+func (c *oncallController) CreateSchedule(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+ userId := middleware.GetUserId(ctx)
+
+ var request scheduleRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ message, definition, tzName := c.validateScheduleRequest(ctx, organizationId, &request, 0)
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ if definition == nil {
+ return
+ }
+
+ now := time.Now().UTC()
+ schedule := &models.OncallSchedule{
+ OrganizationId: organizationId,
+ TeamId: request.TeamId,
+ Name: request.Name,
+ Description: request.Description,
+ Timezone: tzName,
+ Definition: definition,
+ CreatedBy: &userId,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+ id, err := transactional.OncallScheduleRepository.Create(tx, schedule)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to create schedule: %w", err))
+ return
+ }
+ schedule.Id = id
+ ctx.JSON(http.StatusCreated, schedule)
+}
+
+func (c *oncallController) GetSchedule(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ schedule, ok := c.loadSchedule(ctx, organizationId)
+ if !ok {
+ return
+ }
+ now := time.Now().UTC()
+ overrides, err := transactional.OncallOverrideRepository.ListForRange(tx, schedule.Id, now, now.AddDate(0, 0, oncall.MaxOverrideDurationDays))
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list overrides: %w", err))
+ return
+ }
+ if overrides == nil {
+ overrides = []*models.OncallOverride{}
+ }
+ ctx.JSON(http.StatusOK, gin.H{"schedule": schedule, "overrides": overrides})
+}
+
+func (c *oncallController) UpdateSchedule(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ schedule, ok := c.loadSchedule(ctx, organizationId)
+ if !ok {
+ return
+ }
+ var request scheduleRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ message, definition, tzName := c.validateScheduleRequest(ctx, organizationId, &request, schedule.Id)
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ if definition == nil {
+ return
+ }
+
+ schedule.TeamId = request.TeamId
+ schedule.Name = request.Name
+ schedule.Description = request.Description
+ schedule.Timezone = tzName
+ schedule.Definition = definition
+ schedule.UpdatedAt = time.Now().UTC()
+ if err := transactional.OncallScheduleRepository.Update(tx, schedule); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to update schedule: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, schedule)
+}
+
+func (c *oncallController) DeleteSchedule(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ schedule, ok := c.loadSchedule(ctx, organizationId)
+ if !ok {
+ return
+ }
+
+ referencing, err := oncall.PoliciesReferencing(tx, organizationId, oncall.TargetSchedule, schedule.Id)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check referencing policies: %w", err))
+ return
+ }
+ if len(referencing) > 0 {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "This schedule is used by escalation policy(ies): " + strings.Join(referencing, ", ") + ". Remove those steps first."})
+ return
+ }
+
+ if err := transactional.OncallScheduleRepository.Delete(tx, schedule.Id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete schedule: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Schedule deleted"})
+}
+
+func (c *oncallController) Timeline(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ schedule, ok := c.loadSchedule(ctx, organizationId)
+ if !ok {
+ return
+ }
+ from, err := time.Parse(time.RFC3339, ctx.Query("from"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid or missing from parameter"})
+ return
+ }
+ to, err := time.Parse(time.RFC3339, ctx.Query("to"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid or missing to parameter"})
+ return
+ }
+ from = from.UTC()
+ to = to.UTC()
+ if !from.Before(to) {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "from must be before to"})
+ return
+ }
+ if to.Sub(from) > time.Duration(oncall.MaxTimelineRangeDays)*24*time.Hour {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "The timeline range can span at most 62 days"})
+ return
+ }
+
+ tz, err := time.LoadLocation(schedule.Timezone)
+ if err != nil {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("schedule %d has an unloadable timezone %q, rendering in UTC: %w", schedule.Id, schedule.Timezone, err))
+ tz = time.UTC
+ }
+ definition, err := oncall.ParseDefinition(schedule.Definition)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("stored schedule definition failed to parse (schedule=%d): %w", schedule.Id, err))
+ return
+ }
+ overrides, err := transactional.OncallOverrideRepository.ListForRange(tx, schedule.Id, from, to)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list overrides: %w", err))
+ return
+ }
+
+ type layerTimeline struct {
+ Id string `json:"id"`
+ Name string `json:"name"`
+ Shifts []oncall.Shift `json:"shifts"`
+ }
+ layers := make([]layerTimeline, 0, len(definition.Layers))
+ userIds := map[int]bool{}
+ for i := range definition.Layers {
+ layer := &definition.Layers[i]
+ shifts := oncall.ResolveLayerRange(layer, tz, from, to)
+ if shifts == nil {
+ shifts = []oncall.Shift{}
+ }
+ for _, shift := range shifts {
+ userIds[shift.UserId] = true
+ }
+ layers = append(layers, layerTimeline{Id: layer.Id, Name: layer.Name, Shifts: shifts})
+ }
+ final := oncall.ResolveRange(definition, tz, overrides, from, to)
+ if final == nil {
+ final = []oncall.Shift{}
+ }
+ for _, shift := range final {
+ userIds[shift.UserId] = true
+ }
+
+ members, err := transactional.OrganizationRepository.GetMembersWithDetails(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load members: %w", err))
+ return
+ }
+ memberById := map[int]*models.OrganizationMember{}
+ for _, member := range members {
+ memberById[member.Id] = member
+ }
+ users := map[string]*scheduleUserInfo{}
+ for userId := range userIds {
+ if member, ok := memberById[userId]; ok {
+ users[strconv.Itoa(userId)] = &scheduleUserInfo{Name: member.Name, Email: member.Email}
+ } else {
+ users[strconv.Itoa(userId)] = nil
+ }
+ }
+
+ ctx.JSON(http.StatusOK, gin.H{
+ "schedule": gin.H{"id": schedule.Id, "name": schedule.Name, "timezone": schedule.Timezone, "teamId": schedule.TeamId},
+ "from": from,
+ "to": to,
+ "layers": layers,
+ "final": final,
+ "users": users,
+ })
+}
+
+func (c *oncallController) CreateOverride(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+ userId := middleware.GetUserId(ctx)
+
+ schedule, ok := c.loadSchedule(ctx, organizationId)
+ if !ok {
+ return
+ }
+ var request createOverrideRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if !request.StartAt.Before(request.EndAt) {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "The override must end after it starts."})
+ return
+ }
+ if request.EndAt.Sub(request.StartAt) > time.Duration(oncall.MaxOverrideDurationDays)*24*time.Hour {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "An override can last at most 30 days."})
+ return
+ }
+ role, err := transactional.OrganizationRepository.GetUserRole(tx, organizationId, request.UserId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check override user: %w", err))
+ return
+ }
+ if role == "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "The covering user must be a member of the organization."})
+ return
+ }
+
+ override := &models.OncallOverride{
+ ScheduleId: schedule.Id,
+ UserId: request.UserId,
+ StartAt: request.StartAt.UTC(),
+ EndAt: request.EndAt.UTC(),
+ CreatedBy: &userId,
+ CreatedAt: time.Now().UTC(),
+ }
+ id, err := transactional.OncallOverrideRepository.Create(tx, override)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to create override: %w", err))
+ return
+ }
+ override.Id = id
+ ctx.JSON(http.StatusCreated, override)
+}
+
+func (c *oncallController) DeleteOverride(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+ userId := middleware.GetUserId(ctx)
+ role := middleware.GetUserOrgRole(ctx)
+
+ schedule, ok := c.loadSchedule(ctx, organizationId)
+ if !ok {
+ return
+ }
+ overrideId, err := strconv.Atoi(ctx.Param("overrideId"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid override ID"})
+ return
+ }
+ override, err := transactional.OncallOverrideRepository.FindById(tx, overrideId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load override: %w", err))
+ return
+ }
+ if override == nil || override.ScheduleId != schedule.Id {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Override not found"})
+ return
+ }
+
+ isAdmin := role == "owner" || role == "admin"
+ isCreator := override.CreatedBy != nil && *override.CreatedBy == userId
+ isCovering := override.UserId == userId
+ if !isAdmin && !isCreator && !isCovering {
+ ctx.JSON(http.StatusForbidden, gin.H{"error": "Only the creator, the covering user, or an admin can delete an override"})
+ return
+ }
+ if err := transactional.OncallOverrideRepository.Delete(tx, override.Id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete override: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Override deleted"})
+}
+
+// Now is the org-wide overview: per team, per schedule, who is on call and who
+// is next.
+func (c *oncallController) Now(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ teams, err := transactional.TeamRepository.ListByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list teams: %w", err))
+ return
+ }
+ schedules, err := transactional.OncallScheduleRepository.ListByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list schedules: %w", err))
+ return
+ }
+ members, err := transactional.OrganizationRepository.GetMembersWithDetails(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load members: %w", err))
+ return
+ }
+ memberById := map[int]*models.OrganizationMember{}
+ for _, member := range members {
+ memberById[member.Id] = member
+ }
+
+ now := time.Now().UTC()
+ allOverrides, err := transactional.OncallOverrideRepository.ListForRangeByOrganization(tx, organizationId, now, now.AddDate(0, 0, 35))
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list overrides: %w", err))
+ return
+ }
+ overridesBySchedule := map[int][]*models.OncallOverride{}
+ for _, override := range allOverrides {
+ overridesBySchedule[override.ScheduleId] = append(overridesBySchedule[override.ScheduleId], override)
+ }
+
+ type scheduleNow struct {
+ Id int `json:"id"`
+ Name string `json:"name"`
+ Oncall []oncall.OncallUser `json:"oncall"`
+ Until *time.Time `json:"until"`
+ NextUp *oncall.OncallUser `json:"nextUp"`
+ NextAt *time.Time `json:"nextAt"`
+ }
+ type teamNow struct {
+ Team *models.TeamWithCounts `json:"team"`
+ Schedules []scheduleNow `json:"schedules"`
+ }
+
+ schedulesByTeam := map[int][]*models.OncallSchedule{}
+ for _, schedule := range schedules {
+ schedulesByTeam[schedule.TeamId] = append(schedulesByTeam[schedule.TeamId], schedule)
+ }
+
+ response := make([]teamNow, 0, len(teams))
+ for _, team := range teams {
+ entry := teamNow{Team: team, Schedules: []scheduleNow{}}
+ for _, schedule := range schedulesByTeam[team.Id] {
+ tz, err := time.LoadLocation(schedule.Timezone)
+ if err != nil {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("schedule %d has an unloadable timezone %q, rendering in UTC: %w", schedule.Id, schedule.Timezone, err))
+ tz = time.UTC
+ }
+ definition, err := oncall.ParseDefinition(schedule.Definition)
+ if err != nil {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("stored schedule definition failed to parse (schedule=%d): %w", schedule.Id, err))
+ continue
+ }
+ overrides := overridesBySchedule[schedule.Id]
+ item := scheduleNow{Id: schedule.Id, Name: schedule.Name, Oncall: []oncall.OncallUser{}}
+ for _, onCallUserId := range oncall.ResolveAt(definition, tz, overrides, now) {
+ if member, ok := memberById[onCallUserId]; ok {
+ item.Oncall = append(item.Oncall, oncall.OncallUser{UserId: onCallUserId, Name: member.Name, Email: member.Email})
+ }
+ }
+ current, next := oncall.CurrentAndNext(definition, tz, overrides, now)
+ if current != nil {
+ until := current.End
+ item.Until = &until
+ }
+ if next != nil {
+ if member, ok := memberById[next.UserId]; ok {
+ item.NextUp = &oncall.OncallUser{UserId: next.UserId, Name: member.Name, Email: member.Email}
+ nextAt := next.Start
+ item.NextAt = &nextAt
+ }
+ }
+ entry.Schedules = append(entry.Schedules, item)
+ }
+ response = append(response, entry)
+ }
+ ctx.JSON(http.StatusOK, gin.H{"teams": response})
+}
+
+// Current is the project-scoped ownership seam: the owning team and current
+// on-call for the project, consumed by the issue page and the escalation
+// engine's UI.
+func (c *oncallController) Current(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ result, err := oncall.CurrentOnCallForProject(tx, projectId, time.Now().UTC())
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to resolve current on-call: %w", err))
+ return
+ }
+ if result == nil {
+ ctx.JSON(http.StatusOK, gin.H{"team": nil, "schedules": []oncall.ScheduleRef{}, "oncall": []oncall.OncallUser{}})
+ return
+ }
+ ctx.JSON(http.StatusOK, result)
+}
+
+func (c *oncallController) loadSchedule(ctx *gin.Context, organizationId int) (*models.OncallSchedule, bool) {
+ tx := db.GetTx(ctx)
+ scheduleId, err := strconv.Atoi(ctx.Param("scheduleId"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid schedule ID"})
+ return nil, false
+ }
+ schedule, err := transactional.OncallScheduleRepository.FindById(tx, scheduleId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load schedule: %w", err))
+ return nil, false
+ }
+ if schedule == nil || schedule.OrganizationId != organizationId {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Schedule not found"})
+ return nil, false
+ }
+ return schedule, true
+}
+
+// validateScheduleRequest validates the shared create/update payload. It
+// returns a 422 message, the normalized definition, and the timezone to store.
+// A nil definition together with an empty message means a 500 was already
+// written.
+func (c *oncallController) validateScheduleRequest(ctx *gin.Context, organizationId int, request *scheduleRequest, currentScheduleId int) (string, models.JSONText, string) {
+ tx := db.GetTx(ctx)
+
+ if request.Name == "" {
+ return "A schedule name is required.", nil, ""
+ }
+ if len(request.Name) > 100 {
+ return "The schedule name can be at most 100 characters.", nil, ""
+ }
+ team, err := transactional.TeamRepository.FindById(tx, request.TeamId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load team: %w", err))
+ return "", nil, ""
+ }
+ if team == nil || team.OrganizationId != organizationId {
+ return "The schedule must belong to a team in this organization.", nil, ""
+ }
+ existing, err := transactional.OncallScheduleRepository.FindByOrganizationAndName(tx, organizationId, request.Name)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check schedule name: %w", err))
+ return "", nil, ""
+ }
+ if existing != nil && existing.Id != currentScheduleId {
+ return "A schedule with this name already exists.", nil, ""
+ }
+
+ tzName := request.Timezone
+ if tzName == "" {
+ organization, err := transactional.OrganizationRepository.FindById(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load organization: %w", err))
+ return "", nil, ""
+ }
+ if organization != nil && organization.Timezone != "" {
+ tzName = organization.Timezone
+ } else {
+ tzName = "UTC"
+ }
+ }
+ if _, err := oncall.LoadTimezone(tzName); err != nil {
+ return err.Error(), nil, ""
+ }
+
+ definition, err := oncall.ParseDefinition(request.Definition)
+ if err != nil {
+ return err.Error(), nil, ""
+ }
+ memberMessage, err := c.checkLayerMembers(tx, organizationId, definition)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate schedule members: %w", err))
+ return "", nil, ""
+ }
+ if memberMessage != "" {
+ return memberMessage, nil, ""
+ }
+ normalized, err := oncall.MarshalDefinition(definition)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to marshal schedule definition: %w", err))
+ return "", nil, ""
+ }
+ return "", models.JSONText(normalized), tzName
+}
+
+func (c *oncallController) checkLayerMembers(tx *sql.Tx, organizationId int, definition *models.OncallScheduleDefinition) (string, error) {
+ members, err := transactional.OrganizationRepository.GetMembersWithDetails(tx, organizationId)
+ if err != nil {
+ return "", err
+ }
+ memberSet := make(map[int]bool, len(members))
+ for _, member := range members {
+ memberSet[member.Id] = true
+ }
+ for _, layer := range definition.Layers {
+ for _, layerUserId := range layer.UserIds {
+ if !memberSet[layerUserId] {
+ return "Layer \"" + layer.Name + "\" includes someone who is not a member of the organization.", nil
+ }
+ }
+ }
+ return "", nil
+}
diff --git a/backend/app/controllers/page.controller.go b/backend/app/controllers/page.controller.go
new file mode 100644
index 00000000..c79693e7
--- /dev/null
+++ b/backend/app/controllers/page.controller.go
@@ -0,0 +1,190 @@
+package controllers
+
+import (
+ "net/http"
+ "strconv"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+
+ "github.com/gin-gonic/gin"
+ traceway "go.tracewayapp.com"
+)
+
+type pageController struct{}
+
+var PageController = pageController{}
+
+type listPagesRequest struct {
+ Status string `json:"status"`
+ Pagination PaginationParams `json:"pagination" binding:"required"`
+}
+
+var validPageStatusFilters = map[string]bool{
+ "": true, "active": true,
+ models.PageStatusOpen: true, models.PageStatusAcknowledged: true, models.PageStatusResolved: true,
+}
+
+func (c *pageController) List(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ var request listPagesRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if !validPageStatusFilters[request.Status] {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid status filter"})
+ return
+ }
+
+ total, err := transactional.PageRepository.CountByProject(tx, projectId, request.Status)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to count pages: %w", err))
+ return
+ }
+ offset := (request.Pagination.Page - 1) * request.Pagination.PageSize
+ pages, err := transactional.PageRepository.FindByProject(tx, projectId, request.Status, request.Pagination.PageSize, offset)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list pages: %w", err))
+ return
+ }
+ if pages == nil {
+ pages = []*models.Page{}
+ }
+
+ totalPages := int64(0)
+ if request.Pagination.PageSize > 0 {
+ totalPages = (int64(total) + int64(request.Pagination.PageSize) - 1) / int64(request.Pagination.PageSize)
+ }
+ ctx.JSON(http.StatusOK, PaginatedResponse[*models.Page]{
+ Data: pages,
+ Pagination: Pagination{
+ Page: request.Pagination.Page,
+ PageSize: request.Pagination.PageSize,
+ Total: int64(total),
+ TotalPages: totalPages,
+ },
+ })
+}
+
+func (c *pageController) Get(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ page, ok := c.loadPage(ctx)
+ if !ok {
+ return
+ }
+ notifications, err := transactional.PageNotificationRepository.FindByPage(tx, page.Id)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load page notifications: %w", err))
+ return
+ }
+ if notifications == nil {
+ notifications = []*models.PageNotification{}
+ }
+
+ users := map[string]string{}
+ members, err := transactional.OrganizationRepository.GetMembersWithDetails(tx, page.OrganizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load members: %w", err))
+ return
+ }
+ for _, member := range members {
+ users[strconv.Itoa(member.Id)] = member.Name
+ }
+
+ ctx.JSON(http.StatusOK, gin.H{"page": page, "notifications": notifications, "users": users})
+}
+
+// Acknowledge deliberately has no write-access gate beyond project read
+// access: acking is incident response, and a paged responder must never be
+// blocked by a readonly role.
+func (c *pageController) Acknowledge(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ userId := middleware.GetUserId(ctx)
+ page, ok := c.loadPage(ctx)
+ if !ok {
+ return
+ }
+ acknowledged, err := oncall.AcknowledgePage(tx, page.Id, &userId, oncall.AckViaDashboard, time.Now().UTC())
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to acknowledge page: %w", err))
+ return
+ }
+ if !acknowledged {
+ current, err := transactional.PageRepository.FindById(tx, page.Id)
+ if err != nil || current == nil {
+ ctx.JSON(http.StatusConflict, gin.H{"error": "Page is not open"})
+ return
+ }
+ ctx.JSON(http.StatusConflict, gin.H{"error": "Page is not open", "status": current.Status})
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Page acknowledged"})
+}
+
+func (c *pageController) Resolve(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ userId := middleware.GetUserId(ctx)
+ page, ok := c.loadPage(ctx)
+ if !ok {
+ return
+ }
+ resolved, err := oncall.ResolvePage(tx, page.Id, userId, time.Now().UTC())
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to resolve page: %w", err))
+ return
+ }
+ if !resolved {
+ ctx.JSON(http.StatusConflict, gin.H{"error": "Page is already resolved", "status": models.PageStatusResolved})
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Page resolved"})
+}
+
+func (c *pageController) OpenCount(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ count, err := transactional.PageRepository.CountOpenByProject(tx, projectId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to count open pages: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"count": count})
+}
+
+func (c *pageController) loadPage(ctx *gin.Context) (*models.Page, bool) {
+ tx := db.GetTx(ctx)
+ projectId, err := middleware.GetProjectId(ctx)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return nil, false
+ }
+ pageId, err := strconv.Atoi(ctx.Param("id"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid page ID"})
+ return nil, false
+ }
+ page, err := transactional.PageRepository.FindById(tx, pageId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load page: %w", err))
+ return nil, false
+ }
+ if page == nil || page.ProjectId != projectId {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Page not found"})
+ return nil, false
+ }
+ return page, true
+}
diff --git a/backend/app/controllers/project.controller.go b/backend/app/controllers/project.controller.go
index f4c1dcc3..71db6be0 100644
--- a/backend/app/controllers/project.controller.go
+++ b/backend/app/controllers/project.controller.go
@@ -8,6 +8,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
"github.com/tracewayapp/traceway/backend/app/profiling"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
"net/http"
@@ -313,6 +314,9 @@ func (p projectController) DeleteProject(c *gin.Context) {
if project.Name != request.Name {
return false, nil
}
+ if err := outbox.CancelForProject(tx, projectId); err != nil {
+ return false, err
+ }
if err := transactional.ProjectRepository.Delete(tx, projectId); err != nil {
return false, err
}
diff --git a/backend/app/controllers/routes.go b/backend/app/controllers/routes.go
index 9a44c34c..05322a96 100644
--- a/backend/app/controllers/routes.go
+++ b/backend/app/controllers/routes.go
@@ -215,6 +215,55 @@ func RegisterControllers(router *gin.RouterGroup) {
router.POST("/notification-history", middleware.UseAppAuth, middleware.RequireProjectAccess, NotificationHistoryController.List)
+ router.GET("/organizations/:organizationId/teams", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, TeamController.List)
+ router.POST("/organizations/:organizationId/teams", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, TeamController.Create)
+ router.PUT("/organizations/:organizationId/teams/:teamId", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, TeamController.Update)
+ router.DELETE("/organizations/:organizationId/teams/:teamId", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, TeamController.Delete)
+ router.PUT("/organizations/:organizationId/teams/:teamId/members", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, TeamController.SetMembers)
+ router.PUT("/organizations/:organizationId/teams/:teamId/projects", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, TeamController.SetProjects)
+
+ router.GET("/organizations/:organizationId/schedules", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, OncallController.ListSchedules)
+ router.POST("/organizations/:organizationId/schedules", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, OncallController.CreateSchedule)
+ router.GET("/organizations/:organizationId/schedules/:scheduleId", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, OncallController.GetSchedule)
+ router.PUT("/organizations/:organizationId/schedules/:scheduleId", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, OncallController.UpdateSchedule)
+ router.DELETE("/organizations/:organizationId/schedules/:scheduleId", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, OncallController.DeleteSchedule)
+ router.GET("/organizations/:organizationId/schedules/:scheduleId/timeline", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, OncallController.Timeline)
+ router.POST("/organizations/:organizationId/schedules/:scheduleId/overrides", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, OncallController.CreateOverride)
+ router.DELETE("/organizations/:organizationId/schedules/:scheduleId/overrides/:overrideId", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, OncallController.DeleteOverride)
+ router.GET("/organizations/:organizationId/oncall/now", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, OncallController.Now)
+ router.GET("/oncall/current", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, OncallController.Current)
+
+ router.GET("/escalation-policies", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, EscalationPolicyController.ListForProject)
+ router.GET("/organizations/:organizationId/escalation-policies", middleware.UseAppAuth, middleware.RequireOrganizationAccess, middleware.Transactional, EscalationPolicyController.ListForOrganization)
+ router.POST("/organizations/:organizationId/escalation-policies", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, EscalationPolicyController.Create)
+ router.PUT("/organizations/:organizationId/escalation-policies/:id", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, EscalationPolicyController.Update)
+ router.DELETE("/organizations/:organizationId/escalation-policies/:id", middleware.UseAppAuth, middleware.RequireAdminAccess, middleware.Transactional, EscalationPolicyController.Delete)
+
+ router.POST("/pages", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, PageController.List)
+ router.GET("/pages/open-count", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, PageController.OpenCount)
+ router.GET("/pages/:id", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, PageController.Get)
+ router.POST("/pages/:id/acknowledge", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, PageController.Acknowledge)
+ router.POST("/pages/:id/resolve", middleware.UseAppAuth, middleware.RequireProjectAccess, middleware.Transactional, PageController.Resolve)
+
+ router.GET("/contact-methods", middleware.UseAppAuth, middleware.Transactional, ContactMethodController.List)
+ router.POST("/contact-methods", middleware.UseAppAuth, middleware.RateLimitPerUser(20, 10*time.Minute), middleware.Transactional, ContactMethodController.Create)
+ router.PUT("/contact-methods/:id", middleware.UseAppAuth, middleware.RateLimitPerUser(30, 10*time.Minute), middleware.Transactional, ContactMethodController.Update)
+ router.DELETE("/contact-methods/:id", middleware.UseAppAuth, middleware.Transactional, ContactMethodController.Delete)
+ // Rate limited like the mutating routes: a contact method can point at any
+ // address/number/webhook, so an uncapped test button is an outbound relay.
+ router.POST("/contact-methods/:id/test", middleware.UseAppAuth, middleware.RateLimitPerUser(10, 10*time.Minute), ContactMethodController.Test)
+ // No Transactional: Verify answers 422 for a wrong code, and the consumed
+ // attempt must still commit, so it manages its own transactions (the same
+ // reason the OAuth grant endpoints do).
+ router.POST("/contact-methods/:id/verify", middleware.UseAppAuth, middleware.RateLimitPerUser(10, time.Minute), ContactMethodController.Verify)
+ router.POST("/contact-methods/:id/resend-code", middleware.UseAppAuth, middleware.RateLimitPerUser(3, 5*time.Minute), middleware.Transactional, ContactMethodController.ResendCode)
+
+ router.GET("/user-notification-rules", middleware.UseAppAuth, middleware.Transactional, UserNotificationRuleController.Get)
+ router.PUT("/user-notification-rules", middleware.UseAppAuth, middleware.Transactional, UserNotificationRuleController.Put)
+
+ router.GET("/ack/:token", middleware.RateLimitPerIP(30, time.Minute), middleware.Transactional, AckController.Get)
+ router.POST("/ack/:token", middleware.RateLimitPerIP(10, time.Minute), middleware.Transactional, AckController.Acknowledge)
+
for _, register := range ExtensionRoutes {
register(router)
}
diff --git a/backend/app/controllers/team.controller.go b/backend/app/controllers/team.controller.go
new file mode 100644
index 00000000..ae23027a
--- /dev/null
+++ b/backend/app/controllers/team.controller.go
@@ -0,0 +1,425 @@
+package controllers
+
+import (
+ "database/sql"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ traceway "go.tracewayapp.com"
+)
+
+type teamController struct{}
+
+var TeamController = teamController{}
+
+type teamResponse struct {
+ *models.TeamWithCounts
+ Members []*models.TeamMemberWithUser `json:"members"`
+ Projects []teamProjectResponse `json:"projects"`
+}
+
+type teamProjectResponse struct {
+ ProjectId uuid.UUID `json:"projectId"`
+ Name string `json:"name"`
+}
+
+type createTeamRequest struct {
+ Name string `json:"name"`
+ Description string `json:"description"`
+ MemberUserIds []int `json:"memberUserIds"`
+ ProjectIds []uuid.UUID `json:"projectIds"`
+}
+
+// MemberUserIds and ProjectIds are optional; when present the whole edit
+// applies in one transaction.
+type updateTeamRequest struct {
+ Name string `json:"name"`
+ Description string `json:"description"`
+ MemberUserIds *[]int `json:"memberUserIds"`
+ ProjectIds *[]uuid.UUID `json:"projectIds"`
+}
+
+type setTeamMembersRequest struct {
+ UserIds []int `json:"userIds"`
+}
+
+type setTeamProjectsRequest struct {
+ ProjectIds []uuid.UUID `json:"projectIds"`
+}
+
+func (c *teamController) List(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ teams, err := transactional.TeamRepository.ListByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list teams: %w", err))
+ return
+ }
+ members, err := transactional.TeamRepository.ListMembersWithUsersByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list team members: %w", err))
+ return
+ }
+ projects, err := transactional.TeamRepository.ListProjectsByOrganization(tx, organizationId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to list team projects: %w", err))
+ return
+ }
+
+ membersByTeam := map[int][]*models.TeamMemberWithUser{}
+ for _, member := range members {
+ membersByTeam[member.TeamId] = append(membersByTeam[member.TeamId], member)
+ }
+ projectsByTeam := map[int][]teamProjectResponse{}
+ for _, project := range projects {
+ projectsByTeam[project.TeamId] = append(projectsByTeam[project.TeamId], teamProjectResponse{ProjectId: project.ProjectId, Name: project.Name})
+ }
+
+ response := make([]teamResponse, 0, len(teams))
+ for _, team := range teams {
+ entry := teamResponse{TeamWithCounts: team, Members: membersByTeam[team.Id], Projects: projectsByTeam[team.Id]}
+ if entry.Members == nil {
+ entry.Members = []*models.TeamMemberWithUser{}
+ }
+ if entry.Projects == nil {
+ entry.Projects = []teamProjectResponse{}
+ }
+ response = append(response, entry)
+ }
+ ctx.JSON(http.StatusOK, gin.H{"teams": response})
+}
+
+func (c *teamController) Create(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ var request createTeamRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if message := validateTeamName(request.Name); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ existing, err := transactional.TeamRepository.FindByOrganizationAndName(tx, organizationId, request.Name)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check team name: %w", err))
+ return
+ }
+ if existing != nil {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "A team with this name already exists."})
+ return
+ }
+ if message, err := c.checkMembersInOrg(tx, organizationId, request.MemberUserIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate team members: %w", err))
+ return
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+
+ now := time.Now().UTC()
+ team := &models.Team{
+ OrganizationId: organizationId,
+ Name: request.Name,
+ Description: request.Description,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+ teamId, err := transactional.TeamRepository.Create(tx, team)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to create team: %w", err))
+ return
+ }
+ team.Id = teamId
+
+ if len(request.MemberUserIds) > 0 {
+ if err := transactional.TeamRepository.SetMembers(tx, teamId, request.MemberUserIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to set team members: %w", err))
+ return
+ }
+ }
+ if len(request.ProjectIds) > 0 {
+ if message, err := c.checkProjectsAssignable(tx, organizationId, teamId, request.ProjectIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate team projects: %w", err))
+ return
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ if err := transactional.TeamRepository.SetProjects(tx, teamId, request.ProjectIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to set team projects: %w", err))
+ return
+ }
+ }
+ ctx.JSON(http.StatusCreated, team)
+}
+
+func (c *teamController) Update(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ team, ok := c.loadTeam(ctx, organizationId)
+ if !ok {
+ return
+ }
+ var request updateTeamRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if message := validateTeamName(request.Name); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ existing, err := transactional.TeamRepository.FindByOrganizationAndName(tx, organizationId, request.Name)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check team name: %w", err))
+ return
+ }
+ if existing != nil && existing.Id != team.Id {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "A team with this name already exists."})
+ return
+ }
+
+ if request.MemberUserIds != nil {
+ if message, err := c.checkMembersInOrg(tx, organizationId, *request.MemberUserIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate team members: %w", err))
+ return
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ }
+ if request.ProjectIds != nil {
+ if message, err := c.checkProjectsAssignable(tx, organizationId, team.Id, *request.ProjectIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate team projects: %w", err))
+ return
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ }
+
+ team.Name = request.Name
+ team.Description = request.Description
+ team.UpdatedAt = time.Now().UTC()
+ if err := transactional.TeamRepository.Update(tx, team); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to update team: %w", err))
+ return
+ }
+ if request.MemberUserIds != nil {
+ if err := transactional.TeamRepository.SetMembers(tx, team.Id, *request.MemberUserIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to set team members: %w", err))
+ return
+ }
+ }
+ if request.ProjectIds != nil {
+ if err := transactional.TeamRepository.SetProjects(tx, team.Id, *request.ProjectIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to set team projects: %w", err))
+ return
+ }
+ }
+ ctx.JSON(http.StatusOK, team)
+}
+
+func (c *teamController) Delete(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ team, ok := c.loadTeam(ctx, organizationId)
+ if !ok {
+ return
+ }
+
+ message, err := c.checkPolicyReferences(tx, organizationId, team.Id)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to check referencing policies: %w", err))
+ return
+ }
+ if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+
+ if err := transactional.TeamRepository.Delete(tx, team.Id); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to delete team: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Team deleted"})
+}
+
+func (c *teamController) SetMembers(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ team, ok := c.loadTeam(ctx, organizationId)
+ if !ok {
+ return
+ }
+ var request setTeamMembersRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if message, err := c.checkMembersInOrg(tx, organizationId, request.UserIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate team members: %w", err))
+ return
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ if err := transactional.TeamRepository.SetMembers(tx, team.Id, request.UserIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to set team members: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Members updated"})
+}
+
+func (c *teamController) SetProjects(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ organizationId := middleware.GetOrganizationId(ctx)
+
+ team, ok := c.loadTeam(ctx, organizationId)
+ if !ok {
+ return
+ }
+ var request setTeamProjectsRequest
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if message, err := c.checkProjectsAssignable(tx, organizationId, team.Id, request.ProjectIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to validate team projects: %w", err))
+ return
+ } else if message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ if err := transactional.TeamRepository.SetProjects(tx, team.Id, request.ProjectIds); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to set team projects: %w", err))
+ return
+ }
+ ctx.JSON(http.StatusOK, gin.H{"message": "Projects updated"})
+}
+
+func (c *teamController) loadTeam(ctx *gin.Context, organizationId int) (*models.Team, bool) {
+ tx := db.GetTx(ctx)
+ teamId, err := strconv.Atoi(ctx.Param("teamId"))
+ if err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid team ID"})
+ return nil, false
+ }
+ team, err := transactional.TeamRepository.FindById(tx, teamId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load team: %w", err))
+ return nil, false
+ }
+ if team == nil || team.OrganizationId != organizationId {
+ ctx.JSON(http.StatusNotFound, gin.H{"error": "Team not found"})
+ return nil, false
+ }
+ return team, true
+}
+
+// checkPolicyReferences returns a user-facing message when an escalation policy
+// targets the team or one of its schedules.
+func (c *teamController) checkPolicyReferences(tx *sql.Tx, organizationId int, teamId int) (string, error) {
+ referencing, err := oncall.PoliciesReferencing(tx, organizationId, oncall.TargetTeam, teamId)
+ if err != nil {
+ return "", err
+ }
+ if len(referencing) > 0 {
+ return "This team is used by escalation policy(ies): " + strings.Join(referencing, ", ") + ". Remove those steps first.", nil
+ }
+
+ schedules, err := transactional.OncallScheduleRepository.ListByTeam(tx, teamId)
+ if err != nil {
+ return "", err
+ }
+ scheduleIds := make([]int, 0, len(schedules))
+ for _, schedule := range schedules {
+ scheduleIds = append(scheduleIds, schedule.Id)
+ }
+ referencing, err = oncall.PoliciesReferencing(tx, organizationId, oncall.TargetSchedule, scheduleIds...)
+ if err != nil {
+ return "", err
+ }
+ if len(referencing) > 0 {
+ return "Deleting this team would delete schedules used by escalation policy(ies): " + strings.Join(referencing, ", ") + ". Remove those steps first.", nil
+ }
+ return "", nil
+}
+
+// checkMembersInOrg returns a user-facing message when a userId is not a
+// member of the organization or appears twice.
+func (c *teamController) checkMembersInOrg(tx *sql.Tx, organizationId int, userIds []int) (string, error) {
+ members, err := transactional.OrganizationRepository.GetMembersWithDetails(tx, organizationId)
+ if err != nil {
+ return "", err
+ }
+ memberSet := make(map[int]bool, len(members))
+ for _, member := range members {
+ memberSet[member.Id] = true
+ }
+ seen := map[int]bool{}
+ for _, userId := range userIds {
+ if !memberSet[userId] {
+ return "Every team member must be a member of the organization.", nil
+ }
+ if seen[userId] {
+ return "The same member is listed twice.", nil
+ }
+ seen[userId] = true
+ }
+ return "", nil
+}
+
+// checkProjectsAssignable enforces org membership of each project and the
+// one-owning-team-per-project rule.
+func (c *teamController) checkProjectsAssignable(tx *sql.Tx, organizationId int, teamId int, projectIds []uuid.UUID) (string, error) {
+ seen := map[uuid.UUID]bool{}
+ for _, projectId := range projectIds {
+ if seen[projectId] {
+ return "The same project is listed twice.", nil
+ }
+ seen[projectId] = true
+ project, err := transactional.ProjectRepository.FindById(tx, projectId)
+ if err != nil {
+ return "", err
+ }
+ if project == nil || project.OrganizationId == nil || *project.OrganizationId != organizationId {
+ return "Every project must belong to this organization.", nil
+ }
+ owner, err := transactional.TeamRepository.FindProjectTeam(tx, projectId)
+ if err != nil {
+ return "", err
+ }
+ if owner != nil && owner.TeamId != teamId {
+ return "The project \"" + project.Name + "\" is already owned by another team.", nil
+ }
+ }
+ return "", nil
+}
+
+func validateTeamName(name string) string {
+ if name == "" {
+ return "A team name is required."
+ }
+ if len(name) > 100 {
+ return "The team name can be at most 100 characters."
+ }
+ return ""
+}
diff --git a/backend/app/controllers/user_notification_rule.controller.go b/backend/app/controllers/user_notification_rule.controller.go
new file mode 100644
index 00000000..52ba347e
--- /dev/null
+++ b/backend/app/controllers/user_notification_rule.controller.go
@@ -0,0 +1,136 @@
+package controllers
+
+import (
+ "net/http"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+
+ "github.com/gin-gonic/gin"
+ traceway "go.tracewayapp.com"
+)
+
+type userNotificationRuleController struct{}
+
+var UserNotificationRuleController = userNotificationRuleController{}
+
+const (
+ maxRuleStepsPerChain = 10
+ maxRuleDelayMinutes = 120
+)
+
+type notificationRuleStep struct {
+ Id int `json:"id,omitempty"`
+ ContactMethodId int `json:"contactMethodId"`
+ DelayMinutes int `json:"delayMinutes"`
+}
+
+type notificationRuleChains struct {
+ High []notificationRuleStep `json:"high"`
+ Low []notificationRuleStep `json:"low"`
+}
+
+func (c *userNotificationRuleController) Get(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ userId := middleware.GetUserId(ctx)
+
+ rules, err := transactional.UserNotificationRuleRepository.FindByUser(tx, userId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load notification rules: %w", err))
+ return
+ }
+ response := notificationRuleChains{High: []notificationRuleStep{}, Low: []notificationRuleStep{}}
+ for _, rule := range rules {
+ step := notificationRuleStep{Id: rule.Id, ContactMethodId: rule.ContactMethodId, DelayMinutes: rule.DelayMinutes}
+ if rule.Urgency == models.UrgencyHigh {
+ response.High = append(response.High, step)
+ } else {
+ response.Low = append(response.Low, step)
+ }
+ }
+ ctx.JSON(http.StatusOK, response)
+}
+
+// Put replaces both chains atomically; positions come from array order.
+func (c *userNotificationRuleController) Put(ctx *gin.Context) {
+ tx := db.GetTx(ctx)
+ userId := middleware.GetUserId(ctx)
+
+ var request notificationRuleChains
+ if err := ctx.ShouldBindJSON(&request); err != nil {
+ ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request body"})
+ return
+ }
+ if len(request.High) > maxRuleStepsPerChain || len(request.Low) > maxRuleStepsPerChain {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "A chain can have at most 10 steps."})
+ return
+ }
+
+ methods, err := transactional.UserContactMethodRepository.FindByUser(tx, userId)
+ if err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to load contact methods: %w", err))
+ return
+ }
+ methodById := make(map[int]*models.UserContactMethod, len(methods))
+ for _, method := range methods {
+ methodById[method.Id] = method
+ }
+
+ now := time.Now().UTC()
+ var rules []*models.UserNotificationRule
+ appendChain := func(urgency string, steps []notificationRuleStep) string {
+ for position, step := range steps {
+ if step.DelayMinutes < 0 || step.DelayMinutes > maxRuleDelayMinutes {
+ return "Step delays must be between 0 and 120 minutes."
+ }
+ method, ok := methodById[step.ContactMethodId]
+ if !ok {
+ return "Every step must reference one of your own contact methods."
+ }
+ if !method.Enabled {
+ return "A step references a disabled contact method. Enable it first."
+ }
+ if !method.Verified {
+ return "Verify " + smsRuleLabel(method) + " before using it in a rule."
+ }
+ rules = append(rules, &models.UserNotificationRule{
+ UserId: userId,
+ Urgency: urgency,
+ Position: position,
+ DelayMinutes: step.DelayMinutes,
+ ContactMethodId: step.ContactMethodId,
+ CreatedAt: now,
+ })
+ }
+ return ""
+ }
+ if message := appendChain(models.UrgencyHigh, request.High); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+ if message := appendChain(models.UrgencyLow, request.Low); message != "" {
+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})
+ return
+ }
+
+ if err := transactional.UserNotificationRuleRepository.ReplaceForUser(tx, userId, rules); err != nil {
+ ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to save notification rules: %w", err))
+ return
+ }
+ c.Get(ctx)
+}
+
+func smsRuleLabel(method *models.UserContactMethod) string {
+ if method.MethodType != "sms" {
+ return "this contact method"
+ }
+ number := oncall.SMSPhoneNumber(method.Config)
+ if number == "" {
+ return "this phone number"
+ }
+ return number
+}
diff --git a/backend/app/dbtest/dbtest.go b/backend/app/dbtest/dbtest.go
new file mode 100644
index 00000000..bb00ce0e
--- /dev/null
+++ b/backend/app/dbtest/dbtest.go
@@ -0,0 +1,48 @@
+// Package dbtest boots the dual in-memory SQLite databases for tests that
+// exercise the default (untagged) storage backends.
+package dbtest
+
+import (
+ "database/sql"
+ "testing"
+
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/migrations"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ _ "modernc.org/sqlite"
+)
+
+// SetupSQLite points db.DB and db.TelemetryDB at fresh in-memory SQLite
+// databases, initializes config and models, and runs the sqlite migrations.
+// Cleanup closes both databases.
+func SetupSQLite(t *testing.T) {
+ t.Helper()
+
+ openMemory := func(name string) *sql.DB {
+ conn, err := sql.Open("sqlite", ":memory:")
+ if err != nil {
+ t.Fatalf("open in-memory sqlite (%s): %v", name, err)
+ }
+ conn.SetMaxOpenConns(1)
+ if _, err := conn.Exec("PRAGMA foreign_keys = ON"); err != nil {
+ t.Fatalf("enable foreign keys: %v", err)
+ }
+ return conn
+ }
+ db.DB = openMemory("main")
+ db.TelemetryDB = openMemory("telemetry")
+ db.Driver = lit.SQLite
+ if config.Config == nil {
+ config.Init(config.LoadFromEnv())
+ }
+ models.Init(db.Driver)
+ if err := migrations.Run("sqlite"); err != nil {
+ t.Fatalf("run migrations: %v", err)
+ }
+ t.Cleanup(func() {
+ db.DB.Close()
+ db.TelemetryDB.Close()
+ })
+}
diff --git a/backend/app/middleware/rate_limit.middleware.go b/backend/app/middleware/rate_limit.middleware.go
index 4307d03c..463b5647 100644
--- a/backend/app/middleware/rate_limit.middleware.go
+++ b/backend/app/middleware/rate_limit.middleware.go
@@ -2,48 +2,87 @@ package middleware
import (
"net/http"
+ "strconv"
"sync"
"time"
"github.com/gin-gonic/gin"
)
-// RateLimitPerIP returns a fixed-window per-IP rate limiter for unauthenticated
-// endpoints that persist state per request (e.g. the device-authorize endpoint,
-// which inserts a main-DB row per call). Buckets live in memory; stale ones are
-// swept on each request, so the map stays bounded by the number of distinct IPs
-// seen within one window.
-func RateLimitPerIP(maxRequests int, window time.Duration) gin.HandlerFunc {
- type bucket struct {
- windowStart time.Time
- count int
+// FixedWindowLimiter is a fixed-window in-memory rate limiter keyed by an
+// arbitrary string (IP, user id, phone number, ...). Stale buckets are swept
+// on each call, so the map stays bounded by the number of distinct keys seen
+// within one window.
+type FixedWindowLimiter struct {
+ mu sync.Mutex
+ maxRequests int
+ window time.Duration
+ buckets map[string]*limiterBucket
+}
+
+type limiterBucket struct {
+ windowStart time.Time
+ count int
+}
+
+func NewFixedWindowLimiter(maxRequests int, window time.Duration) *FixedWindowLimiter {
+ return &FixedWindowLimiter{
+ maxRequests: maxRequests,
+ window: window,
+ buckets: map[string]*limiterBucket{},
}
- var mu sync.Mutex
- buckets := map[string]*bucket{}
+}
- return func(c *gin.Context) {
- now := time.Now()
+// Allow consumes one slot for key and reports whether the request is within
+// the limit.
+func (l *FixedWindowLimiter) Allow(key string) bool {
+ now := time.Now()
- mu.Lock()
- for ip, b := range buckets {
- if now.Sub(b.windowStart) > window {
- delete(buckets, ip)
- }
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ for k, b := range l.buckets {
+ if now.Sub(b.windowStart) > l.window {
+ delete(l.buckets, k)
}
- ip := c.ClientIP()
- b, ok := buckets[ip]
- if !ok {
- b = &bucket{windowStart: now}
- buckets[ip] = b
- }
- b.count++
- count := b.count
- mu.Unlock()
+ }
+ b, ok := l.buckets[key]
+ if !ok {
+ b = &limiterBucket{windowStart: now}
+ l.buckets[key] = b
+ }
+ b.count++
+ return b.count <= l.maxRequests
+}
- if count > maxRequests {
+func rateLimitWithKey(maxRequests int, window time.Duration, keyOf func(c *gin.Context) string) gin.HandlerFunc {
+ limiter := NewFixedWindowLimiter(maxRequests, window)
+ return func(c *gin.Context) {
+ if !limiter.Allow(keyOf(c)) {
c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "slow_down"})
return
}
c.Next()
}
}
+
+// RateLimitPerIP returns a fixed-window per-IP rate limiter for unauthenticated
+// endpoints that persist state per request (e.g. the device-authorize endpoint,
+// which inserts a main-DB row per call).
+func RateLimitPerIP(maxRequests int, window time.Duration) gin.HandlerFunc {
+ return rateLimitWithKey(maxRequests, window, func(c *gin.Context) string {
+ return c.ClientIP()
+ })
+}
+
+// RateLimitPerUser keys the limit by the authenticated user (UseAppAuth must
+// run first), so users behind a shared NAT don't exhaust each other's budget
+// and an attacker cannot widen theirs by rotating IPs. Requests without a
+// resolved user fall back to the client IP so the limit still holds.
+func RateLimitPerUser(maxRequests int, window time.Duration) gin.HandlerFunc {
+ return rateLimitWithKey(maxRequests, window, func(c *gin.Context) string {
+ if userId := GetUserId(c); userId != 0 {
+ return "u:" + strconv.Itoa(userId)
+ }
+ return "ip:" + c.ClientIP()
+ })
+}
diff --git a/backend/app/middleware/require_admin_access.middleware.go b/backend/app/middleware/require_admin_access.middleware.go
index 43b3163a..feeeaa04 100644
--- a/backend/app/middleware/require_admin_access.middleware.go
+++ b/backend/app/middleware/require_admin_access.middleware.go
@@ -17,15 +17,23 @@ const UserOrgRoleContextKey = "userOrgRole"
var RequireAdminAccess gin.HandlerFunc
func InitRequireAdminAccess() {
- RequireAdminAccess = func(c *gin.Context) {
+ RequireAdminAccess = requireOrgRole(func(role string) bool {
+ return role == "owner" || role == "admin"
+ }, "Admin or owner access required")
+}
+
+// requireOrgRole builds middleware that resolves the caller's role in the
+// :organizationId organization and rejects with 403 when permitted returns
+// false. Sets OrganizationIdContextKey and UserOrgRoleContextKey.
+func requireOrgRole(permitted func(role string) bool, denialMessage string) gin.HandlerFunc {
+ return func(c *gin.Context) {
userId := GetUserId(c)
if userId == 0 {
c.AbortWithStatus(http.StatusUnauthorized)
return
}
- organizationIdStr := c.Param("organizationId")
- organizationId, err := strconv.Atoi(organizationIdStr)
+ organizationId, err := strconv.Atoi(c.Param("organizationId"))
if err != nil {
c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": "Invalid organization ID"})
return
@@ -40,8 +48,8 @@ func InitRequireAdminAccess() {
return
}
- if role != "owner" && role != "admin" {
- c.AbortWithStatusJSON(http.StatusForbidden, gin.H{"error": "Admin or owner access required"})
+ if !permitted(role) {
+ c.AbortWithStatusJSON(http.StatusForbidden, gin.H{"error": denialMessage})
return
}
diff --git a/backend/app/middleware/require_organization_access.middleware.go b/backend/app/middleware/require_organization_access.middleware.go
new file mode 100644
index 00000000..70137bd2
--- /dev/null
+++ b/backend/app/middleware/require_organization_access.middleware.go
@@ -0,0 +1,16 @@
+package middleware
+
+import (
+ "github.com/gin-gonic/gin"
+)
+
+// RequireOrganizationAccess mirrors RequireAdminAccess but accepts any org
+// role (readonly included): it only requires membership in the
+// :organizationId organization. Sets the same context keys.
+var RequireOrganizationAccess gin.HandlerFunc
+
+func InitRequireOrganizationAccess() {
+ RequireOrganizationAccess = requireOrgRole(func(role string) bool {
+ return role != ""
+ }, "Organization membership required")
+}
diff --git a/backend/app/middleware/transactional.middleware.go b/backend/app/middleware/transactional.middleware.go
index ec307c6d..87707a24 100644
--- a/backend/app/middleware/transactional.middleware.go
+++ b/backend/app/middleware/transactional.middleware.go
@@ -36,7 +36,29 @@ func Transactional(c *gin.Context) {
c.AbortWithStatus(http.StatusInternalServerError)
panic(err)
}
+ runCommitHooks(c)
} else {
txHandle.Rollback()
}
}
+
+const commitHooksContextKey = "txCommitHooks"
+
+// OnCommit queues fn to run after the Transactional middleware successfully
+// commits the request transaction; queued fns are dropped on rollback. Use it
+// for side effects that must only fire once the transaction's writes are
+// visible to other connections (e.g. waking the outbox drain worker, which
+// would otherwise poll before the enqueued row exists and go back to sleep).
+func OnCommit(c *gin.Context, fn func()) {
+ hooks, _ := c.Get(commitHooksContextKey)
+ fns, _ := hooks.([]func())
+ c.Set(commitHooksContextKey, append(fns, fn))
+}
+
+func runCommitHooks(c *gin.Context) {
+ hooks, _ := c.Get(commitHooksContextKey)
+ fns, _ := hooks.([]func())
+ for _, fn := range fns {
+ fn()
+ }
+}
diff --git a/backend/app/migrations/pg/0068_create_teams.up.sql b/backend/app/migrations/pg/0068_create_teams.up.sql
new file mode 100644
index 00000000..b7d7639d
--- /dev/null
+++ b/backend/app/migrations/pg/0068_create_teams.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS teams (
+ id SERIAL PRIMARY KEY,
+ organization_id INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ name VARCHAR(100) NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0069_create_teams_org_name_unique.up.sql b/backend/app/migrations/pg/0069_create_teams_org_name_unique.up.sql
new file mode 100644
index 00000000..7dfec405
--- /dev/null
+++ b/backend/app/migrations/pg/0069_create_teams_org_name_unique.up.sql
@@ -0,0 +1 @@
+CREATE UNIQUE INDEX IF NOT EXISTS teams_org_name_unique ON teams (organization_id, LOWER(name))
diff --git a/backend/app/migrations/pg/0070_create_team_members.up.sql b/backend/app/migrations/pg/0070_create_team_members.up.sql
new file mode 100644
index 00000000..e5f366d2
--- /dev/null
+++ b/backend/app/migrations/pg/0070_create_team_members.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS team_members (
+ id SERIAL PRIMARY KEY,
+ team_id INT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+ user_id INT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ position INT NOT NULL DEFAULT 0,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ UNIQUE(team_id, user_id)
+)
diff --git a/backend/app/migrations/pg/0071_create_project_teams.up.sql b/backend/app/migrations/pg/0071_create_project_teams.up.sql
new file mode 100644
index 00000000..b263931a
--- /dev/null
+++ b/backend/app/migrations/pg/0071_create_project_teams.up.sql
@@ -0,0 +1,7 @@
+CREATE TABLE IF NOT EXISTS project_teams (
+ id SERIAL PRIMARY KEY,
+ project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ team_id INT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ UNIQUE(project_id)
+)
diff --git a/backend/app/migrations/pg/0072_create_oncall_schedules.up.sql b/backend/app/migrations/pg/0072_create_oncall_schedules.up.sql
new file mode 100644
index 00000000..44d7aca4
--- /dev/null
+++ b/backend/app/migrations/pg/0072_create_oncall_schedules.up.sql
@@ -0,0 +1,12 @@
+CREATE TABLE IF NOT EXISTS oncall_schedules (
+ id SERIAL PRIMARY KEY,
+ organization_id INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ team_id INT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+ name VARCHAR(100) NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ timezone VARCHAR(64) NOT NULL DEFAULT 'UTC',
+ definition JSONB NOT NULL DEFAULT '{}',
+ created_by INT REFERENCES users(id),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0073_create_oncall_schedules_org_name_unique.up.sql b/backend/app/migrations/pg/0073_create_oncall_schedules_org_name_unique.up.sql
new file mode 100644
index 00000000..53a86619
--- /dev/null
+++ b/backend/app/migrations/pg/0073_create_oncall_schedules_org_name_unique.up.sql
@@ -0,0 +1 @@
+CREATE UNIQUE INDEX IF NOT EXISTS oncall_schedules_org_name_unique ON oncall_schedules (organization_id, LOWER(name))
diff --git a/backend/app/migrations/pg/0074_create_oncall_overrides.up.sql b/backend/app/migrations/pg/0074_create_oncall_overrides.up.sql
new file mode 100644
index 00000000..efcedccb
--- /dev/null
+++ b/backend/app/migrations/pg/0074_create_oncall_overrides.up.sql
@@ -0,0 +1,9 @@
+CREATE TABLE IF NOT EXISTS oncall_overrides (
+ id SERIAL PRIMARY KEY,
+ schedule_id INT NOT NULL REFERENCES oncall_schedules(id) ON DELETE CASCADE,
+ user_id INT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ start_at TIMESTAMPTZ NOT NULL,
+ end_at TIMESTAMPTZ NOT NULL,
+ created_by INT REFERENCES users(id),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0075_create_oncall_overrides_schedule_end_idx.up.sql b/backend/app/migrations/pg/0075_create_oncall_overrides_schedule_end_idx.up.sql
new file mode 100644
index 00000000..358547a8
--- /dev/null
+++ b/backend/app/migrations/pg/0075_create_oncall_overrides_schedule_end_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS oncall_overrides_schedule_end_idx ON oncall_overrides (schedule_id, end_at)
diff --git a/backend/app/migrations/pg/0076_create_escalation_policies.up.sql b/backend/app/migrations/pg/0076_create_escalation_policies.up.sql
new file mode 100644
index 00000000..e77c9c1d
--- /dev/null
+++ b/backend/app/migrations/pg/0076_create_escalation_policies.up.sql
@@ -0,0 +1,9 @@
+CREATE TABLE IF NOT EXISTS escalation_policies (
+ id SERIAL PRIMARY KEY,
+ organization_id INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ name VARCHAR(100) NOT NULL,
+ definition JSONB NOT NULL DEFAULT '{}',
+ created_by INT REFERENCES users(id),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0077_create_escalation_policies_org_name_unique.up.sql b/backend/app/migrations/pg/0077_create_escalation_policies_org_name_unique.up.sql
new file mode 100644
index 00000000..7bfbb3b2
--- /dev/null
+++ b/backend/app/migrations/pg/0077_create_escalation_policies_org_name_unique.up.sql
@@ -0,0 +1 @@
+CREATE UNIQUE INDEX IF NOT EXISTS escalation_policies_org_name_unique ON escalation_policies (organization_id, LOWER(name))
diff --git a/backend/app/migrations/pg/0078_create_pages.up.sql b/backend/app/migrations/pg/0078_create_pages.up.sql
new file mode 100644
index 00000000..0b035472
--- /dev/null
+++ b/backend/app/migrations/pg/0078_create_pages.up.sql
@@ -0,0 +1,28 @@
+CREATE TABLE IF NOT EXISTS pages (
+ id SERIAL PRIMARY KEY,
+ organization_id INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ policy_id INT REFERENCES escalation_policies(id) ON DELETE SET NULL,
+ policy_snapshot JSONB NOT NULL DEFAULT '{}',
+ rule_id INT,
+ rule_name VARCHAR(200) NOT NULL DEFAULT '',
+ rule_type VARCHAR(50) NOT NULL DEFAULT '',
+ subject TEXT NOT NULL DEFAULT '',
+ body TEXT NOT NULL DEFAULT '',
+ url TEXT NOT NULL DEFAULT '',
+ severity VARCHAR(20) NOT NULL DEFAULT '',
+ status VARCHAR(20) NOT NULL DEFAULT 'open',
+ dedup_key VARCHAR(300) NOT NULL DEFAULT '',
+ event_count INT NOT NULL DEFAULT 1,
+ last_event_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ escalation_level INT NOT NULL DEFAULT -1,
+ repeat_iteration INT NOT NULL DEFAULT 0,
+ next_escalation_at TIMESTAMPTZ,
+ last_escalated_at TIMESTAMPTZ,
+ acknowledged_by INT,
+ acknowledged_at TIMESTAMPTZ,
+ resolved_by INT,
+ resolved_at TIMESTAMPTZ,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0079_create_pages_dedup_open_unique.up.sql b/backend/app/migrations/pg/0079_create_pages_dedup_open_unique.up.sql
new file mode 100644
index 00000000..02de9f07
--- /dev/null
+++ b/backend/app/migrations/pg/0079_create_pages_dedup_open_unique.up.sql
@@ -0,0 +1 @@
+CREATE UNIQUE INDEX IF NOT EXISTS pages_dedup_open_unique ON pages (dedup_key) WHERE status <> 'resolved'
diff --git a/backend/app/migrations/pg/0080_create_pages_due_idx.up.sql b/backend/app/migrations/pg/0080_create_pages_due_idx.up.sql
new file mode 100644
index 00000000..21ee1283
--- /dev/null
+++ b/backend/app/migrations/pg/0080_create_pages_due_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS pages_due_idx ON pages (status, next_escalation_at)
diff --git a/backend/app/migrations/pg/0081_create_user_contact_methods.up.sql b/backend/app/migrations/pg/0081_create_user_contact_methods.up.sql
new file mode 100644
index 00000000..f67b4497
--- /dev/null
+++ b/backend/app/migrations/pg/0081_create_user_contact_methods.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS user_contact_methods (
+ id SERIAL PRIMARY KEY,
+ user_id INT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ method_type VARCHAR(50) NOT NULL,
+ config JSONB NOT NULL DEFAULT '{}',
+ enabled BOOLEAN NOT NULL DEFAULT true,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0082_create_page_notifications.up.sql b/backend/app/migrations/pg/0082_create_page_notifications.up.sql
new file mode 100644
index 00000000..e4bbd1c4
--- /dev/null
+++ b/backend/app/migrations/pg/0082_create_page_notifications.up.sql
@@ -0,0 +1,13 @@
+CREATE TABLE IF NOT EXISTS page_notifications (
+ id SERIAL PRIMARY KEY,
+ page_id INT NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
+ level INT NOT NULL DEFAULT 0,
+ iteration INT NOT NULL DEFAULT 0,
+ user_id INT,
+ target_desc VARCHAR(300) NOT NULL DEFAULT '',
+ method_type VARCHAR(50) NOT NULL DEFAULT '',
+ status VARCHAR(20) NOT NULL DEFAULT 'pending',
+ error_msg TEXT NOT NULL DEFAULT '',
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ sent_at TIMESTAMPTZ
+)
diff --git a/backend/app/migrations/pg/0083_create_notification_outbox.up.sql b/backend/app/migrations/pg/0083_create_notification_outbox.up.sql
new file mode 100644
index 00000000..c662c70c
--- /dev/null
+++ b/backend/app/migrations/pg/0083_create_notification_outbox.up.sql
@@ -0,0 +1,19 @@
+CREATE TABLE IF NOT EXISTS notification_outbox (
+ id SERIAL PRIMARY KEY,
+ kind VARCHAR(30) NOT NULL,
+ status VARCHAR(20) NOT NULL DEFAULT 'pending',
+ adapter_type VARCHAR(50) NOT NULL,
+ adapter_config TEXT NOT NULL DEFAULT '{}',
+ message TEXT NOT NULL DEFAULT '{}',
+ attempts INT NOT NULL DEFAULT 0,
+ next_attempt_at TIMESTAMPTZ NOT NULL,
+ claimed_at TIMESTAMPTZ,
+ cancel_key VARCHAR(100) NOT NULL DEFAULT '',
+ page_notification_id INT,
+ rule_id INT,
+ project_id UUID,
+ channel_name VARCHAR(200) NOT NULL DEFAULT '',
+ last_error TEXT NOT NULL DEFAULT '',
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ sent_at TIMESTAMPTZ
+)
diff --git a/backend/app/migrations/pg/0084_create_notification_outbox_due_idx.up.sql b/backend/app/migrations/pg/0084_create_notification_outbox_due_idx.up.sql
new file mode 100644
index 00000000..c56d422f
--- /dev/null
+++ b/backend/app/migrations/pg/0084_create_notification_outbox_due_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS notification_outbox_due_idx ON notification_outbox (status, next_attempt_at)
diff --git a/backend/app/migrations/pg/0085_create_notification_outbox_cancel_idx.up.sql b/backend/app/migrations/pg/0085_create_notification_outbox_cancel_idx.up.sql
new file mode 100644
index 00000000..89e8832b
--- /dev/null
+++ b/backend/app/migrations/pg/0085_create_notification_outbox_cancel_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS notification_outbox_cancel_idx ON notification_outbox (cancel_key) WHERE cancel_key <> ''
diff --git a/backend/app/migrations/pg/0086_create_user_notification_rules.up.sql b/backend/app/migrations/pg/0086_create_user_notification_rules.up.sql
new file mode 100644
index 00000000..262ceb8e
--- /dev/null
+++ b/backend/app/migrations/pg/0086_create_user_notification_rules.up.sql
@@ -0,0 +1,9 @@
+CREATE TABLE IF NOT EXISTS user_notification_rules (
+ id SERIAL PRIMARY KEY,
+ user_id INT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ urgency VARCHAR(10) NOT NULL,
+ position INT NOT NULL DEFAULT 0,
+ delay_minutes INT NOT NULL DEFAULT 0,
+ contact_method_id INT NOT NULL REFERENCES user_contact_methods(id) ON DELETE CASCADE,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+)
diff --git a/backend/app/migrations/pg/0087_create_user_notification_rules_user_idx.up.sql b/backend/app/migrations/pg/0087_create_user_notification_rules_user_idx.up.sql
new file mode 100644
index 00000000..727c11cb
--- /dev/null
+++ b/backend/app/migrations/pg/0087_create_user_notification_rules_user_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS user_notification_rules_user_urgency_idx ON user_notification_rules (user_id, urgency, position)
diff --git a/backend/app/migrations/pg/0088_add_verified_to_user_contact_methods.up.sql b/backend/app/migrations/pg/0088_add_verified_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..dc614da6
--- /dev/null
+++ b/backend/app/migrations/pg/0088_add_verified_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN IF NOT EXISTS verified BOOLEAN NOT NULL DEFAULT true
diff --git a/backend/app/migrations/pg/0089_add_verification_code_hash_to_user_contact_methods.up.sql b/backend/app/migrations/pg/0089_add_verification_code_hash_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..c6744852
--- /dev/null
+++ b/backend/app/migrations/pg/0089_add_verification_code_hash_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN IF NOT EXISTS verification_code_hash VARCHAR(64) NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/pg/0090_add_verification_expires_at_to_user_contact_methods.up.sql b/backend/app/migrations/pg/0090_add_verification_expires_at_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..0244ddd4
--- /dev/null
+++ b/backend/app/migrations/pg/0090_add_verification_expires_at_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN IF NOT EXISTS verification_expires_at TIMESTAMPTZ
diff --git a/backend/app/migrations/pg/0091_add_verification_attempts_to_user_contact_methods.up.sql b/backend/app/migrations/pg/0091_add_verification_attempts_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..1dd79b57
--- /dev/null
+++ b/backend/app/migrations/pg/0091_add_verification_attempts_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN IF NOT EXISTS verification_attempts INT NOT NULL DEFAULT 0
diff --git a/backend/app/migrations/pg/0092_add_urgency_to_pages.up.sql b/backend/app/migrations/pg/0092_add_urgency_to_pages.up.sql
new file mode 100644
index 00000000..f7624f0e
--- /dev/null
+++ b/backend/app/migrations/pg/0092_add_urgency_to_pages.up.sql
@@ -0,0 +1 @@
+ALTER TABLE pages ADD COLUMN IF NOT EXISTS urgency VARCHAR(10) NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/pg/0093_add_acknowledged_via_to_pages.up.sql b/backend/app/migrations/pg/0093_add_acknowledged_via_to_pages.up.sql
new file mode 100644
index 00000000..8908cf1a
--- /dev/null
+++ b/backend/app/migrations/pg/0093_add_acknowledged_via_to_pages.up.sql
@@ -0,0 +1 @@
+ALTER TABLE pages ADD COLUMN IF NOT EXISTS acknowledged_via VARCHAR(20) NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/pg/0094_add_scheduled_for_to_page_notifications.up.sql b/backend/app/migrations/pg/0094_add_scheduled_for_to_page_notifications.up.sql
new file mode 100644
index 00000000..5a6a1bf8
--- /dev/null
+++ b/backend/app/migrations/pg/0094_add_scheduled_for_to_page_notifications.up.sql
@@ -0,0 +1 @@
+ALTER TABLE page_notifications ADD COLUMN IF NOT EXISTS scheduled_for TIMESTAMPTZ
diff --git a/backend/app/migrations/pg/0095_add_ack_token_hash_to_page_notifications.up.sql b/backend/app/migrations/pg/0095_add_ack_token_hash_to_page_notifications.up.sql
new file mode 100644
index 00000000..6d72bae0
--- /dev/null
+++ b/backend/app/migrations/pg/0095_add_ack_token_hash_to_page_notifications.up.sql
@@ -0,0 +1 @@
+ALTER TABLE page_notifications ADD COLUMN IF NOT EXISTS ack_token_hash VARCHAR(64) NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/pg/0096_create_page_notifications_ack_token_unique.up.sql b/backend/app/migrations/pg/0096_create_page_notifications_ack_token_unique.up.sql
new file mode 100644
index 00000000..9f9ff568
--- /dev/null
+++ b/backend/app/migrations/pg/0096_create_page_notifications_ack_token_unique.up.sql
@@ -0,0 +1 @@
+CREATE UNIQUE INDEX IF NOT EXISTS page_notifications_ack_token_unique ON page_notifications (ack_token_hash) WHERE ack_token_hash <> ''
diff --git a/backend/app/migrations/pg/0097_create_pages_project_created_idx.up.sql b/backend/app/migrations/pg/0097_create_pages_project_created_idx.up.sql
new file mode 100644
index 00000000..05a8501b
--- /dev/null
+++ b/backend/app/migrations/pg/0097_create_pages_project_created_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS pages_project_created_idx ON pages (project_id, created_at DESC)
diff --git a/backend/app/migrations/pg/0098_create_page_notifications_page_idx.up.sql b/backend/app/migrations/pg/0098_create_page_notifications_page_idx.up.sql
new file mode 100644
index 00000000..5c9126e0
--- /dev/null
+++ b/backend/app/migrations/pg/0098_create_page_notifications_page_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS page_notifications_page_idx ON page_notifications (page_id)
diff --git a/backend/app/migrations/pg/0099_create_user_contact_methods_user_idx.up.sql b/backend/app/migrations/pg/0099_create_user_contact_methods_user_idx.up.sql
new file mode 100644
index 00000000..2fa49bc3
--- /dev/null
+++ b/backend/app/migrations/pg/0099_create_user_contact_methods_user_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS user_contact_methods_user_idx ON user_contact_methods (user_id)
diff --git a/backend/app/migrations/sqlite/0037_create_teams.up.sql b/backend/app/migrations/sqlite/0037_create_teams.up.sql
new file mode 100644
index 00000000..20f2b7b3
--- /dev/null
+++ b/backend/app/migrations/sqlite/0037_create_teams.up.sql
@@ -0,0 +1,10 @@
+CREATE TABLE IF NOT EXISTS teams (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ organization_id INTEGER NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ name TEXT NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ created_at DATETIME NOT NULL,
+ updated_at DATETIME NOT NULL
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS teams_org_name_unique ON teams (organization_id, LOWER(name));
diff --git a/backend/app/migrations/sqlite/0038_create_team_members.up.sql b/backend/app/migrations/sqlite/0038_create_team_members.up.sql
new file mode 100644
index 00000000..429f91b8
--- /dev/null
+++ b/backend/app/migrations/sqlite/0038_create_team_members.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS team_members (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ team_id INTEGER NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+ user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ position INTEGER NOT NULL DEFAULT 0,
+ created_at DATETIME NOT NULL,
+ UNIQUE(team_id, user_id)
+);
diff --git a/backend/app/migrations/sqlite/0039_create_project_teams.up.sql b/backend/app/migrations/sqlite/0039_create_project_teams.up.sql
new file mode 100644
index 00000000..0d72d953
--- /dev/null
+++ b/backend/app/migrations/sqlite/0039_create_project_teams.up.sql
@@ -0,0 +1,7 @@
+CREATE TABLE IF NOT EXISTS project_teams (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ team_id INTEGER NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+ created_at DATETIME NOT NULL,
+ UNIQUE(project_id)
+);
diff --git a/backend/app/migrations/sqlite/0040_create_oncall_schedules.up.sql b/backend/app/migrations/sqlite/0040_create_oncall_schedules.up.sql
new file mode 100644
index 00000000..47fe8bd6
--- /dev/null
+++ b/backend/app/migrations/sqlite/0040_create_oncall_schedules.up.sql
@@ -0,0 +1,14 @@
+CREATE TABLE IF NOT EXISTS oncall_schedules (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ organization_id INTEGER NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ team_id INTEGER NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
+ name TEXT NOT NULL,
+ description TEXT NOT NULL DEFAULT '',
+ timezone TEXT NOT NULL DEFAULT 'UTC',
+ definition TEXT NOT NULL DEFAULT '{}',
+ created_by INTEGER REFERENCES users(id),
+ created_at DATETIME NOT NULL,
+ updated_at DATETIME NOT NULL
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS oncall_schedules_org_name_unique ON oncall_schedules (organization_id, LOWER(name));
diff --git a/backend/app/migrations/sqlite/0041_create_oncall_overrides.up.sql b/backend/app/migrations/sqlite/0041_create_oncall_overrides.up.sql
new file mode 100644
index 00000000..b443dcf6
--- /dev/null
+++ b/backend/app/migrations/sqlite/0041_create_oncall_overrides.up.sql
@@ -0,0 +1,11 @@
+CREATE TABLE IF NOT EXISTS oncall_overrides (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ schedule_id INTEGER NOT NULL REFERENCES oncall_schedules(id) ON DELETE CASCADE,
+ user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ start_at DATETIME NOT NULL,
+ end_at DATETIME NOT NULL,
+ created_by INTEGER REFERENCES users(id),
+ created_at DATETIME NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS oncall_overrides_schedule_end_idx ON oncall_overrides (schedule_id, end_at);
diff --git a/backend/app/migrations/sqlite/0042_create_escalation_policies.up.sql b/backend/app/migrations/sqlite/0042_create_escalation_policies.up.sql
new file mode 100644
index 00000000..e15341ee
--- /dev/null
+++ b/backend/app/migrations/sqlite/0042_create_escalation_policies.up.sql
@@ -0,0 +1,11 @@
+CREATE TABLE IF NOT EXISTS escalation_policies (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ organization_id INTEGER NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ name TEXT NOT NULL,
+ definition TEXT NOT NULL DEFAULT '{}',
+ created_by INTEGER REFERENCES users(id),
+ created_at DATETIME NOT NULL,
+ updated_at DATETIME NOT NULL
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS escalation_policies_org_name_unique ON escalation_policies (organization_id, LOWER(name));
diff --git a/backend/app/migrations/sqlite/0043_create_pages.up.sql b/backend/app/migrations/sqlite/0043_create_pages.up.sql
new file mode 100644
index 00000000..c828bb83
--- /dev/null
+++ b/backend/app/migrations/sqlite/0043_create_pages.up.sql
@@ -0,0 +1,32 @@
+CREATE TABLE IF NOT EXISTS pages (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ organization_id INTEGER NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ policy_id INTEGER REFERENCES escalation_policies(id) ON DELETE SET NULL,
+ policy_snapshot TEXT NOT NULL DEFAULT '{}',
+ rule_id INTEGER,
+ rule_name TEXT NOT NULL DEFAULT '',
+ rule_type TEXT NOT NULL DEFAULT '',
+ subject TEXT NOT NULL DEFAULT '',
+ body TEXT NOT NULL DEFAULT '',
+ url TEXT NOT NULL DEFAULT '',
+ severity TEXT NOT NULL DEFAULT '',
+ status TEXT NOT NULL DEFAULT 'open',
+ dedup_key TEXT NOT NULL DEFAULT '',
+ event_count INTEGER NOT NULL DEFAULT 1,
+ last_event_at DATETIME NOT NULL,
+ escalation_level INTEGER NOT NULL DEFAULT -1,
+ repeat_iteration INTEGER NOT NULL DEFAULT 0,
+ next_escalation_at DATETIME,
+ last_escalated_at DATETIME,
+ acknowledged_by INTEGER,
+ acknowledged_at DATETIME,
+ resolved_by INTEGER,
+ resolved_at DATETIME,
+ created_at DATETIME NOT NULL,
+ updated_at DATETIME NOT NULL
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS pages_dedup_open_unique ON pages (dedup_key) WHERE status <> 'resolved';
+
+CREATE INDEX IF NOT EXISTS pages_due_idx ON pages (status, next_escalation_at);
diff --git a/backend/app/migrations/sqlite/0044_create_user_contact_methods.up.sql b/backend/app/migrations/sqlite/0044_create_user_contact_methods.up.sql
new file mode 100644
index 00000000..abacda17
--- /dev/null
+++ b/backend/app/migrations/sqlite/0044_create_user_contact_methods.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS user_contact_methods (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ method_type TEXT NOT NULL,
+ config TEXT NOT NULL DEFAULT '{}',
+ enabled INTEGER NOT NULL DEFAULT 1,
+ created_at DATETIME NOT NULL
+);
diff --git a/backend/app/migrations/sqlite/0045_create_page_notifications.up.sql b/backend/app/migrations/sqlite/0045_create_page_notifications.up.sql
new file mode 100644
index 00000000..61428f58
--- /dev/null
+++ b/backend/app/migrations/sqlite/0045_create_page_notifications.up.sql
@@ -0,0 +1,13 @@
+CREATE TABLE IF NOT EXISTS page_notifications (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
+ level INTEGER NOT NULL DEFAULT 0,
+ iteration INTEGER NOT NULL DEFAULT 0,
+ user_id INTEGER,
+ target_desc TEXT NOT NULL DEFAULT '',
+ method_type TEXT NOT NULL DEFAULT '',
+ status TEXT NOT NULL DEFAULT 'pending',
+ error_msg TEXT NOT NULL DEFAULT '',
+ created_at DATETIME NOT NULL,
+ sent_at DATETIME
+);
diff --git a/backend/app/migrations/sqlite/0046_create_notification_outbox.up.sql b/backend/app/migrations/sqlite/0046_create_notification_outbox.up.sql
new file mode 100644
index 00000000..794af7de
--- /dev/null
+++ b/backend/app/migrations/sqlite/0046_create_notification_outbox.up.sql
@@ -0,0 +1,23 @@
+CREATE TABLE IF NOT EXISTS notification_outbox (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ kind TEXT NOT NULL,
+ status TEXT NOT NULL DEFAULT 'pending',
+ adapter_type TEXT NOT NULL,
+ adapter_config TEXT NOT NULL DEFAULT '{}',
+ message TEXT NOT NULL DEFAULT '{}',
+ attempts INTEGER NOT NULL DEFAULT 0,
+ next_attempt_at DATETIME NOT NULL,
+ claimed_at DATETIME,
+ cancel_key TEXT NOT NULL DEFAULT '',
+ page_notification_id INTEGER,
+ rule_id INTEGER,
+ project_id TEXT,
+ channel_name TEXT NOT NULL DEFAULT '',
+ last_error TEXT NOT NULL DEFAULT '',
+ created_at DATETIME NOT NULL,
+ sent_at DATETIME
+);
+
+CREATE INDEX IF NOT EXISTS notification_outbox_due_idx ON notification_outbox (status, next_attempt_at);
+
+CREATE INDEX IF NOT EXISTS notification_outbox_cancel_idx ON notification_outbox (cancel_key) WHERE cancel_key <> '';
diff --git a/backend/app/migrations/sqlite/0047_create_user_notification_rules.up.sql b/backend/app/migrations/sqlite/0047_create_user_notification_rules.up.sql
new file mode 100644
index 00000000..abd8bcc9
--- /dev/null
+++ b/backend/app/migrations/sqlite/0047_create_user_notification_rules.up.sql
@@ -0,0 +1,9 @@
+CREATE TABLE IF NOT EXISTS user_notification_rules (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+ urgency TEXT NOT NULL,
+ position INTEGER NOT NULL DEFAULT 0,
+ delay_minutes INTEGER NOT NULL DEFAULT 0,
+ contact_method_id INTEGER NOT NULL REFERENCES user_contact_methods(id) ON DELETE CASCADE,
+ created_at DATETIME NOT NULL
+)
diff --git a/backend/app/migrations/sqlite/0048_create_user_notification_rules_user_idx.up.sql b/backend/app/migrations/sqlite/0048_create_user_notification_rules_user_idx.up.sql
new file mode 100644
index 00000000..727c11cb
--- /dev/null
+++ b/backend/app/migrations/sqlite/0048_create_user_notification_rules_user_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS user_notification_rules_user_urgency_idx ON user_notification_rules (user_id, urgency, position)
diff --git a/backend/app/migrations/sqlite/0049_add_verified_to_user_contact_methods.up.sql b/backend/app/migrations/sqlite/0049_add_verified_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..aaa7e104
--- /dev/null
+++ b/backend/app/migrations/sqlite/0049_add_verified_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN verified INTEGER NOT NULL DEFAULT 1
diff --git a/backend/app/migrations/sqlite/0050_add_verification_code_hash_to_user_contact_methods.up.sql b/backend/app/migrations/sqlite/0050_add_verification_code_hash_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..77bce4c3
--- /dev/null
+++ b/backend/app/migrations/sqlite/0050_add_verification_code_hash_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN verification_code_hash TEXT NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/sqlite/0051_add_verification_expires_at_to_user_contact_methods.up.sql b/backend/app/migrations/sqlite/0051_add_verification_expires_at_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..edf2cecf
--- /dev/null
+++ b/backend/app/migrations/sqlite/0051_add_verification_expires_at_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN verification_expires_at DATETIME
diff --git a/backend/app/migrations/sqlite/0052_add_verification_attempts_to_user_contact_methods.up.sql b/backend/app/migrations/sqlite/0052_add_verification_attempts_to_user_contact_methods.up.sql
new file mode 100644
index 00000000..44409bd7
--- /dev/null
+++ b/backend/app/migrations/sqlite/0052_add_verification_attempts_to_user_contact_methods.up.sql
@@ -0,0 +1 @@
+ALTER TABLE user_contact_methods ADD COLUMN verification_attempts INTEGER NOT NULL DEFAULT 0
diff --git a/backend/app/migrations/sqlite/0053_add_urgency_to_pages.up.sql b/backend/app/migrations/sqlite/0053_add_urgency_to_pages.up.sql
new file mode 100644
index 00000000..476685be
--- /dev/null
+++ b/backend/app/migrations/sqlite/0053_add_urgency_to_pages.up.sql
@@ -0,0 +1 @@
+ALTER TABLE pages ADD COLUMN urgency TEXT NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/sqlite/0054_add_acknowledged_via_to_pages.up.sql b/backend/app/migrations/sqlite/0054_add_acknowledged_via_to_pages.up.sql
new file mode 100644
index 00000000..f881f1d8
--- /dev/null
+++ b/backend/app/migrations/sqlite/0054_add_acknowledged_via_to_pages.up.sql
@@ -0,0 +1 @@
+ALTER TABLE pages ADD COLUMN acknowledged_via TEXT NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/sqlite/0055_add_scheduled_for_to_page_notifications.up.sql b/backend/app/migrations/sqlite/0055_add_scheduled_for_to_page_notifications.up.sql
new file mode 100644
index 00000000..549557af
--- /dev/null
+++ b/backend/app/migrations/sqlite/0055_add_scheduled_for_to_page_notifications.up.sql
@@ -0,0 +1 @@
+ALTER TABLE page_notifications ADD COLUMN scheduled_for DATETIME
diff --git a/backend/app/migrations/sqlite/0056_add_ack_token_hash_to_page_notifications.up.sql b/backend/app/migrations/sqlite/0056_add_ack_token_hash_to_page_notifications.up.sql
new file mode 100644
index 00000000..7f3e60ce
--- /dev/null
+++ b/backend/app/migrations/sqlite/0056_add_ack_token_hash_to_page_notifications.up.sql
@@ -0,0 +1 @@
+ALTER TABLE page_notifications ADD COLUMN ack_token_hash TEXT NOT NULL DEFAULT ''
diff --git a/backend/app/migrations/sqlite/0057_create_page_notifications_ack_token_unique.up.sql b/backend/app/migrations/sqlite/0057_create_page_notifications_ack_token_unique.up.sql
new file mode 100644
index 00000000..9f9ff568
--- /dev/null
+++ b/backend/app/migrations/sqlite/0057_create_page_notifications_ack_token_unique.up.sql
@@ -0,0 +1 @@
+CREATE UNIQUE INDEX IF NOT EXISTS page_notifications_ack_token_unique ON page_notifications (ack_token_hash) WHERE ack_token_hash <> ''
diff --git a/backend/app/migrations/sqlite/0058_create_pages_project_created_idx.up.sql b/backend/app/migrations/sqlite/0058_create_pages_project_created_idx.up.sql
new file mode 100644
index 00000000..05a8501b
--- /dev/null
+++ b/backend/app/migrations/sqlite/0058_create_pages_project_created_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS pages_project_created_idx ON pages (project_id, created_at DESC)
diff --git a/backend/app/migrations/sqlite/0059_create_page_notifications_page_idx.up.sql b/backend/app/migrations/sqlite/0059_create_page_notifications_page_idx.up.sql
new file mode 100644
index 00000000..5c9126e0
--- /dev/null
+++ b/backend/app/migrations/sqlite/0059_create_page_notifications_page_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS page_notifications_page_idx ON page_notifications (page_id)
diff --git a/backend/app/migrations/sqlite/0060_create_user_contact_methods_user_idx.up.sql b/backend/app/migrations/sqlite/0060_create_user_contact_methods_user_idx.up.sql
new file mode 100644
index 00000000..2fa49bc3
--- /dev/null
+++ b/backend/app/migrations/sqlite/0060_create_user_contact_methods_user_idx.up.sql
@@ -0,0 +1 @@
+CREATE INDEX IF NOT EXISTS user_contact_methods_user_idx ON user_contact_methods (user_id)
diff --git a/backend/app/models/escalation.model.go b/backend/app/models/escalation.model.go
new file mode 100644
index 00000000..402b52e8
--- /dev/null
+++ b/backend/app/models/escalation.model.go
@@ -0,0 +1,105 @@
+package models
+
+import (
+ "time"
+
+ "github.com/google/uuid"
+)
+
+type EscalationPolicy struct {
+ Id int `json:"id" lit:"id"`
+ OrganizationId int `json:"organizationId" lit:"organization_id"`
+ Name string `json:"name" lit:"name"`
+ Definition JSONText `json:"definition" lit:"definition"`
+ CreatedBy *int `json:"createdBy" lit:"created_by"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+}
+
+const (
+ PageStatusOpen = "open"
+ PageStatusAcknowledged = "acknowledged"
+ PageStatusResolved = "resolved"
+)
+
+type Page struct {
+ Id int `json:"id" lit:"id"`
+ OrganizationId int `json:"organizationId" lit:"organization_id"`
+ ProjectId uuid.UUID `json:"projectId" lit:"project_id"`
+ PolicyId *int `json:"policyId" lit:"policy_id"`
+ PolicySnapshot JSONText `json:"policySnapshot" lit:"policy_snapshot"`
+ RuleId *int `json:"ruleId" lit:"rule_id"`
+ RuleName string `json:"ruleName" lit:"rule_name"`
+ RuleType string `json:"ruleType" lit:"rule_type"`
+ Subject string `json:"subject" lit:"subject"`
+ Body string `json:"body" lit:"body"`
+ URL string `json:"url" lit:"url"`
+ Severity string `json:"severity" lit:"severity"`
+ Urgency string `json:"urgency" lit:"urgency"`
+ Status string `json:"status" lit:"status"`
+ DedupKey string `json:"-" lit:"dedup_key"`
+ EventCount int `json:"eventCount" lit:"event_count"`
+ LastEventAt time.Time `json:"lastEventAt" lit:"last_event_at"`
+ EscalationLevel int `json:"escalationLevel" lit:"escalation_level"`
+ RepeatIteration int `json:"repeatIteration" lit:"repeat_iteration"`
+ NextEscalationAt *time.Time `json:"nextEscalationAt" lit:"next_escalation_at"`
+ LastEscalatedAt *time.Time `json:"lastEscalatedAt" lit:"last_escalated_at"`
+ AcknowledgedBy *int `json:"acknowledgedBy" lit:"acknowledged_by"`
+ AcknowledgedVia string `json:"acknowledgedVia" lit:"acknowledged_via"`
+ AcknowledgedAt *time.Time `json:"acknowledgedAt" lit:"acknowledged_at"`
+ ResolvedBy *int `json:"resolvedBy" lit:"resolved_by"`
+ ResolvedAt *time.Time `json:"resolvedAt" lit:"resolved_at"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+}
+
+type UserContactMethod struct {
+ Id int `json:"id" lit:"id"`
+ UserId int `json:"userId" lit:"user_id"`
+ MethodType string `json:"methodType" lit:"method_type"`
+ Config JSONText `json:"config" lit:"config"`
+ Enabled bool `json:"enabled" lit:"enabled"`
+ Verified bool `json:"verified" lit:"verified"`
+ VerificationCodeHash string `json:"-" lit:"verification_code_hash"`
+ VerificationExpiresAt *time.Time `json:"-" lit:"verification_expires_at"`
+ VerificationAttempts int `json:"-" lit:"verification_attempts"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+}
+
+const (
+ UrgencyHigh = "high"
+ UrgencyLow = "low"
+)
+
+type UserNotificationRule struct {
+ Id int `json:"id" lit:"id"`
+ UserId int `json:"userId" lit:"user_id"`
+ Urgency string `json:"urgency" lit:"urgency"`
+ Position int `json:"position" lit:"position"`
+ DelayMinutes int `json:"delayMinutes" lit:"delay_minutes"`
+ ContactMethodId int `json:"contactMethodId" lit:"contact_method_id"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+}
+
+const (
+ PageNotificationPending = "pending"
+ PageNotificationSent = "sent"
+ PageNotificationFailed = "failed"
+ PageNotificationCancelled = "cancelled"
+)
+
+type PageNotification struct {
+ Id int `json:"id" lit:"id"`
+ PageId int `json:"pageId" lit:"page_id"`
+ Level int `json:"level" lit:"level"`
+ Iteration int `json:"iteration" lit:"iteration"`
+ UserId *int `json:"userId" lit:"user_id"`
+ TargetDesc string `json:"targetDesc" lit:"target_desc"`
+ MethodType string `json:"methodType" lit:"method_type"`
+ Status string `json:"status" lit:"status"`
+ ErrorMsg string `json:"errorMsg" lit:"error_msg"`
+ ScheduledFor *time.Time `json:"scheduledFor" lit:"scheduled_for"`
+ AckTokenHash string `json:"-" lit:"ack_token_hash"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ SentAt *time.Time `json:"sentAt" lit:"sent_at"`
+}
diff --git a/backend/app/models/models.go b/backend/app/models/models.go
index e2b1984a..50a9e216 100644
--- a/backend/app/models/models.go
+++ b/backend/app/models/models.go
@@ -12,6 +12,20 @@ func (metricRegistryNaming) GetTableNameFromStructName(string) string {
return "metric_registry"
}
+// The default pluralizer would produce "escalation_policys".
+type escalationPolicyNaming struct{ lit.DefaultDbNamingStrategy }
+
+func (escalationPolicyNaming) GetTableNameFromStructName(string) string {
+ return "escalation_policies"
+}
+
+// The default pluralizer would produce "outbox_deliveries".
+type notificationOutboxNaming struct{ lit.DefaultDbNamingStrategy }
+
+func (notificationOutboxNaming) GetTableNameFromStructName(string) string {
+ return "notification_outbox"
+}
+
func Init(driver lit.Driver) {
lit.RegisterModel[Project](driver)
lit.RegisterModel[User](driver)
@@ -40,6 +54,22 @@ func Init(driver lit.Driver) {
lit.RegisterModel[NotificationChannel](driver)
lit.RegisterModel[NotificationRule](driver)
lit.RegisterModel[NotificationRuleWithChannel](driver)
+ lit.RegisterModel[Team](driver)
+ lit.RegisterModel[TeamWithCounts](driver)
+ lit.RegisterModel[TeamProjectRow](driver)
+ lit.RegisterModel[TeamMember](driver)
+ lit.RegisterModel[TeamMemberWithUser](driver)
+ lit.RegisterModel[ProjectTeam](driver)
+ lit.RegisterModel[OncallSchedule](driver)
+ lit.RegisterModel[OncallOverride](driver)
+ lit.RegisterModelWithNaming[EscalationPolicy](driver, escalationPolicyNaming{})
+ lit.RegisterModel[Page](driver)
+ lit.RegisterModel[UserContactMethod](driver)
+ lit.RegisterModel[UserNotificationRule](driver)
+ lit.RegisterModel[PageNotification](driver)
+ lit.RegisterModelWithNaming[OutboxDelivery](driver, notificationOutboxNaming{})
+ lit.RegisterModel[OutboxRuleEnqueue](driver)
+ lit.RegisterModel[OutboxHealthCounts](driver)
for _, register := range ExtensionModelRegistrations {
register(driver)
diff --git a/backend/app/models/notification_message.model.go b/backend/app/models/notification_message.model.go
new file mode 100644
index 00000000..4f9e1982
--- /dev/null
+++ b/backend/app/models/notification_message.model.go
@@ -0,0 +1,30 @@
+package models
+
+type NotificationSeverity string
+
+const (
+ NotificationSeverityInfo NotificationSeverity = "info"
+ NotificationSeverityWarning NotificationSeverity = "warning"
+ NotificationSeverityCritical NotificationSeverity = "critical"
+)
+
+// NotificationMessage is what adapters deliver. Field names are the persisted
+// JSON shape of notification_outbox.message, so renaming a field is a
+// wire-format change for rows in flight across a restart or upgrade.
+type NotificationMessage struct {
+ Subject string
+ Body string
+ HTMLBody string
+ Severity NotificationSeverity
+ RuleType string
+ RuleName string
+ URL string
+ Endpoint string
+
+ // DedupToken is the stable identity of what fired within the rule
+ // (exception hash, endpoint, task or metric name; empty for rule-level
+ // conditions). Page dedup keys are built from it — never from URL, which
+ // can embed a time window that changes every fire. Not persisted: dedup
+ // only matters at page-open time.
+ DedupToken string `json:"-"`
+}
diff --git a/backend/app/models/oncall.model.go b/backend/app/models/oncall.model.go
new file mode 100644
index 00000000..cdbaa9a4
--- /dev/null
+++ b/backend/app/models/oncall.model.go
@@ -0,0 +1,105 @@
+package models
+
+import (
+ "time"
+
+ "github.com/google/uuid"
+)
+
+type Team struct {
+ Id int `json:"id" lit:"id"`
+ OrganizationId int `json:"organizationId" lit:"organization_id"`
+ Name string `json:"name" lit:"name"`
+ Description string `json:"description" lit:"description"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+}
+
+type TeamMember struct {
+ Id int `json:"id" lit:"id"`
+ TeamId int `json:"teamId" lit:"team_id"`
+ UserId int `json:"userId" lit:"user_id"`
+ Position int `json:"position" lit:"position"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+}
+
+type TeamMemberWithUser struct {
+ TeamId int `json:"teamId" lit:"team_id"`
+ UserId int `json:"userId" lit:"user_id"`
+ Position int `json:"position" lit:"position"`
+ Name string `json:"name" lit:"name"`
+ Email string `json:"email" lit:"email"`
+}
+
+type ProjectTeam struct {
+ Id int `json:"id" lit:"id"`
+ ProjectId uuid.UUID `json:"projectId" lit:"project_id"`
+ TeamId int `json:"teamId" lit:"team_id"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+}
+
+type OncallSchedule struct {
+ Id int `json:"id" lit:"id"`
+ OrganizationId int `json:"organizationId" lit:"organization_id"`
+ TeamId int `json:"teamId" lit:"team_id"`
+ Name string `json:"name" lit:"name"`
+ Description string `json:"description" lit:"description"`
+ Timezone string `json:"timezone" lit:"timezone"`
+ Definition JSONText `json:"definition" lit:"definition"`
+ CreatedBy *int `json:"createdBy" lit:"created_by"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+}
+
+type OncallOverride struct {
+ Id int `json:"id" lit:"id"`
+ ScheduleId int `json:"scheduleId" lit:"schedule_id"`
+ UserId int `json:"userId" lit:"user_id"`
+ StartAt time.Time `json:"startAt" lit:"start_at"`
+ EndAt time.Time `json:"endAt" lit:"end_at"`
+ CreatedBy *int `json:"createdBy" lit:"created_by"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+}
+
+type TeamWithCounts struct {
+ Id int `json:"id" lit:"id"`
+ OrganizationId int `json:"organizationId" lit:"organization_id"`
+ Name string `json:"name" lit:"name"`
+ Description string `json:"description" lit:"description"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
+ MemberCount int `json:"memberCount" lit:"member_count"`
+ ProjectCount int `json:"projectCount" lit:"project_count"`
+ ScheduleCount int `json:"scheduleCount" lit:"schedule_count"`
+}
+
+type TeamProjectRow struct {
+ TeamId int `json:"teamId" lit:"team_id"`
+ ProjectId uuid.UUID `json:"projectId" lit:"project_id"`
+ Name string `json:"name" lit:"name"`
+}
+
+type OncallScheduleDefinition struct {
+ SchemaVersion int `json:"schemaVersion"`
+ Layers []OncallLayer `json:"layers"`
+}
+
+type OncallLayer struct {
+ Id string `json:"id"`
+ Name string `json:"name"`
+ RotationType string `json:"rotationType"`
+ HandoffTime string `json:"handoffTime"`
+ HandoffDay int `json:"handoffDay"`
+ IntervalDays int `json:"intervalDays"`
+ RotationStart string `json:"rotationStart"`
+ UserIds []int `json:"userIds"`
+ Restrictions []OncallRestriction `json:"restrictions"`
+}
+
+type OncallRestriction struct {
+ Type string `json:"type"`
+ StartTime string `json:"startTime"`
+ EndTime string `json:"endTime"`
+ StartDay int `json:"startDay"`
+ EndDay int `json:"endDay"`
+}
diff --git a/backend/app/models/outbox.model.go b/backend/app/models/outbox.model.go
new file mode 100644
index 00000000..6b8ecbfb
--- /dev/null
+++ b/backend/app/models/outbox.model.go
@@ -0,0 +1,53 @@
+package models
+
+import (
+ "time"
+
+ "github.com/google/uuid"
+)
+
+const (
+ OutboxKindRule = "rule"
+ OutboxKindPage = "page"
+ OutboxKindVerification = "verification"
+
+ OutboxPending = "pending"
+ OutboxSending = "sending"
+ OutboxSent = "sent"
+ OutboxFailed = "failed"
+ OutboxCancelled = "cancelled"
+)
+
+// OutboxDelivery is one persisted notification send. AdapterConfig can hold
+// secrets (SMTP credentials, webhook secrets), so it is never serialized to
+// JSON responses.
+type OutboxDelivery struct {
+ Id int `json:"id" lit:"id"`
+ Kind string `json:"kind" lit:"kind"`
+ Status string `json:"status" lit:"status"`
+ AdapterType string `json:"adapterType" lit:"adapter_type"`
+ AdapterConfig JSONText `json:"-" lit:"adapter_config"`
+ Message JSONText `json:"-" lit:"message"`
+ Attempts int `json:"attempts" lit:"attempts"`
+ NextAttemptAt time.Time `json:"nextAttemptAt" lit:"next_attempt_at"`
+ ClaimedAt *time.Time `json:"claimedAt" lit:"claimed_at"`
+ CancelKey string `json:"cancelKey" lit:"cancel_key"`
+ PageNotificationId *int `json:"pageNotificationId" lit:"page_notification_id"`
+ RuleId *int `json:"ruleId" lit:"rule_id"`
+ ProjectId *uuid.UUID `json:"projectId" lit:"project_id"`
+ ChannelName string `json:"channelName" lit:"channel_name"`
+ LastError string `json:"lastError" lit:"last_error"`
+ CreatedAt time.Time `json:"createdAt" lit:"created_at"`
+ SentAt *time.Time `json:"sentAt" lit:"sent_at"`
+}
+
+type OutboxRuleEnqueue struct {
+ RuleId int `lit:"rule_id"`
+ LastEnqueuedAt time.Time `lit:"last_enqueued_at"`
+}
+
+type OutboxHealthCounts struct {
+ PendingCount int `lit:"pending_count"`
+ SendingCount int `lit:"sending_count"`
+ FailedCount int `lit:"failed_count"`
+}
diff --git a/backend/app/monitoring/outbox_reporter.go b/backend/app/monitoring/outbox_reporter.go
new file mode 100644
index 00000000..f332960d
--- /dev/null
+++ b/backend/app/monitoring/outbox_reporter.go
@@ -0,0 +1,59 @@
+package monitoring
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ traceway "go.tracewayapp.com"
+
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+)
+
+const outboxReportInterval = 10 * time.Second
+
+type outboxBaselines struct {
+ sent uint64
+ terminalFailures uint64
+ first bool
+}
+
+func StartOutboxReporter(ctx context.Context) {
+ go func() {
+ defer traceway.Recover()
+
+ ticker := time.NewTicker(outboxReportInterval)
+ defer ticker.Stop()
+
+ baselines := &outboxBaselines{first: true}
+ reportOutboxOnce(baselines)
+
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ reportOutboxOnce(baselines)
+ }
+ }
+ }()
+}
+
+func reportOutboxOnce(baselines *outboxBaselines) {
+ stats, err := outbox.HealthSnapshot()
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to snapshot outbox health for metrics: %w", err))
+ return
+ }
+ traceway.CaptureMetric("traceway.outbox.pending", float64(stats.Pending))
+ traceway.CaptureMetric("traceway.outbox.sending", float64(stats.Sending))
+ traceway.CaptureMetric("traceway.outbox.oldest_pending_sec", float64(stats.OldestPendingAgeSec))
+ traceway.CaptureMetric("traceway.outbox.failed_rows", float64(stats.FailedRows))
+ if !baselines.first {
+ traceway.CaptureMetric("traceway.outbox.sent.delta", float64(stats.SentTotal-baselines.sent))
+ traceway.CaptureMetric("traceway.outbox.terminal_failures.delta", float64(stats.TerminalFailuresTotal-baselines.terminalFailures))
+ }
+ baselines.sent = stats.SentTotal
+ baselines.terminalFailures = stats.TerminalFailuresTotal
+ baselines.first = false
+}
diff --git a/backend/app/notifications/adapter_email.go b/backend/app/notifications/adapter_email.go
index 963d82e7..e8f85c0f 100644
--- a/backend/app/notifications/adapter_email.go
+++ b/backend/app/notifications/adapter_email.go
@@ -13,6 +13,8 @@ import (
"github.com/tracewayapp/traceway/backend/app/services"
)
+const smtpTimeout = 10 * time.Second
+
type EmailAdapter struct {
Recipients []string `json:"recipients"`
}
@@ -69,11 +71,19 @@ func (a *EmailAdapter) Send(ctx context.Context, msg Message) error {
}
func sendMailWithTimeout(ctx context.Context, addr string, auth smtp.Auth, from string, to []string, msg []byte) error {
- conn, err := net.DialTimeout("tcp", addr, 10*time.Second)
+ dialer := net.Dialer{Timeout: smtpTimeout}
+ conn, err := dialer.DialContext(ctx, "tcp", addr)
if err != nil {
return fmt.Errorf("SMTP dial failed: %w", err)
}
+ // Must be set before smtp.NewClient, which blocks reading the server greeting.
+ deadline := time.Now().Add(smtpTimeout)
+ if ctxDeadline, ok := ctx.Deadline(); ok && ctxDeadline.Before(deadline) {
+ deadline = ctxDeadline
+ }
+ conn.SetDeadline(deadline)
+
host, _, _ := net.SplitHostPort(addr)
client, err := smtp.NewClient(conn, host)
if err != nil {
@@ -82,8 +92,6 @@ func sendMailWithTimeout(ctx context.Context, addr string, auth smtp.Auth, from
}
defer client.Close()
- conn.SetDeadline(time.Now().Add(10 * time.Second))
-
if ok, _ := client.Extension("STARTTLS"); ok {
if err := client.StartTLS(&tls.Config{ServerName: host}); err != nil {
return fmt.Errorf("SMTP STARTTLS failed: %w", err)
diff --git a/backend/app/notifications/adapter_sms.go b/backend/app/notifications/adapter_sms.go
new file mode 100644
index 00000000..8dee1e77
--- /dev/null
+++ b/backend/app/notifications/adapter_sms.go
@@ -0,0 +1,119 @@
+package notifications
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "net/url"
+ "regexp"
+ "strings"
+ "time"
+ "unicode/utf8"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+)
+
+var e164Pattern = regexp.MustCompile(`^\+[1-9]\d{1,14}$`)
+
+// SmsAdapter delivers via the Twilio Messages API. Without Twilio credentials
+// SMS is not offered at all (contact-method creation is rejected and the
+// escalator skips SMS methods), so Send only has to fail loudly for deliveries
+// already queued when the credentials were removed.
+type SmsAdapter struct {
+ PhoneNumber string `json:"phoneNumber"`
+}
+
+func (a *SmsAdapter) Type() string { return "sms" }
+
+func (a *SmsAdapter) Validate() error {
+ if !e164Pattern.MatchString(a.PhoneNumber) {
+ return fmt.Errorf("The phone number must be in international E.164 format (e.g. +12025550123).")
+ }
+ return nil
+}
+
+func (a *SmsAdapter) Send(ctx context.Context, msg Message) error {
+ cfg := config.Config
+ if !cfg.TwilioEnabled() {
+ // Never report success here: the outbox would mark the row sent and
+ // the page would look delivered to a phone that got nothing. The
+ // number is masked because errors reach logs and the issues feed.
+ return fmt.Errorf("sms delivery to %s is not configured: no Twilio credentials", MaskPhoneNumber(a.PhoneNumber))
+ }
+
+ form := url.Values{"To": {a.PhoneNumber}, "Body": {smsText(msg)}}
+ if cfg.TwilioMessagingServiceSID != "" {
+ form.Set("MessagingServiceSid", cfg.TwilioMessagingServiceSID)
+ } else {
+ form.Set("From", cfg.TwilioFromNumber)
+ }
+
+ endpoint := "https://api.twilio.com/2010-04-01/Accounts/" + url.PathEscape(cfg.TwilioAccountSID) + "/Messages.json"
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, strings.NewReader(form.Encode()))
+ if err != nil {
+ return err
+ }
+ req.SetBasicAuth(cfg.TwilioAccountSID, cfg.TwilioAuthToken)
+ req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+
+ client := &http.Client{Timeout: 10 * time.Second}
+ resp, err := client.Do(req)
+ if err != nil {
+ return fmt.Errorf("twilio request failed: %w", err)
+ }
+ defer resp.Body.Close()
+
+ if resp.StatusCode == http.StatusOK || resp.StatusCode == http.StatusCreated {
+ return nil
+ }
+ payload, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
+ var twilioError struct {
+ Code int `json:"code"`
+ Message string `json:"message"`
+ }
+ if json.Unmarshal(payload, &twilioError) == nil && twilioError.Message != "" {
+ message := strings.ReplaceAll(twilioError.Message, a.PhoneNumber, MaskPhoneNumber(a.PhoneNumber))
+ return fmt.Errorf("twilio returned %d (code %d): %s", resp.StatusCode, twilioError.Code, message)
+ }
+ return fmt.Errorf("twilio returned %d", resp.StatusCode)
+}
+
+// MaskPhoneNumber keeps only the last 4 digits, so a number can be shown in
+// logs, errors and delivery records without disclosing it.
+func MaskPhoneNumber(number string) string {
+ if len(number) <= 4 {
+ return "***"
+ }
+ return "***" + number[len(number)-4:]
+}
+
+const smsSubjectLimit = 110
+
+// smsText builds the compact SMS body: severity tag, truncated subject, and
+// the ack link. Kept within roughly two GSM segments.
+func smsText(msg Message) string {
+ tag := "[Traceway]"
+ switch msg.Severity {
+ case SeverityCritical:
+ tag = "[Traceway CRITICAL]"
+ case SeverityWarning:
+ tag = "[Traceway WARNING]"
+ }
+ subject := msg.Subject
+ if len(subject) > smsSubjectLimit {
+ // Truncate on a rune boundary: a byte slice can split a multi-byte
+ // character and produce invalid UTF-8.
+ cut := smsSubjectLimit - 1
+ for cut > 0 && !utf8.RuneStart(subject[cut]) {
+ cut--
+ }
+ subject = subject[:cut] + "…"
+ }
+ text := tag + " " + subject
+ if msg.URL != "" {
+ text += " Ack: " + msg.URL
+ }
+ return text
+}
diff --git a/backend/app/notifications/adapter_sms_test.go b/backend/app/notifications/adapter_sms_test.go
new file mode 100644
index 00000000..6b41db3e
--- /dev/null
+++ b/backend/app/notifications/adapter_sms_test.go
@@ -0,0 +1,120 @@
+package notifications
+
+import (
+ "context"
+ "strings"
+ "testing"
+ "unicode/utf8"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+)
+
+func TestSmsValidateE164(t *testing.T) {
+ cases := []struct {
+ number string
+ valid bool
+ }{
+ {"+12025550123", true},
+ {"+381641234567", true},
+ {"+4915112345678", true},
+ {"12025550123", false},
+ {"+0123456", false},
+ {"+1", false},
+ {"+1202555012345678901", false},
+ {"+1 202 555 0123", false},
+ {"", false},
+ }
+ for _, tc := range cases {
+ adapter := &SmsAdapter{PhoneNumber: tc.number}
+ err := adapter.Validate()
+ if tc.valid && err != nil {
+ t.Errorf("%q should validate, got %v", tc.number, err)
+ }
+ if !tc.valid && err == nil {
+ t.Errorf("%q should be rejected", tc.number)
+ }
+ }
+}
+
+func TestSmsTextComposition(t *testing.T) {
+ msg := Message{Subject: "Error rate breached on checkout-api", Severity: SeverityCritical, URL: "https://x.example/ack/twk_abc"}
+ text := smsText(msg)
+ if !strings.HasPrefix(text, "[Traceway CRITICAL] ") {
+ t.Errorf("missing severity tag: %q", text)
+ }
+ if !strings.Contains(text, "Ack: https://x.example/ack/twk_abc") {
+ t.Errorf("missing ack link: %q", text)
+ }
+
+ long := Message{Subject: strings.Repeat("x", 300), Severity: SeverityInfo, URL: "https://x.example/ack/twk_abc"}
+ longText := smsText(long)
+ if !strings.Contains(longText, "…") {
+ t.Errorf("long subject should be truncated: %d chars", len(longText))
+ }
+ if !strings.Contains(longText, "twk_abc") {
+ t.Error("truncation must never eat the ack link")
+ }
+
+ multibyte := Message{Subject: strings.Repeat("ж", 120), Severity: SeverityInfo}
+ multibyteText := smsText(multibyte)
+ if !utf8.ValidString(multibyteText) {
+ t.Errorf("truncation split a multi-byte rune: %q", multibyteText)
+ }
+ if !strings.Contains(multibyteText, "…") {
+ t.Error("multi-byte subject should be truncated")
+ }
+}
+
+func TestMaskPhoneNumber(t *testing.T) {
+ cases := map[string]string{
+ "+12025550123": "***0123",
+ "+381641234567": "***4567",
+ "+1": "***",
+ "": "***",
+ }
+ for number, want := range cases {
+ if got := MaskPhoneNumber(number); got != want {
+ t.Errorf("MaskPhoneNumber(%q) = %q, want %q", number, got, want)
+ }
+ }
+}
+
+// Without Twilio credentials SMS is never offered, so a delivery can only
+// reach Send if it was queued before the credentials were removed. It must
+// fail rather than report a success the phone never saw, and it must not leak
+// the full number into the error.
+func TestSmsSendFailsWithoutTwilio(t *testing.T) {
+ if config.Config == nil {
+ config.Init(config.LoadFromEnv())
+ }
+ if config.Config.TwilioEnabled() {
+ t.Skip("Twilio configured in this environment")
+ }
+ adapter := &SmsAdapter{PhoneNumber: "+12025550123"}
+ err := adapter.Send(context.Background(), Message{Subject: "s", Severity: SeverityInfo})
+ if err == nil {
+ t.Fatal("send without Twilio credentials must fail, not silently succeed")
+ }
+ if strings.Contains(err.Error(), "+12025550123") {
+ t.Errorf("error must not contain the full phone number: %v", err)
+ }
+}
+
+func TestTwilioEnabledRequiresSender(t *testing.T) {
+ cases := []struct {
+ name string
+ cfg config.Cfg
+ want bool
+ }{
+ {"nothing set", config.Cfg{}, false},
+ {"credentials without a sender", config.Cfg{TwilioAccountSID: "AC", TwilioAuthToken: "t"}, false},
+ {"sender without credentials", config.Cfg{TwilioFromNumber: "+12025550123"}, false},
+ {"credentials with from number", config.Cfg{TwilioAccountSID: "AC", TwilioAuthToken: "t", TwilioFromNumber: "+12025550123"}, true},
+ {"credentials with messaging service", config.Cfg{TwilioAccountSID: "AC", TwilioAuthToken: "t", TwilioMessagingServiceSID: "MG"}, true},
+ }
+ for _, tc := range cases {
+ if got := tc.cfg.TwilioEnabled(); got != tc.want {
+ t.Errorf("%s: TwilioEnabled() = %v, want %v", tc.name, got, tc.want)
+ }
+ }
+}
diff --git a/backend/app/notifications/adapters.go b/backend/app/notifications/adapters.go
index 966d2158..02389134 100644
--- a/backend/app/notifications/adapters.go
+++ b/backend/app/notifications/adapters.go
@@ -4,26 +4,22 @@ import (
"context"
"encoding/json"
"fmt"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
)
-type Severity string
+// Message and Severity live in models (as NotificationMessage /
+// NotificationSeverity) so the outbox package can persist them without
+// importing this package; the aliases keep every call site unchanged.
+type Severity = models.NotificationSeverity
const (
- SeverityInfo Severity = "info"
- SeverityWarning Severity = "warning"
- SeverityCritical Severity = "critical"
+ SeverityInfo = models.NotificationSeverityInfo
+ SeverityWarning = models.NotificationSeverityWarning
+ SeverityCritical = models.NotificationSeverityCritical
)
-type Message struct {
- Subject string
- Body string
- HTMLBody string
- Severity Severity
- RuleType string
- RuleName string
- URL string
- Endpoint string
-}
+type Message = models.NotificationMessage
type Adapter interface {
Type() string
@@ -69,6 +65,12 @@ func NewAdapter(channelType string, configJSON json.RawMessage) (Adapter, error)
return nil, fmt.Errorf("invalid telegram config: %w", err)
}
return &cfg, nil
+ case "sms":
+ var cfg SmsAdapter
+ if err := json.Unmarshal(configJSON, &cfg); err != nil {
+ return nil, fmt.Errorf("invalid sms config: %w", err)
+ }
+ return &cfg, nil
default:
return nil, fmt.Errorf("unknown channel type: %s", channelType)
}
diff --git a/backend/app/notifications/dispatch.go b/backend/app/notifications/dispatch.go
index c02e5554..09d43436 100644
--- a/backend/app/notifications/dispatch.go
+++ b/backend/app/notifications/dispatch.go
@@ -3,6 +3,7 @@ package notifications
import (
"context"
"database/sql"
+ "encoding/json"
"fmt"
"strings"
"time"
@@ -10,6 +11,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
traceway "go.tracewayapp.com"
@@ -23,39 +25,76 @@ func sanitizeForDB(s string) string {
return s
}
-func dispatch(rule *models.NotificationRuleWithChannel, msg Message) {
+// dispatch durably commits the notification: escalation rules open a page,
+// everything else lands in the outbox for the drain worker to deliver with
+// retries. It performs no network I/O. Returns true only when the commitment
+// is persisted; callers use it to gate dedup recording, and the cooldown is
+// recorded here at enqueue time (the durable promise) so a rule cannot
+// re-fire while the outbox is still retrying.
+func dispatch(rule *models.NotificationRuleWithChannel, msg Message) bool {
channel, dbErr := db.ExecuteTransaction(func(tx *sql.Tx) (*models.NotificationChannel, error) {
return transactional.NotificationChannelRepository.FindById(tx, rule.ChannelId)
})
if dbErr != nil || channel == nil {
recordFiredNotification(rule, msg, "failed", "failed to load channel")
- return
+ return false
}
- adapter, err := NewAdapter(channel.ChannelType, channel.Config)
- if err != nil {
- recordFiredNotification(rule, msg, "failed", err.Error())
- return
- }
-
- ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
- defer cancel()
-
msg.RuleType = rule.RuleType
msg.RuleName = rule.Name
if rule.Severity != "" {
msg.Severity = Severity(rule.Severity)
}
- err = adapter.Send(ctx, msg)
+ // Escalation channels do not send anything themselves: they open (or
+ // dedup into) an on-call page, and the escalator + outbox do all
+ // notifying. The page open is itself the durable commitment.
+ if channel.ChannelType == "escalation" {
+ if pageOpener == nil {
+ recordFiredNotification(rule, msg, "failed", "escalation pager not initialized")
+ return false
+ }
+ opened, err := pageOpener(channel.Config, rule, msg)
+ if err != nil {
+ recordFiredNotification(rule, msg, "failed", err.Error())
+ traceway.CaptureException(fmt.Errorf("failed to open page (rule=%d): %w", rule.Id, err))
+ return false
+ }
+ cooldowns.recordFire(rule.Id)
+ status := "sent"
+ if !opened {
+ status = "deduped"
+ }
+ recordFiredNotification(rule, msg, status, "")
+ return true
+ }
+
+ ruleId := rule.Id
+ projectId := rule.ProjectId
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return outbox.Enqueue(tx, outbox.Delivery{
+ Kind: models.OutboxKindRule,
+ AdapterType: channel.ChannelType,
+ // Snapshot: later channel edits do not affect queued sends.
+ AdapterConfig: json.RawMessage(channel.Config),
+ Message: msg,
+ RuleId: &ruleId,
+ ProjectId: &projectId,
+ ChannelName: channel.Name,
+ })
+ })
if err != nil {
- recordFiredNotification(rule, msg, "failed", err.Error())
- traceway.CaptureException(fmt.Errorf("notification dispatch failed (rule=%d, channel=%s): %w", rule.Id, rule.ChannelName, err))
- return
+ // No outbox row exists, so the terminal hook can never record this
+ // delivery; the audit row must be written here or the notification
+ // vanishes from the history entirely.
+ recordFiredNotification(rule, msg, "failed", "failed to enqueue: "+err.Error())
+ traceway.CaptureException(fmt.Errorf("failed to enqueue notification (rule=%d, channel=%s): %w", rule.Id, rule.ChannelName, err))
+ return false
}
cooldowns.recordFire(rule.Id)
- recordFiredNotification(rule, msg, "sent", "")
+ outbox.Wake()
+ return true
}
func recordFiredNotification(rule *models.NotificationRuleWithChannel, msg Message, status string, errorMsg string) {
diff --git a/backend/app/notifications/dispatch_outbox_test.go b/backend/app/notifications/dispatch_outbox_test.go
new file mode 100644
index 00000000..f58306d2
--- /dev/null
+++ b/backend/app/notifications/dispatch_outbox_test.go
@@ -0,0 +1,152 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package notifications
+
+import (
+ "database/sql"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+type dispatchFixture struct {
+ Rule *models.NotificationRuleWithChannel
+ Channel *models.NotificationChannel
+}
+
+func setupDispatchDB(t *testing.T) *dispatchFixture {
+ t.Helper()
+
+ dbtest.SetupSQLite(t)
+ t.Cleanup(func() {
+ cooldowns.mu.Lock()
+ cooldowns.fired = make(map[int]time.Time)
+ cooldowns.mu.Unlock()
+ })
+
+ fixture := &dispatchFixture{}
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ org, err := transactional.OrganizationRepository.Create(tx, "Acme", "UTC")
+ if err != nil {
+ return struct{}{}, err
+ }
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "api", "gin", org.Id)
+ if err != nil {
+ return struct{}{}, err
+ }
+ now := time.Now().UTC()
+ channel := &models.NotificationChannel{
+ ProjectId: project.Id,
+ Name: "Ops Slack",
+ ChannelType: "slack",
+ Config: []byte(`{"webhookUrl":"https://hooks.example.com/original"}`),
+ Enabled: true,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+ channelId, err := transactional.NotificationChannelRepository.Create(tx, channel)
+ if err != nil {
+ return struct{}{}, err
+ }
+ channel.Id = channelId
+ fixture.Channel = channel
+ fixture.Rule = &models.NotificationRuleWithChannel{
+ Id: 42, ProjectId: project.Id, ChannelId: channelId,
+ Name: "New errors", RuleType: "new_error", CooldownMinutes: 15,
+ ChannelType: "slack", ChannelName: "Ops Slack",
+ }
+ return struct{}{}, nil
+ })
+ if err != nil {
+ t.Fatalf("seed: %v", err)
+ }
+ return fixture
+}
+
+func outboxRows(t *testing.T) []*models.OutboxDelivery {
+ t.Helper()
+ rows, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindDue(tx, time.Now().UTC().Add(time.Minute), 100)
+ })
+ if err != nil {
+ t.Fatalf("load outbox rows: %v", err)
+ }
+ return rows
+}
+
+func TestDispatchEnqueuesSnapshot(t *testing.T) {
+ fixture := setupDispatchDB(t)
+
+ if !dispatch(fixture.Rule, Message{Subject: "s", Body: "b", Severity: SeverityCritical}) {
+ t.Fatal("dispatch should report a durable enqueue")
+ }
+ rows := outboxRows(t)
+ if len(rows) != 1 {
+ t.Fatalf("expected 1 outbox row, got %d", len(rows))
+ }
+ row := rows[0]
+ if row.AdapterType != "slack" || string(row.AdapterConfig) != `{"webhookUrl":"https://hooks.example.com/original"}` {
+ t.Errorf("snapshot mismatch: %s %s", row.AdapterType, row.AdapterConfig)
+ }
+ if row.RuleId == nil || *row.RuleId != 42 || row.ChannelName != "Ops Slack" {
+ t.Errorf("bookkeeping fields wrong: %+v", row)
+ }
+
+ // Mutating the channel after enqueue must not affect the queued row.
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ fixture.Channel.Config = []byte(`{"webhookUrl":"https://hooks.example.com/CHANGED"}`)
+ return struct{}{}, transactional.NotificationChannelRepository.Update(tx, fixture.Channel)
+ })
+ if err != nil {
+ t.Fatalf("update channel: %v", err)
+ }
+ if row := outboxRows(t)[0]; string(row.AdapterConfig) != `{"webhookUrl":"https://hooks.example.com/original"}` {
+ t.Errorf("queued row changed after channel edit: %s", row.AdapterConfig)
+ }
+
+ if cooldowns.canFire(fixture.Rule.Id, fixture.Rule.CooldownMinutes) {
+ t.Error("cooldown should be recorded at enqueue")
+ }
+}
+
+func TestDispatchFailureLeavesNoCooldown(t *testing.T) {
+ fixture := setupDispatchDB(t)
+ fixture.Rule.ChannelId = 99999
+
+ if dispatch(fixture.Rule, Message{Subject: "s"}) {
+ t.Fatal("dispatch with a dangling channel should fail")
+ }
+ if len(outboxRows(t)) != 0 {
+ t.Error("no outbox row should exist after a failed dispatch")
+ }
+ if !cooldowns.canFire(fixture.Rule.Id, fixture.Rule.CooldownMinutes) {
+ t.Error("cooldown must not be recorded when dispatch fails")
+ }
+}
+
+func TestSeedCooldownsIncludesOutbox(t *testing.T) {
+ fixture := setupDispatchDB(t)
+
+ if !dispatch(fixture.Rule, Message{Subject: "s"}) {
+ t.Fatal("dispatch failed")
+ }
+ // Fresh tracker simulating a restart before any terminal outcome exists.
+ cooldowns.mu.Lock()
+ cooldowns.fired = make(map[int]time.Time)
+ cooldowns.mu.Unlock()
+
+ enqueued, err := db.ExecuteTransaction(func(tx *sql.Tx) (map[int]time.Time, error) {
+ return transactional.OutboxRepository.LastEnqueuedPerRule(tx)
+ })
+ if err != nil {
+ t.Fatalf("seed query: %v", err)
+ }
+ cooldowns.seed(enqueued)
+ if cooldowns.canFire(fixture.Rule.Id, fixture.Rule.CooldownMinutes) {
+ t.Error("outbox-backed seeding should keep the rule in cooldown after restart")
+ }
+}
diff --git a/backend/app/notifications/egress.go b/backend/app/notifications/egress.go
new file mode 100644
index 00000000..5fc7edfa
--- /dev/null
+++ b/backend/app/notifications/egress.go
@@ -0,0 +1,59 @@
+package notifications
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "net"
+ "net/url"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+)
+
+// ValidateOutboundURL rejects a destination on the server's own network, unless
+// ALLOW_PRIVATE_NOTIFICATION_TARGETS is set. It guards self-scoped surfaces
+// (personal contact methods); project webhook channels are deliberately not
+// guarded.
+func ValidateOutboundURL(raw string) error {
+ if config.Config.AllowPrivateNotificationTargets == "true" {
+ return nil
+ }
+ parsed, err := url.Parse(strings.TrimSpace(raw))
+ if err != nil {
+ return errors.New("The URL is not valid.")
+ }
+ if parsed.Scheme != "http" && parsed.Scheme != "https" {
+ return errors.New("The URL must start with http:// or https://.")
+ }
+ host := parsed.Hostname()
+ if host == "" {
+ return errors.New("The URL is missing a host.")
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+ defer cancel()
+ addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host)
+ if err != nil || len(addrs) == 0 {
+ return fmt.Errorf("The host %q could not be resolved.", host)
+ }
+ for _, addr := range addrs {
+ if isPrivateIP(addr.IP) {
+ return errors.New("The URL must point at a public address.")
+ }
+ }
+ return nil
+}
+
+func isPrivateIP(ip net.IP) bool {
+ if ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() ||
+ ip.IsUnspecified() || ip.IsInterfaceLocalMulticast() {
+ return true
+ }
+ // 100.64.0.0/10 (CGNAT) and 0.0.0.0/8, which net.IP does not classify.
+ if v4 := ip.To4(); v4 != nil {
+ return v4[0] == 0 || (v4[0] == 100 && v4[1] >= 64 && v4[1] <= 127)
+ }
+ return false
+}
diff --git a/backend/app/notifications/evaluator.go b/backend/app/notifications/evaluator.go
index 2257ab41..14c1b14c 100644
--- a/backend/app/notifications/evaluator.go
+++ b/backend/app/notifications/evaluator.go
@@ -4,7 +4,6 @@ import (
"context"
"database/sql"
"fmt"
- "strconv"
"time"
"github.com/tracewayapp/traceway/backend/app/config"
@@ -24,22 +23,28 @@ func StartEvaluator(ctx context.Context) {
}
func pollInterval() time.Duration {
- seconds := 60
- if v := config.Config.NotificationPollSeconds; v != "" {
- if parsed, err := strconv.Atoi(v); err == nil && parsed >= 5 {
- seconds = parsed
- }
- }
- return time.Duration(seconds) * time.Second
+ return config.PollSeconds(config.Config.NotificationPollSeconds, 60)
}
func seedCooldowns(ctx context.Context) {
entries, err := telemetry.FiredNotificationRepository.FindLastFiredPerRule(ctx)
if err != nil {
traceway.CaptureException(fmt.Errorf("failed to seed notification cooldowns: %w", err))
+ } else {
+ cooldowns.seed(entries)
+ }
+
+ // Backstop: fired_notifications rows only exist once an outcome is
+ // terminal, so a crash between enqueue and delivery would otherwise let
+ // the rule re-fire immediately at boot while its outbox row still exists.
+ enqueued, err := db.ExecuteTransaction(func(tx *sql.Tx) (map[int]time.Time, error) {
+ return transactional.OutboxRepository.LastEnqueuedPerRule(tx)
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to seed notification cooldowns from outbox: %w", err))
return
}
- cooldowns.seed(entries)
+ cooldowns.seed(enqueued)
}
func startPolledLoop(ctx context.Context) {
diff --git a/backend/app/notifications/evaluator_event.go b/backend/app/notifications/evaluator_event.go
index ca23ee8b..7486526c 100644
--- a/backend/app/notifications/evaluator_event.go
+++ b/backend/app/notifications/evaluator_event.go
@@ -63,9 +63,11 @@ func evaluateNewError(ctx context.Context, rule *models.NotificationRuleWithChan
continue
}
- dedup.record(dedupKey)
projectName := getProjectName(rule.ProjectId)
msg := buildNewErrorMessage(details, projectName)
+ // Record before dispatch: a persistently failing dispatch retries once
+ // per cooldown window, never on every ingest event.
+ dedup.record(dedupKey)
dispatch(rule, msg)
}
}
@@ -92,9 +94,11 @@ func evaluateErrorRegression(ctx context.Context, rule *models.NotificationRuleW
}
details := getExceptionDetails(ctx, event.ProjectId, hash)
- dedup.record(dedupKey)
projectName := getProjectName(rule.ProjectId)
msg := buildErrorRegressionMessage(details, projectName)
+ // Record before dispatch: a persistently failing dispatch retries once
+ // per cooldown window, never on every ingest event.
+ dedup.record(dedupKey)
dispatch(rule, msg)
}
}
diff --git a/backend/app/notifications/evaluator_event_sqlite.go b/backend/app/notifications/evaluator_event_sqlite.go
index 513ad09d..947117a1 100644
--- a/backend/app/notifications/evaluator_event_sqlite.go
+++ b/backend/app/notifications/evaluator_event_sqlite.go
@@ -64,9 +64,11 @@ func evaluateNewError(ctx context.Context, rule *models.NotificationRuleWithChan
continue
}
- dedup.record(dedupKey)
projectName := getProjectName(rule.ProjectId)
msg := buildNewErrorMessage(details, projectName)
+ // Record before dispatch: a persistently failing dispatch retries once
+ // per cooldown window, never on every ingest event.
+ dedup.record(dedupKey)
dispatch(rule, msg)
}
}
@@ -92,9 +94,11 @@ func evaluateErrorRegression(ctx context.Context, rule *models.NotificationRuleW
}
details := getExceptionDetails(ctx, event.ProjectId, hash)
- dedup.record(dedupKey)
projectName := getProjectName(rule.ProjectId)
msg := buildErrorRegressionMessage(details, projectName)
+ // Record before dispatch: a persistently failing dispatch retries once
+ // per cooldown window, never on every ingest event.
+ dedup.record(dedupKey)
dispatch(rule, msg)
}
}
diff --git a/backend/app/notifications/evaluator_helpers.go b/backend/app/notifications/evaluator_helpers.go
index bae95702..dde1770c 100644
--- a/backend/app/notifications/evaluator_helpers.go
+++ b/backend/app/notifications/evaluator_helpers.go
@@ -91,9 +91,10 @@ func evaluateAiTraceCostEvent(rule *models.NotificationRuleWithChannel, event ho
if dedup.isDuplicate(dedupKey, time.Duration(rule.CooldownMinutes)*time.Minute) {
continue
}
- dedup.record(dedupKey)
-
msg := buildAiTraceCostMessage(at.TraceName, at.TotalCost, cfg.ThresholdCost, projectName)
+ // Record before dispatch: a persistently failing dispatch retries once
+ // per cooldown window, never on every ingest event.
+ dedup.record(dedupKey)
dispatch(rule, msg)
}
}
diff --git a/backend/app/notifications/messages.go b/backend/app/notifications/messages.go
index 9b5f27b7..99430021 100644
--- a/backend/app/notifications/messages.go
+++ b/backend/app/notifications/messages.go
@@ -28,10 +28,11 @@ func endpointTimeRangeURL(now time.Time) string {
func buildEndpointLatencyMessage(percentile string, latencyMs float64, thresholdMs float64, endpoint string, window int, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] %s latency %.0fms on %s", projectName, percentile, latencyMs, endpoint),
- Body: fmt.Sprintf("The %s latency for %s has reached %.0fms over the last %d minutes (threshold: %.0fms).", percentile, endpoint, latencyMs, window, thresholdMs),
- Severity: SeverityWarning,
- URL: endpointTimeRangeURL(time.Now()),
+ Subject: fmt.Sprintf("[%s] %s latency %.0fms on %s", projectName, percentile, latencyMs, endpoint),
+ Body: fmt.Sprintf("The %s latency for %s has reached %.0fms over the last %d minutes (threshold: %.0fms).", percentile, endpoint, latencyMs, window, thresholdMs),
+ Severity: SeverityWarning,
+ URL: endpointTimeRangeURL(time.Now()),
+ DedupToken: endpoint,
}
}
@@ -61,19 +62,21 @@ func buildMetricThresholdMessage(metricName string, value float64, operator stri
aggregation = "avg"
}
return Message{
- Subject: fmt.Sprintf("[%s] Metric %s is %.2f (threshold: %s %.2f)", projectName, metricName, value, operator, threshold),
- Body: fmt.Sprintf("The metric %s has a %s of %.2f over the last %d minutes which violates the threshold %s %.2f.", metricName, aggregation, value, window, operator, threshold),
- Severity: severity,
- URL: "/metrics?preset=1h",
+ Subject: fmt.Sprintf("[%s] Metric %s is %.2f (threshold: %s %.2f)", projectName, metricName, value, operator, threshold),
+ Body: fmt.Sprintf("The metric %s has a %s of %.2f over the last %d minutes which violates the threshold %s %.2f.", metricName, aggregation, value, window, operator, threshold),
+ Severity: severity,
+ URL: "/metrics?preset=1h",
+ DedupToken: metricName,
}
}
func buildNoDataMessage(dataType string, silenceMinutes int, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] No %s data for %d minutes", projectName, dataType, silenceMinutes),
- Body: fmt.Sprintf("No %s data has been received for the last %d minutes.", dataType, silenceMinutes),
- Severity: SeverityCritical,
- URL: "/",
+ Subject: fmt.Sprintf("[%s] No %s data for %d minutes", projectName, dataType, silenceMinutes),
+ Body: fmt.Sprintf("No %s data has been received for the last %d minutes.", dataType, silenceMinutes),
+ Severity: SeverityCritical,
+ URL: "/",
+ DedupToken: dataType,
}
}
@@ -92,10 +95,11 @@ func buildErrorCountMessage(count int64, threshold int64, window int, projectNam
func buildTaskDurationMessage(taskName string, p95Ms float64, thresholdMs float64, window int, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] Task %s P95 %.0fms exceeds %.0fms", projectName, taskName, p95Ms, thresholdMs),
- Body: fmt.Sprintf("The task %s P95 duration is %.0fms over the last %d minutes (threshold: %.0fms).", taskName, p95Ms, window, thresholdMs),
- Severity: SeverityWarning,
- URL: "/tasks?preset=1h",
+ Subject: fmt.Sprintf("[%s] Task %s P95 %.0fms exceeds %.0fms", projectName, taskName, p95Ms, thresholdMs),
+ Body: fmt.Sprintf("The task %s P95 duration is %.0fms over the last %d minutes (threshold: %.0fms).", taskName, p95Ms, window, thresholdMs),
+ Severity: SeverityWarning,
+ URL: "/tasks?preset=1h",
+ DedupToken: taskName,
}
}
@@ -105,10 +109,11 @@ func buildTaskFailureRateMessage(taskName string, rate float64, threshold float6
severity = SeverityCritical
}
return Message{
- Subject: fmt.Sprintf("[%s] Task %s failure rate %.1f%% exceeds %.1f%%", projectName, taskName, rate, threshold),
- Body: fmt.Sprintf("The task %s failed %d of %d executions (%.1f%%) over the last %d minutes (threshold: %.1f%%).", taskName, failed, total, rate, window, threshold),
- Severity: severity,
- URL: "/tasks?preset=1h",
+ Subject: fmt.Sprintf("[%s] Task %s failure rate %.1f%% exceeds %.1f%%", projectName, taskName, rate, threshold),
+ Body: fmt.Sprintf("The task %s failed %d of %d executions (%.1f%%) over the last %d minutes (threshold: %.1f%%).", taskName, failed, total, rate, window, threshold),
+ Severity: severity,
+ URL: "/tasks?preset=1h",
+ DedupToken: taskName,
}
}
@@ -131,40 +136,44 @@ func buildEndpointErrorRateMessage(endpoint string, rate float64, threshold floa
severity = SeverityCritical
}
return Message{
- Subject: fmt.Sprintf("[%s] %s error rate %.1f%%", projectName, endpoint, rate),
- Body: fmt.Sprintf("The endpoint %s has an error rate of %.1f%% (threshold: %.1f%%).", endpoint, rate, threshold),
- Severity: severity,
- URL: endpointTimeRangeURL(time.Now()),
+ Subject: fmt.Sprintf("[%s] %s error rate %.1f%%", projectName, endpoint, rate),
+ Body: fmt.Sprintf("The endpoint %s has an error rate of %.1f%% (threshold: %.1f%%).", endpoint, rate, threshold),
+ Severity: severity,
+ URL: endpointTimeRangeURL(time.Now()),
+ DedupToken: endpoint,
}
}
func buildImpactScoreCriticalMessage(endpoint string, score float64, reason string, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] Endpoint %s impact became critical", projectName, endpoint),
- Body: fmt.Sprintf("The endpoint %s has become critical (impact score: %.2f). Reason: %s", endpoint, score, reason),
- Severity: SeverityCritical,
- URL: endpointTimeRangeURL(time.Now()),
- Endpoint: endpoint,
+ Subject: fmt.Sprintf("[%s] Endpoint %s impact became critical", projectName, endpoint),
+ Body: fmt.Sprintf("The endpoint %s has become critical (impact score: %.2f). Reason: %s", endpoint, score, reason),
+ Severity: SeverityCritical,
+ URL: endpointTimeRangeURL(time.Now()),
+ Endpoint: endpoint,
+ DedupToken: endpoint,
}
}
func buildImpactScoreHighMessage(endpoint string, score float64, reason string, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] Endpoint %s impact became high", projectName, endpoint),
- Body: fmt.Sprintf("The endpoint %s has become high impact (impact score: %.2f). Reason: %s", endpoint, score, reason),
- Severity: SeverityWarning,
- URL: endpointTimeRangeURL(time.Now()),
- Endpoint: endpoint,
+ Subject: fmt.Sprintf("[%s] Endpoint %s impact became high", projectName, endpoint),
+ Body: fmt.Sprintf("The endpoint %s has become high impact (impact score: %.2f). Reason: %s", endpoint, score, reason),
+ Severity: SeverityWarning,
+ URL: endpointTimeRangeURL(time.Now()),
+ Endpoint: endpoint,
+ DedupToken: endpoint,
}
}
func buildImpactScoreMediumMessage(endpoint string, score float64, reason string, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] Endpoint %s impact became medium", projectName, endpoint),
- Body: fmt.Sprintf("The endpoint %s has become medium impact (impact score: %.2f). Reason: %s", endpoint, score, reason),
- Severity: SeverityInfo,
- URL: endpointTimeRangeURL(time.Now()),
- Endpoint: endpoint,
+ Subject: fmt.Sprintf("[%s] Endpoint %s impact became medium", projectName, endpoint),
+ Body: fmt.Sprintf("The endpoint %s has become medium impact (impact score: %.2f). Reason: %s", endpoint, score, reason),
+ Severity: SeverityInfo,
+ URL: endpointTimeRangeURL(time.Now()),
+ Endpoint: endpoint,
+ DedupToken: endpoint,
}
}
@@ -181,10 +190,11 @@ func buildAiTraceCostMessage(traceName string, cost float64, threshold float64,
severity = SeverityCritical
}
return Message{
- Subject: fmt.Sprintf("[%s] AI trace %s cost %s exceeds %s", projectName, traceName, formatCostForMessage(cost), formatCostForMessage(threshold)),
- Body: fmt.Sprintf("The AI trace \"%s\" cost %s, exceeding the threshold of %s.", traceName, formatCostForMessage(cost), formatCostForMessage(threshold)),
- Severity: severity,
- URL: "/ai-traces?preset=1h",
+ Subject: fmt.Sprintf("[%s] AI trace %s cost %s exceeds %s", projectName, traceName, formatCostForMessage(cost), formatCostForMessage(threshold)),
+ Body: fmt.Sprintf("The AI trace \"%s\" cost %s, exceeding the threshold of %s.", traceName, formatCostForMessage(cost), formatCostForMessage(threshold)),
+ Severity: severity,
+ URL: "/ai-traces?preset=1h",
+ DedupToken: traceName,
}
}
@@ -210,21 +220,23 @@ func (d ExceptionDetails) endpointForMessage() string {
func buildNewErrorMessage(details ExceptionDetails, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] New error: %s", projectName, details.ErrorType),
- Body: buildExceptionBody("A new error has been detected: "+details.ErrorType, details),
- Severity: SeverityCritical,
- URL: fmt.Sprintf("/issues/%s", details.Hash),
- Endpoint: details.endpointForMessage(),
+ Subject: fmt.Sprintf("[%s] New error: %s", projectName, details.ErrorType),
+ Body: buildExceptionBody("A new error has been detected: "+details.ErrorType, details),
+ Severity: SeverityCritical,
+ URL: fmt.Sprintf("/issues/%s", details.Hash),
+ Endpoint: details.endpointForMessage(),
+ DedupToken: details.Hash,
}
}
func buildErrorRegressionMessage(details ExceptionDetails, projectName string) Message {
return Message{
- Subject: fmt.Sprintf("[%s] Resolved error reappeared: %s", projectName, details.ErrorType),
- Body: buildExceptionBody("A previously resolved error has reappeared: "+details.ErrorType, details),
- Severity: SeverityCritical,
- URL: fmt.Sprintf("/issues/%s", details.Hash),
- Endpoint: details.endpointForMessage(),
+ Subject: fmt.Sprintf("[%s] Resolved error reappeared: %s", projectName, details.ErrorType),
+ Body: buildExceptionBody("A previously resolved error has reappeared: "+details.ErrorType, details),
+ Severity: SeverityCritical,
+ URL: fmt.Sprintf("/issues/%s", details.Hash),
+ Endpoint: details.endpointForMessage(),
+ DedupToken: details.Hash,
}
}
diff --git a/backend/app/notifications/outbox_hooks.go b/backend/app/notifications/outbox_hooks.go
new file mode 100644
index 00000000..e955e9cc
--- /dev/null
+++ b/backend/app/notifications/outbox_hooks.go
@@ -0,0 +1,64 @@
+package notifications
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ traceway "go.tracewayapp.com"
+)
+
+// AdapterSend is the outbox's registered sender: one attempt through the
+// channel adapter built from the persisted config snapshot.
+func AdapterSend(ctx context.Context, adapterType string, adapterConfig json.RawMessage, msg models.NotificationMessage) error {
+ adapter, err := NewAdapter(adapterType, adapterConfig)
+ if err != nil {
+ return err
+ }
+ return adapter.Send(ctx, msg)
+}
+
+// OnOutboxTerminal records the fired_notifications audit row for rule
+// deliveries once their outcome is final (fired_notifications is append-only
+// telemetry, so it must record the final truth). Page deliveries are logged in
+// page_notifications instead, mirrored by the drain worker.
+func OnOutboxTerminal(row *models.OutboxDelivery, status string, errorMsg string) {
+ if row.Kind != models.OutboxKindRule || row.RuleId == nil || row.ProjectId == nil {
+ return
+ }
+ var msg Message
+ if err := json.Unmarshal(row.Message, &msg); err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to decode outbox message for audit (row=%d): %w", row.Id, err))
+ return
+ }
+ ruleId := *row.RuleId
+ projectId := *row.ProjectId
+ adapterType := row.AdapterType
+ channelName := row.ChannelName
+ go func() {
+ defer traceway.Recover()
+
+ err := telemetry.FiredNotificationRepository.Insert(context.Background(), telemetry.FiredNotification{
+ ProjectId: projectId,
+ RuleId: ruleId,
+ RuleType: msg.RuleType,
+ RuleName: msg.RuleName,
+ ChannelType: adapterType,
+ ChannelName: channelName,
+ Severity: string(msg.Severity),
+ Subject: sanitizeForDB(msg.Subject),
+ Body: sanitizeForDB(msg.Body),
+ Status: status,
+ ErrorMsg: sanitizeForDB(errorMsg),
+ Endpoint: msg.Endpoint,
+ URL: msg.URL,
+ FiredAt: time.Now().UTC(),
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to record fired notification: %w", err))
+ }
+ }()
+}
diff --git a/backend/app/notifications/page_opener.go b/backend/app/notifications/page_opener.go
new file mode 100644
index 00000000..0c80d047
--- /dev/null
+++ b/backend/app/notifications/page_opener.go
@@ -0,0 +1,20 @@
+package notifications
+
+import (
+ "encoding/json"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+// PageOpener opens an on-call page for a rule that targets an escalation
+// channel, reporting whether a new page was opened (false = deduped into one
+// already unresolved). It is implemented by the oncall package and registered
+// from cmd/run.go; the indirection exists because notifications cannot import
+// oncall (oncall imports this package for Message and the adapters).
+type PageOpener func(channelConfig json.RawMessage, rule *models.NotificationRuleWithChannel, msg Message) (opened bool, err error)
+
+var pageOpener PageOpener
+
+func RegisterPageOpener(opener PageOpener) {
+ pageOpener = opener
+}
diff --git a/backend/app/oncall/ack.go b/backend/app/oncall/ack.go
new file mode 100644
index 00000000..e0141d9e
--- /dev/null
+++ b/backend/app/oncall/ack.go
@@ -0,0 +1,37 @@
+package oncall
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+const (
+ AckViaDashboard = "dashboard"
+ AckViaLink = "link"
+)
+
+// AcknowledgePage transitions open -> acknowledged and cancels every queued
+// or retrying delivery under the page's cancel key (mirroring their delivery
+// log rows). Returns false when the page was not open. The single home for
+// ack-side cancellation: used by the dashboard controller and the tokenized
+// ack endpoint alike. userId is nil for anonymous link-acks.
+func AcknowledgePage(tx *sql.Tx, pageId int, userId *int, via string, now time.Time) (bool, error) {
+ acknowledged, err := transactional.PageRepository.Acknowledge(tx, pageId, userId, via, now)
+ if err != nil || !acknowledged {
+ return acknowledged, err
+ }
+ return true, outbox.CancelByKey(tx, outbox.PageCancelKey(pageId))
+}
+
+// ResolvePage transitions open/acknowledged -> resolved and cancels queued
+// deliveries. Returns false when the page was already resolved.
+func ResolvePage(tx *sql.Tx, pageId int, userId int, now time.Time) (bool, error) {
+ resolved, err := transactional.PageRepository.Resolve(tx, pageId, userId, now)
+ if err != nil || !resolved {
+ return resolved, err
+ }
+ return true, outbox.CancelByKey(tx, outbox.PageCancelKey(pageId))
+}
diff --git a/backend/app/oncall/escalator.go b/backend/app/oncall/escalator.go
new file mode 100644
index 00000000..2392b01c
--- /dev/null
+++ b/backend/app/oncall/escalator.go
@@ -0,0 +1,589 @@
+package oncall
+
+import (
+ "context"
+ "crypto/rand"
+ "database/sql"
+ "encoding/base64"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+ "time"
+ "unicode/utf8"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional/shared"
+ traceway "go.tracewayapp.com"
+)
+
+const escalatorAdvisoryLockId = 824737002
+
+// errClaimLost aborts a claim transaction when the guarded escalation-state
+// update matches no row: a concurrent acknowledge/resolve committed after the
+// due check, and its CancelByKey could not see this claim's uncommitted
+// deliveries — so they must never commit. Rolling back is the normal loss of
+// a benign race, not an error worth reporting.
+var errClaimLost = errors.New("page claim lost to a concurrent acknowledge/resolve")
+
+var wakeCh = make(chan struct{}, 1)
+
+// Wake nudges the escalator so a freshly opened page notifies its first level
+// immediately instead of waiting for the next tick. Non-blocking.
+func Wake() {
+ select {
+ case wakeCh <- struct{}{}:
+ default:
+ }
+}
+
+func escalatorPollInterval() time.Duration {
+ return config.PollSeconds(config.Config.OncallPollSeconds, 30)
+}
+
+// StartEscalator runs the ack-based escalation loop. Each tick is purely
+// transactional: due pages advance a level and their deliveries are enqueued
+// into the notification outbox, which owns sending, retries, and crash
+// recovery. Ack/resolve cancels queued deliveries via outbox.CancelByKey.
+func StartEscalator(ctx context.Context) {
+ go func() {
+ defer traceway.Recover()
+
+ ticker := time.NewTicker(escalatorPollInterval())
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ case <-wakeCh:
+ }
+ runEscalatorTick(ctx, time.Now().UTC())
+ }
+ }()
+}
+
+func runEscalatorTick(_ context.Context, now time.Time) {
+ // Each page is claimed in its own transaction so one failing page (e.g. a
+ // target schedule whose stored definition no longer parses) cannot roll
+ // back or block escalation for every other page.
+ pageIds, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]int, error) {
+ duePages, err := transactional.PageRepository.FindDue(tx, now)
+ if err != nil {
+ return nil, err
+ }
+ ids := make([]int, 0, len(duePages))
+ for _, page := range duePages {
+ ids = append(ids, page.Id)
+ }
+ return ids, nil
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("escalator tick failed: %w", err))
+ return
+ }
+
+ enqueued := 0
+ for _, pageId := range pageIds {
+ count, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ if !db.IsSQLite() {
+ if _, err := tx.Exec(fmt.Sprintf("SELECT pg_advisory_xact_lock(%d)", escalatorAdvisoryLockId)); err != nil {
+ return 0, fmt.Errorf("failed to take escalator lock: %w", err)
+ }
+ }
+ page, err := transactional.PageRepository.FindDueById(tx, pageId, now)
+ if err != nil {
+ return 0, err
+ }
+ if page == nil {
+ // Acknowledged, resolved, or claimed by a concurrent escalator
+ // since the due list was read.
+ return 0, nil
+ }
+ return claimPageEscalation(tx, page, now)
+ })
+ if err != nil {
+ if errors.Is(err, errClaimLost) {
+ continue
+ }
+ traceway.CaptureException(fmt.Errorf("failed to escalate page %d: %w", pageId, err))
+ continue
+ }
+ enqueued += count
+ }
+ if enqueued > 0 {
+ outbox.Wake()
+ }
+}
+
+// claimPageEscalation advances one due page to its next escalation level and
+// enqueues its deliveries. Returns the number of outbox rows created. Runs
+// inside the claim transaction.
+func claimPageEscalation(tx *sql.Tx, page *models.Page, now time.Time) (int, error) {
+ definition, err := ParsePolicyDefinition(page.PolicySnapshot)
+ if err != nil || len(definition.Steps) == 0 {
+ traceway.CaptureException(fmt.Errorf("page %d has an unusable policy snapshot, stopping its escalation", page.Id))
+ return 0, updateEscalationState(tx, page.Id, page.EscalationLevel, page.RepeatIteration, nil, now)
+ }
+
+ level := page.EscalationLevel + 1
+ if level >= len(definition.Steps) {
+ return 0, updateEscalationState(tx, page.Id, page.EscalationLevel, page.RepeatIteration, nil, now)
+ }
+ step := definition.Steps[level]
+
+ enqueued := 0
+ notifiedUsers := map[int]bool{}
+ for _, target := range step.Targets {
+ switch target.Type {
+ case TargetSchedule, TargetTeam, TargetUser:
+ userIds, err := resolveUserTarget(tx, page.OrganizationId, target, now)
+ if err != nil {
+ return 0, err
+ }
+ for _, userId := range userIds {
+ if notifiedUsers[userId] {
+ continue
+ }
+ notifiedUsers[userId] = true
+ count, err := claimUserDeliveries(tx, page, level, userId, now)
+ if err != nil {
+ return 0, err
+ }
+ enqueued += count
+ }
+ case TargetChannel:
+ count, err := claimChannelDelivery(tx, page, level, target.Id, now)
+ if err != nil {
+ return 0, err
+ }
+ enqueued += count
+ default:
+ traceway.CaptureException(fmt.Errorf("page %d step %d has unknown target type %q", page.Id, level, target.Type))
+ }
+ }
+
+ lastStep := level == len(definition.Steps)-1
+ switch {
+ case !lastStep:
+ next := now.Add(time.Duration(step.DelayMinutes) * time.Minute)
+ err = updateEscalationState(tx, page.Id, level, page.RepeatIteration, &next, now)
+ case page.RepeatIteration < definition.RepeatCount:
+ next := now.Add(time.Duration(step.DelayMinutes) * time.Minute)
+ err = updateEscalationState(tx, page.Id, -1, page.RepeatIteration+1, &next, now)
+ default:
+ // Exhausted: the page stays open until someone acknowledges or
+ // resolves it, but nothing further is sent.
+ err = updateEscalationState(tx, page.Id, level, page.RepeatIteration, nil, now)
+ }
+ return enqueued, err
+}
+
+// updateEscalationState is the claim path's terminal write: every branch of a
+// claim must go through the status-guarded update so a page acknowledged or
+// resolved mid-claim rolls the whole claim back via errClaimLost.
+func updateEscalationState(tx *sql.Tx, pageId int, level int, iteration int, nextEscalationAt *time.Time, now time.Time) error {
+ updated, err := transactional.PageRepository.UpdateEscalationState(tx, pageId, level, iteration, nextEscalationAt, now)
+ if err != nil {
+ return err
+ }
+ if !updated {
+ return errClaimLost
+ }
+ return nil
+}
+
+// resolveUserTarget maps a schedule/team/user target to concrete user ids.
+// Dangling targets resolve to nothing (reported, never fatal).
+func resolveUserTarget(tx *sql.Tx, organizationId int, target StepTarget, now time.Time) ([]int, error) {
+ switch target.Type {
+ case TargetSchedule:
+ userIds, err := CurrentOnCallForSchedule(tx, target.Id, now)
+ if err != nil {
+ return nil, err
+ }
+ if len(userIds) == 0 {
+ traceway.CaptureException(fmt.Errorf("escalation target schedule %d resolved to nobody on call", target.Id))
+ }
+ return userIds, nil
+ case TargetTeam:
+ team, err := transactional.TeamRepository.FindById(tx, target.Id)
+ if err != nil {
+ return nil, err
+ }
+ if team == nil || team.OrganizationId != organizationId {
+ traceway.CaptureException(fmt.Errorf("escalation target team %d no longer exists", target.Id))
+ return nil, nil
+ }
+ return transactional.TeamRepository.FindMemberUserIds(tx, team.Id)
+ case TargetUser:
+ role, err := transactional.OrganizationRepository.GetUserRole(tx, organizationId, target.Id)
+ if err != nil {
+ return nil, err
+ }
+ if role == "" {
+ traceway.CaptureException(fmt.Errorf("escalation target user %d is no longer an organization member", target.Id))
+ return nil, nil
+ }
+ return []int{target.Id}, nil
+ }
+ return nil, nil
+}
+
+// claimUserDeliveries runs the user's notification-rule chain for the page's
+// urgency: every step is enqueued at once as a scheduled outbox delivery
+// (NotBefore staggered by the step delay) under the page's cancel key, so
+// acknowledging cancels the tail with no separate scheduler. Users without a
+// usable chain fall back to every enabled+verified method immediately, or the
+// account email when none exist. Each delivery row mints its own ack token.
+func claimUserDeliveries(tx *sql.Tx, page *models.Page, level int, userId int, now time.Time) (int, error) {
+ user, err := transactional.UserRepository.FindById(tx, userId)
+ if err != nil {
+ return 0, err
+ }
+ if user == nil {
+ return 0, nil
+ }
+ methods, err := transactional.UserContactMethodRepository.FindEnabledByUser(tx, userId)
+ if err != nil {
+ return 0, err
+ }
+ // Dropping unsendable methods here (rather than at delivery time) keeps the
+ // "no methods left" account-email fallback below reachable: a user whose
+ // only method is an SMS row from when Twilio was still configured must
+ // still be paged.
+ methods = sendableContactMethods(methods)
+ methodById := make(map[int]*models.UserContactMethod, len(methods))
+ for _, method := range methods {
+ methodById[method.Id] = method
+ }
+
+ enqueued := 0
+ appendDelivery := func(methodType string, configJSON json.RawMessage, desc string, delay time.Duration) error {
+ token := newAckToken()
+ sendAt := now.Add(delay)
+ notification := &models.PageNotification{
+ PageId: page.Id,
+ Level: level,
+ Iteration: page.RepeatIteration,
+ UserId: &userId,
+ TargetDesc: truncateTargetDesc(desc),
+ MethodType: methodType,
+ Status: models.PageNotificationPending,
+ ScheduledFor: &sendAt,
+ AckTokenHash: shared.HashAuthToken(token),
+ CreatedAt: now,
+ }
+ notificationId, err := transactional.PageNotificationRepository.Create(tx, notification)
+ if err != nil {
+ return err
+ }
+ var notBefore *time.Time
+ if delay > 0 {
+ notBefore = &sendAt
+ }
+ if _, err := outbox.Enqueue(tx, outbox.Delivery{
+ Kind: models.OutboxKindPage,
+ AdapterType: methodType,
+ AdapterConfig: configJSON,
+ Message: buildPageMessage(page, level, ackURLFor(token)),
+ NotBefore: notBefore,
+ CancelKey: outbox.PageCancelKey(page.Id),
+ PageNotificationId: ¬ificationId,
+ }); err != nil {
+ return err
+ }
+ enqueued++
+ return nil
+ }
+
+ deliveryFor := func(method *models.UserContactMethod) (json.RawMessage, string, bool) {
+ switch method.MethodType {
+ case "email":
+ configJSON, desc := EmailDeliveryFor(user.Email, ParseEmailOverride(method.Config))
+ return configJSON, desc, true
+ case "sms":
+ return json.RawMessage(method.Config), smsDescFor(method.Config), true
+ case "slack", "pushover", "telegram":
+ return json.RawMessage(method.Config), user.Name + " (" + method.MethodType + ")", true
+ default:
+ traceway.CaptureException(fmt.Errorf("user %d has a contact method of unsupported type %q", userId, method.MethodType))
+ return nil, "", false
+ }
+ }
+
+ urgency := page.Urgency
+ if urgency == "" {
+ urgency = ResolveUrgency("", page.Severity)
+ }
+ rules, err := transactional.UserNotificationRuleRepository.FindByUserAndUrgency(tx, userId, urgency)
+ if err != nil {
+ return 0, err
+ }
+ type chainStep struct {
+ methodType string
+ configJSON json.RawMessage
+ desc string
+ delay time.Duration
+ }
+ var chain []chainStep
+ droppedBeforeFirstStep := false
+ for _, rule := range rules {
+ method, ok := methodById[rule.ContactMethodId]
+ if ok {
+ configJSON, desc, sendable := deliveryFor(method)
+ if sendable {
+ chain = append(chain, chainStep{
+ methodType: method.MethodType,
+ configJSON: configJSON,
+ desc: desc,
+ delay: time.Duration(rule.DelayMinutes) * time.Minute,
+ })
+ continue
+ }
+ }
+ if len(chain) == 0 {
+ droppedBeforeFirstStep = true
+ }
+ }
+ if len(chain) > 0 {
+ // Rebase so the earliest surviving step still pages immediately when
+ // the step carrying the chain's zero delay was dropped.
+ offset := time.Duration(0)
+ if droppedBeforeFirstStep {
+ offset = chain[0].delay
+ }
+ for _, step := range chain {
+ delay := step.delay - offset
+ desc := step.desc
+ if delay > 0 {
+ desc = fmt.Sprintf("%s, +%dm", desc, int(delay/time.Minute))
+ }
+ if err := appendDelivery(step.methodType, step.configJSON, desc, delay); err != nil {
+ return 0, err
+ }
+ }
+ return enqueued, nil
+ }
+ if len(rules) > 0 {
+ traceway.CaptureException(fmt.Errorf("user %d has %s-urgency notification rules but no usable steps, falling back", userId, urgency))
+ }
+
+ if len(methods) == 0 {
+ // Email is never silently missing: no configured methods means the
+ // account email is paged.
+ configJSON, desc := EmailDeliveryFor(user.Email, "")
+ if err := appendDelivery("email", configJSON, desc, 0); err != nil {
+ return 0, err
+ }
+ return enqueued, nil
+ }
+ for _, method := range methods {
+ configJSON, desc, ok := deliveryFor(method)
+ if !ok {
+ continue
+ }
+ if err := appendDelivery(method.MethodType, configJSON, desc, 0); err != nil {
+ return 0, err
+ }
+ }
+ return enqueued, nil
+}
+
+// sendableContactMethods drops methods this instance has no transport for.
+// Only SMS can lose its transport: Twilio credentials can be removed after the
+// method was created, and paging into a dead channel is worse than falling
+// back to the account email.
+func sendableContactMethods(methods []*models.UserContactMethod) []*models.UserContactMethod {
+ if config.Config.TwilioEnabled() {
+ return methods
+ }
+ sendable := make([]*models.UserContactMethod, 0, len(methods))
+ for _, method := range methods {
+ if method.MethodType == "sms" {
+ continue
+ }
+ sendable = append(sendable, method)
+ }
+ return sendable
+}
+
+// maxTargetDescLength matches page_notifications.target_desc (VARCHAR(300) on
+// Postgres).
+const maxTargetDescLength = 300
+
+// truncateTargetDesc clamps a delivery-log label to the column limit; an
+// oversized label would fail the insert and roll back the escalation claim.
+func truncateTargetDesc(desc string) string {
+ if len(desc) <= maxTargetDescLength {
+ return desc
+ }
+ trimmed := desc[:maxTargetDescLength]
+ for len(trimmed) > 0 && !utf8.ValidString(trimmed) {
+ trimmed = trimmed[:len(trimmed)-1]
+ }
+ return trimmed
+}
+
+// SMSPhoneNumber extracts the phoneNumber from an sms contact-method config,
+// or "" when missing or malformed.
+func SMSPhoneNumber(config []byte) string {
+ var parsed struct {
+ PhoneNumber string `json:"phoneNumber"`
+ }
+ _ = json.Unmarshal(config, &parsed)
+ return parsed.PhoneNumber
+}
+
+// smsDescFor labels the delivery row. GET /api/pages/:id returns these to
+// every reader of the project, so the number is masked to its last 4 digits:
+// the responder still recognises their own phone, nobody else learns it.
+func smsDescFor(config []byte) string {
+ number := SMSPhoneNumber(config)
+ if number == "" {
+ return "sms"
+ }
+ return notifications.MaskPhoneNumber(number) + " (sms)"
+}
+
+// ParseEmailOverride extracts the optional email override from an email
+// contact-method config; "" means the account email is used.
+func ParseEmailOverride(config []byte) string {
+ var parsed struct {
+ Email string `json:"email"`
+ }
+ _ = json.Unmarshal(config, &parsed)
+ return parsed.Email
+}
+
+// EmailDeliveryFor builds the email adapter config (and display description)
+// for a recipient, applying the override when present. Overrides are masked in
+// the delivery log; the account email is not.
+func EmailDeliveryFor(accountEmail string, override string) (json.RawMessage, string) {
+ email := accountEmail
+ desc := accountEmail
+ if override != "" {
+ email = override
+ desc = maskEmail(override)
+ }
+ configJSON, _ := json.Marshal(map[string]any{"recipients": []string{email}})
+ return configJSON, desc + " (email)"
+}
+
+// maskEmail keeps the first character of the local part and the whole domain.
+func maskEmail(email string) string {
+ at := strings.LastIndex(email, "@")
+ if at <= 0 {
+ return email
+ }
+ return email[:1] + "***" + email[at:]
+}
+
+// claimChannelDelivery records the delivery-log row for a plain-channel target
+// (method_type "channel" for display) and enqueues the outbox delivery with
+// the channel's real adapter type and a config snapshot. Channel messages
+// carry the dashboard link, never an ack token: an anyone-can-click token in a
+// shared room defeats attribution.
+func claimChannelDelivery(tx *sql.Tx, page *models.Page, level int, channelId int, now time.Time) (int, error) {
+ channel, err := transactional.NotificationChannelRepository.FindById(tx, channelId)
+ if err != nil {
+ return 0, err
+ }
+ if channel == nil || channel.ChannelType == "escalation" {
+ traceway.CaptureException(fmt.Errorf("escalation target channel %d no longer exists or is not deliverable", channelId))
+ return 0, nil
+ }
+ message := buildPageMessage(page, level, AckLink(page))
+ scheduledFor := now
+ notification := &models.PageNotification{
+ PageId: page.Id,
+ Level: level,
+ Iteration: page.RepeatIteration,
+ TargetDesc: "Channel: " + channel.Name,
+ MethodType: "channel",
+ Status: models.PageNotificationPending,
+ ScheduledFor: &scheduledFor,
+ CreatedAt: now,
+ }
+ notificationId, err := transactional.PageNotificationRepository.Create(tx, notification)
+ if err != nil {
+ return 0, err
+ }
+ if _, err := outbox.Enqueue(tx, outbox.Delivery{
+ Kind: models.OutboxKindPage,
+ AdapterType: channel.ChannelType,
+ AdapterConfig: json.RawMessage(channel.Config),
+ Message: message,
+ CancelKey: outbox.PageCancelKey(page.Id),
+ PageNotificationId: ¬ificationId,
+ }); err != nil {
+ return 0, err
+ }
+ return 1, nil
+}
+
+// buildPageMessage builds the delivery message. ackURL is per-delivery: user
+// deliveries carry their own tokenized link, channel deliveries the dashboard
+// link.
+func buildPageMessage(page *models.Page, level int, ackURL string) notifications.Message {
+ prefix := "[Page] "
+ if level > 0 {
+ prefix = fmt.Sprintf("[Page — escalation L%d] ", level+1)
+ }
+ body := page.Body
+ if body != "" {
+ body += "\n\n"
+ }
+ body += "Acknowledge this page: " + ackURL
+
+ severity := notifications.Severity(page.Severity)
+ if severity == "" {
+ severity = notifications.SeverityCritical
+ }
+ return notifications.Message{
+ Subject: prefix + page.Subject,
+ Body: body,
+ Severity: severity,
+ RuleType: page.RuleType,
+ RuleName: page.RuleName,
+ URL: ackURL,
+ }
+}
+
+// newAckToken mints an opaque per-delivery acknowledge token. Only its SHA-256
+// hash is stored; the plaintext exists solely inside the outgoing message.
+func newAckToken() string {
+ b := make([]byte, 32)
+ if _, err := rand.Read(b); err != nil {
+ panic(err)
+ }
+ return "twk_" + base64.RawURLEncoding.EncodeToString(b)
+}
+
+// appBaseURL is the dashboard origin for outgoing links. On-call deployments
+// should set APP_BASE_URL; the localhost fallback mirrors the email service.
+func appBaseURL() string {
+ base := config.Config.AppBaseURL
+ if base == "" {
+ base = "http://localhost:5173"
+ }
+ return strings.TrimRight(base, "/")
+}
+
+// ackURLFor builds the public no-login acknowledge link for a delivery token.
+func ackURLFor(token string) string {
+ return appBaseURL() + "/ack/" + token
+}
+
+// AckLink builds the absolute dashboard link that acknowledges a page. The
+// project id is included so the dashboard can switch to the page's project —
+// pages are resolved against the viewer's selected project.
+func AckLink(page *models.Page) string {
+ return appBaseURL() + "/on-call?page=" + strconv.Itoa(page.Id) + "&projectId=" + page.ProjectId.String()
+}
diff --git a/backend/app/oncall/escalator_test.go b/backend/app/oncall/escalator_test.go
new file mode 100644
index 00000000..b324e54b
--- /dev/null
+++ b/backend/app/oncall/escalator_test.go
@@ -0,0 +1,589 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package oncall
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "fmt"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/services"
+)
+
+type escalatorFixture struct {
+ OrgId int
+ ProjectId uuid.UUID
+ Alice int
+ Bob int
+}
+
+func setupEscalatorDB(t *testing.T) *escalatorFixture {
+ t.Helper()
+
+ dbtest.SetupSQLite(t)
+ services.InitEmail()
+ outbox.RegisterSender(notifications.AdapterSend)
+
+ fixture := &escalatorFixture{}
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ org, err := transactional.OrganizationRepository.Create(tx, "Acme", "UTC")
+ if err != nil {
+ return struct{}{}, err
+ }
+ fixture.OrgId = org.Id
+ alice, err := transactional.UserRepository.Create(tx, "alice@example.com", "Alice", "x")
+ if err != nil {
+ return struct{}{}, err
+ }
+ bob, err := transactional.UserRepository.Create(tx, "bob@example.com", "Bob", "x")
+ if err != nil {
+ return struct{}{}, err
+ }
+ fixture.Alice = alice.Id
+ fixture.Bob = bob.Id
+ for _, userId := range []int{alice.Id, bob.Id} {
+ if _, err := transactional.OrganizationRepository.AddUser(tx, org.Id, userId, "user"); err != nil {
+ return struct{}{}, err
+ }
+ }
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "api", "gin", org.Id)
+ if err != nil {
+ return struct{}{}, err
+ }
+ fixture.ProjectId = project.Id
+ return struct{}{}, nil
+ })
+ if err != nil {
+ t.Fatalf("seed fixture: %v", err)
+ }
+ return fixture
+}
+
+func createPolicy(t *testing.T, orgId int, definition string) int {
+ t.Helper()
+ id, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ now := time.Now().UTC()
+ return transactional.EscalationPolicyRepository.Create(tx, &models.EscalationPolicy{
+ OrganizationId: orgId,
+ Name: fmt.Sprintf("policy-%d", now.UnixNano()),
+ Definition: models.JSONText(definition),
+ CreatedAt: now,
+ UpdatedAt: now,
+ })
+ })
+ if err != nil {
+ t.Fatalf("create policy: %v", err)
+ }
+ return id
+}
+
+func openTestPageForPolicy(t *testing.T, fixture *escalatorFixture, policyId int, dedupKey string) *models.Page {
+ t.Helper()
+ if _, err := openPage(openPageParams{
+ PolicyId: policyId,
+ ProjectId: fixture.ProjectId,
+ RuleName: "Test rule",
+ RuleType: "new_error",
+ Subject: "Something broke",
+ Body: "It really broke",
+ URL: "/issues/abc",
+ Severity: "critical",
+ DedupKey: dedupKey,
+ }); err != nil {
+ t.Fatalf("open page: %v", err)
+ }
+ page := findPageByDedupKey(t, dedupKey)
+ if page == nil {
+ t.Fatal("expected a page to exist")
+ }
+ return page
+}
+
+func findPageByDedupKey(t *testing.T, dedupKey string) *models.Page {
+ t.Helper()
+ page, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.Page, error) {
+ return transactional.PageRepository.FindUnresolvedByDedupKey(tx, dedupKey)
+ })
+ if err != nil {
+ t.Fatalf("find page: %v", err)
+ }
+ return page
+}
+
+func reloadPage(t *testing.T, id int) *models.Page {
+ t.Helper()
+ page, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.Page, error) {
+ return transactional.PageRepository.FindById(tx, id)
+ })
+ if err != nil || page == nil {
+ t.Fatalf("reload page %d: %v", id, err)
+ }
+ return page
+}
+
+func pageNotifications(t *testing.T, pageId int) []*models.PageNotification {
+ t.Helper()
+ rows, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.PageNotification, error) {
+ return transactional.PageNotificationRepository.FindByPage(tx, pageId)
+ })
+ if err != nil {
+ t.Fatalf("load notifications: %v", err)
+ }
+ return rows
+}
+
+// tickAndDrain runs one escalator tick followed by one outbox drain tick, so
+// enqueued deliveries actually send (via the log-only email adapter). The
+// drain runs slightly ahead of the tick time because Enqueue stamps rows with
+// its own wall clock; in production Wake() drains with a fresh timestamp.
+func tickAndDrain(t *testing.T, now time.Time) {
+ t.Helper()
+ runEscalatorTick(context.Background(), now)
+ drainAt := now.Add(2 * time.Second)
+ if realNow := time.Now().UTC().Add(2 * time.Second); realNow.After(drainAt) {
+ drainAt = realNow
+ }
+ outbox.DrainOnce(context.Background(), drainAt)
+}
+
+func twoStepPolicy(alice, bob int) string {
+ return fmt.Sprintf(`{"schemaVersion":1,"steps":[{"targets":[{"type":"user","id":%d}],"delayMinutes":5},{"targets":[{"type":"user","id":%d}],"delayMinutes":5}],"repeatCount":0}`, alice, bob)
+}
+
+func TestEscalatorNotifiesFirstLevelImmediately(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "rule1|/issues/abc")
+
+ now := time.Now().UTC()
+ tickAndDrain(t, now)
+
+ page = reloadPage(t, page.Id)
+ if page.EscalationLevel != 0 {
+ t.Errorf("escalation level = %d, want 0", page.EscalationLevel)
+ }
+ if page.NextEscalationAt == nil {
+ t.Fatal("expected a next escalation time")
+ }
+ if diff := page.NextEscalationAt.Sub(now); diff < 4*time.Minute || diff > 6*time.Minute {
+ t.Errorf("next escalation in %v, want ~5m", diff)
+ }
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected 1 notification, got %d", len(rows))
+ }
+ if rows[0].Status != models.PageNotificationSent {
+ t.Errorf("notification status = %s, want sent (error: %s)", rows[0].Status, rows[0].ErrorMsg)
+ }
+ if rows[0].UserId == nil || *rows[0].UserId != fixture.Alice {
+ t.Errorf("notified user = %v, want alice (%d)", rows[0].UserId, fixture.Alice)
+ }
+ if rows[0].MethodType != "email" {
+ t.Errorf("fallback method = %s, want email", rows[0].MethodType)
+ }
+}
+
+func TestEscalatorAdvancesLevelsAndExhausts(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "rule2|/issues/abc")
+
+ now := time.Now().UTC()
+ tickAndDrain(t, now)
+
+ // Not due yet: nothing should change.
+ tickAndDrain(t, now.Add(2*time.Minute))
+ if rows := pageNotifications(t, page.Id); len(rows) != 1 {
+ t.Fatalf("expected still 1 notification before the delay elapses, got %d", len(rows))
+ }
+
+ // Due: escalate to L2 (bob).
+ tickAndDrain(t, now.Add(6*time.Minute))
+ page = reloadPage(t, page.Id)
+ if page.EscalationLevel != 1 {
+ t.Errorf("escalation level = %d, want 1", page.EscalationLevel)
+ }
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 2 {
+ t.Fatalf("expected 2 notifications, got %d", len(rows))
+ }
+ if rows[1].UserId == nil || *rows[1].UserId != fixture.Bob {
+ t.Errorf("second notification user = %v, want bob (%d)", rows[1].UserId, fixture.Bob)
+ }
+
+ // Last step with repeatCount 0: exhausted right away, stays open,
+ // nothing further on later ticks.
+ if page.NextEscalationAt != nil {
+ t.Errorf("expected escalation exhausted (next = nil), got %v", page.NextEscalationAt)
+ }
+ tickAndDrain(t, now.Add(12*time.Minute))
+ page = reloadPage(t, page.Id)
+ if page.Status != models.PageStatusOpen {
+ t.Errorf("page status = %s, want open after exhaustion", page.Status)
+ }
+ if rows := pageNotifications(t, page.Id); len(rows) != 2 {
+ t.Errorf("expected no further notifications after exhaustion, got %d", len(rows))
+ }
+}
+
+func TestEscalatorRepeatCyclesThenStops(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ definition := fmt.Sprintf(`{"schemaVersion":1,"steps":[{"targets":[{"type":"user","id":%d}],"delayMinutes":5}],"repeatCount":1}`, fixture.Alice)
+ policyId := createPolicy(t, fixture.OrgId, definition)
+ page := openTestPageForPolicy(t, fixture, policyId, "rule3|/issues/abc")
+
+ now := time.Now().UTC()
+ tickAndDrain(t, now) // iteration 0, level 0
+ tickAndDrain(t, now.Add(6*time.Minute)) // repeat: iteration 1, level 0
+ tickAndDrain(t, now.Add(12*time.Minute)) // exhausted
+
+ page = reloadPage(t, page.Id)
+ if page.RepeatIteration != 1 {
+ t.Errorf("repeat iteration = %d, want 1", page.RepeatIteration)
+ }
+ if page.NextEscalationAt != nil {
+ t.Errorf("expected exhaustion, next escalation = %v", page.NextEscalationAt)
+ }
+ if rows := pageNotifications(t, page.Id); len(rows) != 2 {
+ t.Errorf("expected 2 notifications (initial + one repeat), got %d", len(rows))
+ }
+}
+
+func TestAcknowledgeStopsEscalation(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "rule4|/issues/abc")
+
+ now := time.Now().UTC()
+ tickAndDrain(t, now)
+
+ acknowledged, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return AcknowledgePage(tx, page.Id, &fixture.Alice, AckViaDashboard, now)
+ })
+ if err != nil || !acknowledged {
+ t.Fatalf("acknowledge failed: %v (ok=%v)", err, acknowledged)
+ }
+
+ tickAndDrain(t, now.Add(10*time.Minute))
+ if rows := pageNotifications(t, page.Id); len(rows) != 1 {
+ t.Errorf("expected no escalation after ack, got %d notifications", len(rows))
+ }
+
+ // Second ack loses the guarded update.
+ acknowledged, err = db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return AcknowledgePage(tx, page.Id, &fixture.Bob, AckViaDashboard, now)
+ })
+ if err != nil {
+ t.Fatalf("second acknowledge errored: %v", err)
+ }
+ if acknowledged {
+ t.Error("second acknowledge should report no rows affected")
+ }
+}
+
+func TestAcknowledgeCancelsQueuedDeliveries(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "rule9|/issues/abc")
+
+ // Tick WITHOUT draining: the delivery sits queued in the outbox.
+ now := time.Now().UTC()
+ runEscalatorTick(context.Background(), now)
+
+ acknowledged, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return AcknowledgePage(tx, page.Id, &fixture.Alice, AckViaDashboard, now)
+ })
+ if err != nil || !acknowledged {
+ t.Fatalf("acknowledge failed: %v (ok=%v)", err, acknowledged)
+ }
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 || rows[0].Status != models.PageNotificationCancelled {
+ t.Fatalf("expected the queued delivery-log row cancelled, got %+v", rows)
+ }
+
+ // A later drain must not deliver the cancelled row.
+ outbox.DrainOnce(context.Background(), time.Now().UTC().Add(time.Minute))
+ if rows := pageNotifications(t, page.Id); rows[0].Status != models.PageNotificationCancelled {
+ t.Errorf("cancelled delivery resurrected to %s", rows[0].Status)
+ }
+}
+
+func TestClaimRollsBackWhenPageAcknowledgedMidClaim(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "race1|/issues/abc")
+
+ now := time.Now().UTC()
+ acknowledged, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return AcknowledgePage(tx, page.Id, &fixture.Alice, AckViaDashboard, now)
+ })
+ if err != nil || !acknowledged {
+ t.Fatalf("acknowledge: %v (%v)", err, acknowledged)
+ }
+
+ // Simulate the race: a claim that read the page while it was still open
+ // (the stale pre-ack row) commits after the ack. The guarded terminal
+ // update must lose and roll the whole claim back, so none of its
+ // deliveries — which the ack's CancelByKey never saw — can commit.
+ _, err = db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return claimPageEscalation(tx, page, now)
+ })
+ if !errors.Is(err, errClaimLost) {
+ t.Fatalf("expected errClaimLost, got %v", err)
+ }
+ if rows := pageNotifications(t, page.Id); len(rows) != 0 {
+ t.Errorf("claim leaked %d page_notifications rows past the ack", len(rows))
+ }
+ if rows := outboxRowsForPage(t, page.Id); len(rows) != 0 {
+ t.Errorf("claim leaked %d outbox rows past the ack", len(rows))
+ }
+}
+
+func TestPageDedupBumpsAndReleasesOnResolve(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "rule5|/issues/abc")
+
+ now := time.Now().UTC()
+ tickAndDrain(t, now)
+ levelBefore := reloadPage(t, page.Id).EscalationLevel
+
+ // Refire while unresolved: bump only, clock untouched, no re-notify.
+ openTestPageForPolicy(t, fixture, policyId, "rule5|/issues/abc")
+ bumped := reloadPage(t, page.Id)
+ if bumped.EventCount != 2 {
+ t.Errorf("event count = %d, want 2", bumped.EventCount)
+ }
+ if bumped.EscalationLevel != levelBefore {
+ t.Errorf("escalation level changed on refire: %d -> %d", levelBefore, bumped.EscalationLevel)
+ }
+
+ resolved, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return transactional.PageRepository.Resolve(tx, page.Id, fixture.Alice, now)
+ })
+ if err != nil || !resolved {
+ t.Fatalf("resolve failed: %v (ok=%v)", err, resolved)
+ }
+
+ // Same dedup key now opens a fresh page.
+ fresh := openTestPageForPolicy(t, fixture, policyId, "rule5|/issues/abc")
+ if fresh.Id == page.Id {
+ t.Error("expected a fresh page after resolve")
+ }
+ if fresh.EventCount != 1 {
+ t.Errorf("fresh page event count = %d, want 1", fresh.EventCount)
+ }
+}
+
+// A realistic incident storm: one noisy rule firing 200 times while ticks run
+// far more often than the escalation delays. The delivery count must be driven
+// by the policy (2 steps, 1 repeat = 4 notifications), never by how many events
+// arrived or how often the worker polled.
+func TestIncidentStormStaysBoundedByThePolicy(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ definition := fmt.Sprintf(
+ `{"schemaVersion":1,"steps":[{"targets":[{"type":"user","id":%d}],"delayMinutes":5},{"targets":[{"type":"user","id":%d}],"delayMinutes":5}],"repeatCount":1}`,
+ fixture.Alice, fixture.Bob,
+ )
+ policyId := createPolicy(t, fixture.OrgId, definition)
+ page := openTestPageForPolicy(t, fixture, policyId, "storm|/issues/abc")
+
+ start := time.Now().UTC()
+ // 30 minutes of 30-second ticks, with the rule refiring on every tick.
+ for minute := 0; minute < 30; minute++ {
+ for half := 0; half < 2; half++ {
+ openTestPageForPolicy(t, fixture, policyId, "storm|/issues/abc")
+ tickAndDrain(t, start.Add(time.Duration(minute)*time.Minute+time.Duration(half*30)*time.Second))
+ }
+ }
+
+ stormed := reloadPage(t, page.Id)
+ if stormed.EventCount != 61 {
+ t.Errorf("event count = %d, want 61 (1 open + 60 refires)", stormed.EventCount)
+ }
+ rows := pageNotifications(t, page.Id)
+ // L1 alice, L2 bob, then repeat iteration 1: L1 alice, L2 bob.
+ if len(rows) != 4 {
+ t.Fatalf("60 refires and 60 ticks produced %d notifications, want exactly 4", len(rows))
+ }
+ for _, row := range rows {
+ if row.Status != models.PageNotificationSent {
+ t.Errorf("notification %d status = %s, want sent (%s)", row.Id, row.Status, row.ErrorMsg)
+ }
+ }
+ if stormed.NextEscalationAt != nil {
+ t.Errorf("escalation should be exhausted, next = %v", stormed.NextEscalationAt)
+ }
+}
+
+// An override exists to relieve someone. If the escalator paged every user the
+// schedule stack covers, the person who handed off their shift would be paged
+// alongside the person covering it.
+func TestOverridePagesOnlyTheCoveringUser(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ now := time.Now().UTC()
+ definition := fmt.Sprintf(
+ `{"schemaVersion":1,"layers":[{"id":"l1","name":"Base","rotationType":"daily","handoffTime":"09:00","rotationStart":"2020-01-01","userIds":[%d]}]}`,
+ fixture.Alice,
+ )
+ scheduleId, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ teamId, err := transactional.TeamRepository.Create(tx, &models.Team{
+ OrganizationId: fixture.OrgId, Name: "override-team", CreatedAt: now, UpdatedAt: now,
+ })
+ if err != nil {
+ return 0, err
+ }
+ id, err := transactional.OncallScheduleRepository.Create(tx, &models.OncallSchedule{
+ OrganizationId: fixture.OrgId, TeamId: teamId, Name: "override-sched", Timezone: "UTC",
+ Definition: models.JSONText(definition), CreatedAt: now, UpdatedAt: now,
+ })
+ if err != nil {
+ return 0, err
+ }
+ _, err = transactional.OncallOverrideRepository.Create(tx, &models.OncallOverride{
+ ScheduleId: id, UserId: fixture.Bob,
+ StartAt: now.Add(-time.Hour), EndAt: now.Add(time.Hour), CreatedAt: now,
+ })
+ return id, err
+ })
+ if err != nil {
+ t.Fatalf("seed schedule: %v", err)
+ }
+
+ policyDefinition := fmt.Sprintf(`{"schemaVersion":1,"steps":[{"targets":[{"type":"schedule","id":%d}],"delayMinutes":5}],"repeatCount":0}`, scheduleId)
+ policyId := createPolicy(t, fixture.OrgId, policyDefinition)
+ page := openTestPageForPolicy(t, fixture, policyId, "override|/issues/abc")
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected only the covering user paged, got %d deliveries: %+v", len(rows), rows)
+ }
+ if rows[0].UserId == nil || *rows[0].UserId != fixture.Bob {
+ t.Errorf("paged user = %v, want bob (%d) who holds the override", rows[0].UserId, fixture.Bob)
+ }
+}
+
+func TestEscalatorSkipsDanglingTargets(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ definition := fmt.Sprintf(`{"schemaVersion":1,"steps":[{"targets":[{"type":"schedule","id":9999},{"type":"user","id":%d}],"delayMinutes":5}],"repeatCount":0}`, fixture.Alice)
+ policyId := createPolicy(t, fixture.OrgId, definition)
+ page := openTestPageForPolicy(t, fixture, policyId, "rule6|/issues/abc")
+
+ tickAndDrain(t, time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected the dangling schedule to be skipped and alice notified, got %d rows", len(rows))
+ }
+ if rows[0].UserId == nil || *rows[0].UserId != fixture.Alice {
+ t.Errorf("notified user = %v, want alice", rows[0].UserId)
+ }
+}
+
+func TestEscalatorUsesConfiguredContactMethods(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return transactional.UserContactMethodRepository.Create(tx, &models.UserContactMethod{
+ UserId: fixture.Alice,
+ MethodType: "email",
+ Config: models.JSONText(`{"email":"pager@example.com"}`),
+ Enabled: true,
+ Verified: true,
+ CreatedAt: time.Now().UTC(),
+ })
+ })
+ if err != nil {
+ t.Fatalf("create contact method: %v", err)
+ }
+
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ page := openTestPageForPolicy(t, fixture, policyId, "rule7|/issues/abc")
+ tickAndDrain(t, time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected 1 notification, got %d", len(rows))
+ }
+ if rows[0].TargetDesc != "p***@example.com (email)" {
+ t.Errorf("target desc = %q, want the masked override email", rows[0].TargetDesc)
+ }
+}
+
+func TestPolicyValidation(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ cases := []struct {
+ name string
+ json string
+ }{
+ {"no steps", `{"steps":[],"repeatCount":0}`},
+ {"no targets", `{"steps":[{"targets":[],"delayMinutes":5}]}`},
+ {"delay too small", fmt.Sprintf(`{"steps":[{"targets":[{"type":"user","id":%d}],"delayMinutes":0}]}`, fixture.Alice)},
+ {"bad repeat", fmt.Sprintf(`{"steps":[{"targets":[{"type":"user","id":%d}],"delayMinutes":5}],"repeatCount":99}`, fixture.Alice)},
+ {"unknown target type", `{"steps":[{"targets":[{"type":"pigeon","id":1}],"delayMinutes":5}]}`},
+ {"nonexistent user", `{"steps":[{"targets":[{"type":"user","id":424242}],"delayMinutes":5}]}`},
+ {"nonexistent schedule", `{"steps":[{"targets":[{"type":"schedule","id":424242}],"delayMinutes":5}]}`},
+ }
+ for _, tc := range cases {
+ if _, err := ValidatePolicyDefinition(tx, fixture.OrgId, []byte(tc.json)); err == nil {
+ t.Errorf("expected validation error for %s", tc.name)
+ }
+ }
+ valid := fmt.Sprintf(`{"steps":[{"targets":[{"type":"user","id":%d}],"delayMinutes":15}],"repeatCount":1}`, fixture.Alice)
+ if _, err := ValidatePolicyDefinition(tx, fixture.OrgId, []byte(valid)); err != nil {
+ t.Errorf("expected valid policy to pass, got %v", err)
+ }
+ return struct{}{}, nil
+ })
+ if err != nil {
+ t.Fatalf("tx: %v", err)
+ }
+}
+
+func TestOpenPageRequiresMatchingOrg(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ otherOrgId := 0
+ otherProject := uuid.Nil
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ org, err := transactional.OrganizationRepository.Create(tx, "Other", "UTC")
+ if err != nil {
+ return struct{}{}, err
+ }
+ otherOrgId = org.Id
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "other-api", "gin", org.Id)
+ if err != nil {
+ return struct{}{}, err
+ }
+ otherProject = project.Id
+ return struct{}{}, nil
+ })
+ if err != nil {
+ t.Fatalf("seed other org: %v", err)
+ }
+ _ = otherOrgId
+
+ policyId := createPolicy(t, fixture.OrgId, twoStepPolicy(fixture.Alice, fixture.Bob))
+ _, err = openPage(openPageParams{
+ PolicyId: policyId,
+ ProjectId: otherProject,
+ RuleName: "r",
+ RuleType: "new_error",
+ Subject: "s",
+ DedupKey: "cross-org",
+ })
+ if err == nil {
+ t.Error("expected cross-org page open to fail")
+ }
+}
diff --git a/backend/app/oncall/pager.go b/backend/app/oncall/pager.go
new file mode 100644
index 00000000..954d2402
--- /dev/null
+++ b/backend/app/oncall/pager.go
@@ -0,0 +1,195 @@
+package oncall
+
+import (
+ "crypto/sha256"
+ "database/sql"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+// EscalationChannelPolicyId extracts the policyId from an escalation
+// channel's config; 0 means the config is missing or malformed.
+func EscalationChannelPolicyId(configJSON json.RawMessage) int {
+ var cfg struct {
+ PolicyId int `json:"policyId"`
+ }
+ if err := json.Unmarshal(configJSON, &cfg); err != nil {
+ return 0
+ }
+ return cfg.PolicyId
+}
+
+// OpenPageFromDispatch is registered as the notifications package's page
+// opener (see notifications.RegisterPageOpener). It runs inside dispatch for
+// rules targeting an escalation channel: instead of sending anything, it opens
+// (or dedups into) a page; the escalator worker does all notifying.
+func OpenPageFromDispatch(configJSON json.RawMessage, rule *models.NotificationRuleWithChannel, msg notifications.Message) (bool, error) {
+ policyId := EscalationChannelPolicyId(configJSON)
+ if policyId == 0 {
+ return false, errors.New("escalation channel config has no policyId")
+ }
+
+ ruleId := rule.Id
+ opened, err := openPage(openPageParams{
+ PolicyId: policyId,
+ ProjectId: rule.ProjectId,
+ RuleId: &ruleId,
+ RuleName: rule.Name,
+ RuleType: rule.RuleType,
+ Subject: msg.Subject,
+ Body: msg.Body,
+ URL: msg.URL,
+ Severity: string(msg.Severity),
+ DedupKey: pageDedupKey(fmt.Sprintf("%d", rule.Id), msg.DedupToken),
+ })
+ if err != nil {
+ return false, err
+ }
+ if opened {
+ Wake()
+ }
+ return opened, nil
+}
+
+// OpenTestPage opens a real page for the channel test endpoint, exercising the
+// full escalation loop. Reports false when the fire deduped into a previous
+// test page that is still unresolved.
+func OpenTestPage(policyId int, projectId uuid.UUID, channelId int, channelName string) (bool, error) {
+ opened, err := openPage(openPageParams{
+ PolicyId: policyId,
+ ProjectId: projectId,
+ RuleName: "Channel test",
+ RuleType: "test",
+ Subject: fmt.Sprintf("Test page from channel %q", channelName),
+ Body: "This is a test page sent from the escalation channel test button. Acknowledge or resolve it from the On-Call page.",
+ Severity: string(notifications.SeverityInfo),
+ DedupKey: fmt.Sprintf("test|channel:%d", channelId),
+ })
+ if err != nil {
+ return false, err
+ }
+ if opened {
+ Wake()
+ }
+ return opened, nil
+}
+
+const maxDedupKeyLength = 300
+
+// pageDedupKey builds "prefix|token". pages.dedup_key is VARCHAR(300) on
+// Postgres while the token can be unbounded client input (endpoint names,
+// metric names), so an oversized token is replaced with a deterministic
+// digest: the same long token still dedups into the same page instead of
+// failing the insert.
+func pageDedupKey(prefix string, token string) string {
+ key := prefix + "|" + token
+ if len(key) <= maxDedupKeyLength {
+ return key
+ }
+ sum := sha256.Sum256([]byte(token))
+ return prefix + "|" + hex.EncodeToString(sum[:])[:32]
+}
+
+type openPageParams struct {
+ PolicyId int
+ ProjectId uuid.UUID
+ RuleId *int
+ RuleName string
+ RuleType string
+ Subject string
+ Body string
+ URL string
+ Severity string
+ DedupKey string
+}
+
+// openPage creates a page or bumps the unresolved page holding the dedup key.
+// Returns true when a new page was created.
+func openPage(params openPageParams) (bool, error) {
+ opened, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return openPageInTx(tx, params)
+ })
+ if err != nil && db.IsUniqueViolation(err) {
+ // Lost a race with a concurrent fire holding the same dedup key: the
+ // other side created the page, so this fire is just an extra event.
+ _, bumpErr := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return openPageInTx(tx, params)
+ })
+ return false, bumpErr
+ }
+ return opened, err
+}
+
+func openPageInTx(tx *sql.Tx, params openPageParams) (bool, error) {
+ now := time.Now().UTC()
+
+ existing, err := transactional.PageRepository.FindUnresolvedByDedupKey(tx, params.DedupKey)
+ if err != nil {
+ return false, err
+ }
+ if existing != nil {
+ // Never resets the escalation clock and never re-notifies.
+ return false, transactional.PageRepository.BumpEvent(tx, existing.Id, now)
+ }
+
+ policy, err := transactional.EscalationPolicyRepository.FindById(tx, params.PolicyId)
+ if err != nil {
+ return false, err
+ }
+ if policy == nil {
+ return false, fmt.Errorf("escalation policy %d not found", params.PolicyId)
+ }
+ project, err := transactional.ProjectRepository.FindById(tx, params.ProjectId)
+ if err != nil {
+ return false, err
+ }
+ if project == nil || project.OrganizationId == nil || *project.OrganizationId != policy.OrganizationId {
+ return false, fmt.Errorf("escalation policy %d does not belong to the project's organization", params.PolicyId)
+ }
+
+ // Urgency is resolved once at page open and remembered: a low-severity
+ // duplicate bump must never flip an in-flight high-urgency page.
+ policyUrgency := ""
+ if parsedDefinition, defErr := ParsePolicyDefinition(policy.Definition); defErr == nil {
+ policyUrgency = parsedDefinition.Urgency
+ }
+
+ policyId := policy.Id
+ nextEscalationAt := now
+ page := &models.Page{
+ OrganizationId: policy.OrganizationId,
+ ProjectId: params.ProjectId,
+ PolicyId: &policyId,
+ PolicySnapshot: policy.Definition,
+ Urgency: ResolveUrgency(policyUrgency, params.Severity),
+ RuleId: params.RuleId,
+ RuleName: params.RuleName,
+ RuleType: params.RuleType,
+ Subject: params.Subject,
+ Body: params.Body,
+ URL: params.URL,
+ Severity: params.Severity,
+ Status: models.PageStatusOpen,
+ DedupKey: params.DedupKey,
+ EventCount: 1,
+ LastEventAt: now,
+ EscalationLevel: -1,
+ RepeatIteration: 0,
+ NextEscalationAt: &nextEscalationAt,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+ if _, err := transactional.PageRepository.Create(tx, page); err != nil {
+ return false, err
+ }
+ return true, nil
+}
diff --git a/backend/app/oncall/policy.go b/backend/app/oncall/policy.go
new file mode 100644
index 00000000..b06ff20f
--- /dev/null
+++ b/backend/app/oncall/policy.go
@@ -0,0 +1,216 @@
+package oncall
+
+import (
+ "database/sql"
+ "encoding/json"
+ "errors"
+ "fmt"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+const (
+ MaxPolicySteps = 10
+ MaxTargetsPerStep = 10
+ MinStepDelayMinutes = 1
+ MaxStepDelayMinutes = 1440
+ MaxPolicyRepeatCount = 5
+)
+
+const (
+ TargetSchedule = "schedule"
+ TargetUser = "user"
+ TargetTeam = "team"
+ TargetChannel = "channel"
+)
+
+const UrgencyAuto = "auto"
+
+type PolicyDefinition struct {
+ SchemaVersion int `json:"schemaVersion"`
+ Steps []PolicyStep `json:"steps"`
+ RepeatCount int `json:"repeatCount"`
+ // Urgency picks which of each responder's notification-rule chains runs:
+ // ""|"auto" derives from severity, "high"/"low" force it. Optional so
+ // stored snapshots and old policies keep parsing (schemaVersion stays 1).
+ Urgency string `json:"urgency,omitempty"`
+}
+
+// ResolveUrgency maps a policy's urgency setting and a message severity to the
+// page urgency. Auto: critical pages are high urgency, everything else low.
+// An empty severity counts as critical, mirroring buildPageMessage.
+func ResolveUrgency(policyUrgency string, severity string) string {
+ switch policyUrgency {
+ case models.UrgencyHigh:
+ return models.UrgencyHigh
+ case models.UrgencyLow:
+ return models.UrgencyLow
+ }
+ if severity == "" || severity == string(models.NotificationSeverityCritical) {
+ return models.UrgencyHigh
+ }
+ return models.UrgencyLow
+}
+
+type PolicyStep struct {
+ Targets []StepTarget `json:"targets"`
+ DelayMinutes int `json:"delayMinutes"`
+}
+
+type StepTarget struct {
+ Type string `json:"type"`
+ Id int `json:"id"`
+}
+
+// ParsePolicyDefinition parses without target-existence checks; used for
+// stored snapshots where dangling targets are tolerated at run time.
+func ParsePolicyDefinition(raw []byte) (*PolicyDefinition, error) {
+ def := &PolicyDefinition{}
+ if len(raw) > 0 {
+ if err := json.Unmarshal(raw, def); err != nil {
+ return nil, errors.New("The escalation policy definition is not valid JSON.")
+ }
+ }
+ if def.SchemaVersion != 0 && def.SchemaVersion != SchemaVersion {
+ return nil, fmt.Errorf("Unsupported escalation policy schemaVersion %d, this server supports schemaVersion %d.", def.SchemaVersion, SchemaVersion)
+ }
+ def.SchemaVersion = SchemaVersion
+ return def, nil
+}
+
+// ValidatePolicyDefinition parses and fully validates a policy definition for
+// the organization, including target existence. Errors are user-facing 422
+// messages.
+func ValidatePolicyDefinition(tx *sql.Tx, organizationId int, raw []byte) (*PolicyDefinition, error) {
+ def, err := ParsePolicyDefinition(raw)
+ if err != nil {
+ return nil, err
+ }
+ if len(def.Steps) == 0 {
+ return nil, errors.New("An escalation policy needs at least one step.")
+ }
+ if len(def.Steps) > MaxPolicySteps {
+ return nil, fmt.Errorf("An escalation policy can have at most %d steps.", MaxPolicySteps)
+ }
+ if def.RepeatCount < 0 || def.RepeatCount > MaxPolicyRepeatCount {
+ return nil, fmt.Errorf("The repeat count must be between 0 and %d.", MaxPolicyRepeatCount)
+ }
+ switch def.Urgency {
+ case "", UrgencyAuto, models.UrgencyHigh, models.UrgencyLow:
+ default:
+ return nil, errors.New("The urgency must be auto, high, or low.")
+ }
+ for i, step := range def.Steps {
+ if len(step.Targets) == 0 {
+ return nil, fmt.Errorf("Step %d needs at least one target.", i+1)
+ }
+ if len(step.Targets) > MaxTargetsPerStep {
+ return nil, fmt.Errorf("Step %d can have at most %d targets.", i+1, MaxTargetsPerStep)
+ }
+ if step.DelayMinutes < MinStepDelayMinutes || step.DelayMinutes > MaxStepDelayMinutes {
+ return nil, fmt.Errorf("Step %d needs a delay between %d and %d minutes.", i+1, MinStepDelayMinutes, MaxStepDelayMinutes)
+ }
+ for _, target := range step.Targets {
+ if err := validateTarget(tx, organizationId, i, target); err != nil {
+ return nil, err
+ }
+ }
+ }
+ return def, nil
+}
+
+func validateTarget(tx *sql.Tx, organizationId int, stepIndex int, target StepTarget) error {
+ switch target.Type {
+ case TargetSchedule:
+ schedule, err := transactional.OncallScheduleRepository.FindById(tx, target.Id)
+ if err != nil {
+ return err
+ }
+ if schedule == nil || schedule.OrganizationId != organizationId {
+ return fmt.Errorf("Step %d references a schedule that does not exist in this organization.", stepIndex+1)
+ }
+ case TargetTeam:
+ team, err := transactional.TeamRepository.FindById(tx, target.Id)
+ if err != nil {
+ return err
+ }
+ if team == nil || team.OrganizationId != organizationId {
+ return fmt.Errorf("Step %d references a team that does not exist in this organization.", stepIndex+1)
+ }
+ case TargetUser:
+ role, err := transactional.OrganizationRepository.GetUserRole(tx, organizationId, target.Id)
+ if err != nil {
+ return err
+ }
+ if role == "" {
+ return fmt.Errorf("Step %d references a user who is not a member of this organization.", stepIndex+1)
+ }
+ case TargetChannel:
+ channel, err := transactional.NotificationChannelRepository.FindById(tx, target.Id)
+ if err != nil {
+ return err
+ }
+ if channel == nil {
+ return fmt.Errorf("Step %d references a notification channel that does not exist.", stepIndex+1)
+ }
+ if channel.ChannelType == "escalation" {
+ return fmt.Errorf("Step %d cannot target an escalation channel.", stepIndex+1)
+ }
+ project, err := transactional.ProjectRepository.FindById(tx, channel.ProjectId)
+ if err != nil {
+ return err
+ }
+ if project == nil || project.OrganizationId == nil || *project.OrganizationId != organizationId {
+ return fmt.Errorf("Step %d references a notification channel outside this organization.", stepIndex+1)
+ }
+ default:
+ return fmt.Errorf("Step %d has an unknown target type %q.", stepIndex+1, target.Type)
+ }
+ return nil
+}
+
+// PoliciesReferencing returns the names of the organization's escalation
+// policies that target any of the given ids of a type; it backs the delete
+// guards on schedules and teams.
+func PoliciesReferencing(tx *sql.Tx, organizationId int, targetType string, targetIds ...int) ([]string, error) {
+ if len(targetIds) == 0 {
+ return nil, nil
+ }
+ wanted := make(map[int]bool, len(targetIds))
+ for _, id := range targetIds {
+ wanted[id] = true
+ }
+ policies, err := transactional.EscalationPolicyRepository.FindByOrganization(tx, organizationId)
+ if err != nil {
+ return nil, err
+ }
+ var names []string
+ for _, policy := range policies {
+ def, err := ParsePolicyDefinition(policy.Definition)
+ if err != nil {
+ // Fail closed: an unparseable definition cannot be checked.
+ names = append(names, policy.Name)
+ continue
+ }
+ if policyTargets(def, targetType, wanted) {
+ names = append(names, policy.Name)
+ }
+ }
+ return names, nil
+}
+
+func policyTargets(def *PolicyDefinition, targetType string, wanted map[int]bool) bool {
+ for _, step := range def.Steps {
+ for _, target := range step.Targets {
+ if target.Type == targetType && wanted[target.Id] {
+ return true
+ }
+ }
+ }
+ return false
+}
+
+func MarshalPolicyDefinition(def *PolicyDefinition) ([]byte, error) {
+ return json.Marshal(def)
+}
diff --git a/backend/app/oncall/resolve.go b/backend/app/oncall/resolve.go
new file mode 100644
index 00000000..606686ce
--- /dev/null
+++ b/backend/app/oncall/resolve.go
@@ -0,0 +1,238 @@
+package oncall
+
+import (
+ "sort"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+// Shift is one contiguous stretch of a single person being on call. Times are UTC.
+type Shift struct {
+ UserId int `json:"userId"`
+ LayerId string `json:"layerId"`
+ Start time.Time `json:"start"`
+ End time.Time `json:"end"`
+ IsOverride bool `json:"isOverride"`
+}
+
+type span struct {
+ start time.Time
+ end time.Time
+}
+
+// ResolveLayerRange renders one layer's coverage for [from, to) without
+// stacking or overrides: rotation periods intersected with the layer's
+// restriction windows.
+func ResolveLayerRange(layer *models.OncallLayer, tz *time.Location, from, to time.Time) []Shift {
+ if len(layer.UserIds) == 0 || !from.Before(to) {
+ return nil
+ }
+ shifts := layerPeriods(layer, tz, from, to)
+ if len(layer.Restrictions) > 0 {
+ windows := expandRestrictions(layer.Restrictions, tz, from, to)
+ shifts = intersectShifts(shifts, windows)
+ }
+ shifts = clampShifts(shifts, from, to)
+ return coalesceShifts(shifts)
+}
+
+// ResolveRange renders the final schedule for [from, to): per-layer coverage,
+// stacked so that a later layer's coverage replaces earlier layers where they
+// overlap (PagerDuty "higher layer number takes precedence"), with overrides
+// on top of everything.
+func ResolveRange(def *models.OncallScheduleDefinition, tz *time.Location, overrides []*models.OncallOverride, from, to time.Time) []Shift {
+ if !from.Before(to) {
+ return nil
+ }
+ var result []Shift
+ for i := range def.Layers {
+ layerShifts := ResolveLayerRange(&def.Layers[i], tz, from, to)
+ if len(layerShifts) == 0 {
+ continue
+ }
+ result = subtractShifts(result, spansOf(layerShifts))
+ result = append(result, layerShifts...)
+ }
+
+ // Later-created overrides win over earlier ones, and every override wins
+ // over layer coverage, so apply them in created_at order: each subtracts
+ // what came before it.
+ sorted := make([]*models.OncallOverride, len(overrides))
+ copy(sorted, overrides)
+ sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].CreatedAt.Before(sorted[j].CreatedAt) })
+ for _, override := range sorted {
+ start := maxTime(override.StartAt.UTC(), from)
+ end := minTime(override.EndAt.UTC(), to)
+ if !start.Before(end) {
+ continue
+ }
+ result = subtractShifts(result, []span{{start, end}})
+ result = append(result, Shift{
+ UserId: override.UserId,
+ Start: start,
+ End: end,
+ IsOverride: true,
+ })
+ }
+
+ sort.Slice(result, func(i, j int) bool { return result[i].Start.Before(result[j].Start) })
+ return coalesceShifts(result)
+}
+
+// ResolveAt answers "who is on call at t" with the same stacking ResolveRange
+// renders: a later layer replaces the layers under it and an override replaces
+// everything, so at most one person is on call. Returning every covering user
+// instead would page the person an override was meant to relieve, and page one
+// person per layer on a stacked schedule. Empty means nobody is on call.
+func ResolveAt(def *models.OncallScheduleDefinition, tz *time.Location, overrides []*models.OncallOverride, at time.Time) []int {
+ for _, shift := range ResolveRange(def, tz, overrides, at, at.Add(time.Nanosecond)) {
+ if !shift.Start.After(at) && shift.End.After(at) {
+ return []int{shift.UserId}
+ }
+ }
+ return nil
+}
+
+// CurrentAndNext gives the UI "on call until X, next up Y". The lookahead is
+// bounded so sparse schedules terminate.
+func CurrentAndNext(def *models.OncallScheduleDefinition, tz *time.Location, overrides []*models.OncallOverride, now time.Time) (current *Shift, next *Shift) {
+ shifts := ResolveRange(def, tz, overrides, now, now.AddDate(0, 0, 35))
+ for i := range shifts {
+ shift := shifts[i]
+ if !shift.Start.After(now) && shift.End.After(now) {
+ current = &shift
+ continue
+ }
+ if shift.Start.After(now) {
+ next = &shift
+ break
+ }
+ }
+ return current, next
+}
+
+func spansOf(shifts []Shift) []span {
+ spans := make([]span, 0, len(shifts))
+ for _, shift := range shifts {
+ spans = append(spans, span{shift.Start, shift.End})
+ }
+ return mergeSpans(spans)
+}
+
+func mergeSpans(spans []span) []span {
+ if len(spans) == 0 {
+ return nil
+ }
+ sort.Slice(spans, func(i, j int) bool { return spans[i].start.Before(spans[j].start) })
+ merged := []span{spans[0]}
+ for _, s := range spans[1:] {
+ last := &merged[len(merged)-1]
+ if !s.start.After(last.end) {
+ if s.end.After(last.end) {
+ last.end = s.end
+ }
+ continue
+ }
+ merged = append(merged, s)
+ }
+ return merged
+}
+
+// intersectShifts cuts shifts down to the parts covered by windows.
+func intersectShifts(shifts []Shift, windows []span) []Shift {
+ windows = mergeSpans(windows)
+ var result []Shift
+ for _, shift := range shifts {
+ for _, w := range windows {
+ start := maxTime(shift.Start, w.start)
+ end := minTime(shift.End, w.end)
+ if start.Before(end) {
+ cut := shift
+ cut.Start = start
+ cut.End = end
+ result = append(result, cut)
+ }
+ }
+ }
+ return result
+}
+
+// subtractShifts removes the given spans from shifts, splitting where needed.
+func subtractShifts(shifts []Shift, remove []span) []Shift {
+ remove = mergeSpans(remove)
+ var result []Shift
+ for _, shift := range shifts {
+ pieces := []span{{shift.Start, shift.End}}
+ for _, r := range remove {
+ var next []span
+ for _, p := range pieces {
+ if !r.end.After(p.start) || !p.end.After(r.start) {
+ next = append(next, p)
+ continue
+ }
+ if r.start.After(p.start) {
+ next = append(next, span{p.start, minTime(p.end, r.start)})
+ }
+ if r.end.Before(p.end) {
+ next = append(next, span{maxTime(p.start, r.end), p.end})
+ }
+ }
+ pieces = next
+ }
+ for _, p := range pieces {
+ if p.start.Before(p.end) {
+ cut := shift
+ cut.Start = p.start
+ cut.End = p.end
+ result = append(result, cut)
+ }
+ }
+ }
+ return result
+}
+
+func clampShifts(shifts []Shift, from, to time.Time) []Shift {
+ var result []Shift
+ for _, shift := range shifts {
+ start := maxTime(shift.Start, from)
+ end := minTime(shift.End, to)
+ if start.Before(end) {
+ shift.Start = start
+ shift.End = end
+ result = append(result, shift)
+ }
+ }
+ return result
+}
+
+func coalesceShifts(shifts []Shift) []Shift {
+ if len(shifts) == 0 {
+ return nil
+ }
+ sort.Slice(shifts, func(i, j int) bool { return shifts[i].Start.Before(shifts[j].Start) })
+ result := []Shift{shifts[0]}
+ for _, shift := range shifts[1:] {
+ last := &result[len(result)-1]
+ if shift.UserId == last.UserId && shift.LayerId == last.LayerId && shift.IsOverride == last.IsOverride && shift.Start.Equal(last.End) {
+ last.End = shift.End
+ continue
+ }
+ result = append(result, shift)
+ }
+ return result
+}
+
+func maxTime(a, b time.Time) time.Time {
+ if a.After(b) {
+ return a
+ }
+ return b
+}
+
+func minTime(a, b time.Time) time.Time {
+ if a.Before(b) {
+ return a
+ }
+ return b
+}
diff --git a/backend/app/oncall/resolve_test.go b/backend/app/oncall/resolve_test.go
new file mode 100644
index 00000000..b5290105
--- /dev/null
+++ b/backend/app/oncall/resolve_test.go
@@ -0,0 +1,329 @@
+package oncall
+
+import (
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+// daytimeLayer covers 09:00-17:00 every day via a daily restriction.
+func daytimeLayer(id string, userIds []int) models.OncallLayer {
+ return models.OncallLayer{
+ Id: id,
+ Name: "Daytime",
+ RotationType: RotationWeekly,
+ HandoffTime: "09:00",
+ HandoffDay: 1,
+ RotationStart: "2026-06-01",
+ UserIds: userIds,
+ Restrictions: []models.OncallRestriction{
+ {Type: RestrictionDaily, StartTime: "09:00", EndTime: "17:00"},
+ },
+ }
+}
+
+func baseLayer(id string, userIds []int) models.OncallLayer {
+ return models.OncallLayer{
+ Id: id,
+ Name: "Base",
+ RotationType: RotationWeekly,
+ HandoffTime: "09:00",
+ HandoffDay: 1,
+ RotationStart: "2026-06-01",
+ UserIds: userIds,
+ }
+}
+
+func TestDailyRestrictionBusinessHours(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1", Name: "Days", RotationType: RotationDaily, HandoffTime: "09:00",
+ RotationStart: "2026-06-01", UserIds: []int{1, 2},
+ Restrictions: []models.OncallRestriction{{Type: RestrictionDaily, StartTime: "09:00", EndTime: "17:00"}},
+ }
+ from := utc(t, "2026-06-01T00:00:00Z")
+ to := utc(t, "2026-06-03T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 1, utc(t, "2026-06-01T09:00:00Z"), utc(t, "2026-06-01T17:00:00Z"))
+ assertShift(t, shifts[1], 2, utc(t, "2026-06-02T09:00:00Z"), utc(t, "2026-06-02T17:00:00Z"))
+}
+
+func TestDailyRestrictionWrapsMidnight(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1", Name: "Nights", RotationType: RotationDaily, HandoffTime: "22:00",
+ RotationStart: "2026-06-01", UserIds: []int{1},
+ Restrictions: []models.OncallRestriction{{Type: RestrictionDaily, StartTime: "22:00", EndTime: "06:00"}},
+ }
+ from := utc(t, "2026-06-01T00:00:00Z")
+ to := utc(t, "2026-06-03T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ // 00:00-06:00 (tail of prior night), 22:00-06:00, 22:00-24:00 clamp.
+ if len(shifts) != 3 {
+ t.Fatalf("expected 3 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 1, utc(t, "2026-06-01T00:00:00Z"), utc(t, "2026-06-01T06:00:00Z"))
+ assertShift(t, shifts[1], 1, utc(t, "2026-06-01T22:00:00Z"), utc(t, "2026-06-02T06:00:00Z"))
+ assertShift(t, shifts[2], 1, utc(t, "2026-06-02T22:00:00Z"), utc(t, "2026-06-03T00:00:00Z"))
+}
+
+func TestDailyRestrictionInsideSpringForwardGap(t *testing.T) {
+ tz := mustLoad(t, "America/New_York")
+ layer := &models.OncallLayer{
+ Id: "l_1", Name: "Gap", RotationType: RotationDaily, HandoffTime: "00:00",
+ RotationStart: "2026-03-01", UserIds: []int{1},
+ // 02:00 does not exist on 2026-03-08 (DST starts at 02:00) and Go
+ // normalizes it to 01:00 EST — before the window's 01:30 start. An
+ // instant-based wrap check would treat that as a midnight-wrapping
+ // window and inflate it to ~24h; the window must instead vanish for
+ // the transition day.
+ Restrictions: []models.OncallRestriction{{Type: RestrictionDaily, StartTime: "01:30", EndTime: "02:00"}},
+ }
+ from := local(t, tz, "2026-03-07 00:00").UTC()
+ to := local(t, tz, "2026-03-09 12:00").UTC()
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts (03-07 and 03-09, nothing on the transition day), got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 1, local(t, tz, "2026-03-07 01:30").UTC(), local(t, tz, "2026-03-07 02:00").UTC())
+ assertShift(t, shifts[1], 1, local(t, tz, "2026-03-09 01:30").UTC(), local(t, tz, "2026-03-09 02:00").UTC())
+ for _, shift := range shifts {
+ if shift.End.Sub(shift.Start) > time.Hour {
+ t.Errorf("restriction window inflated to %v, want <= 1h", shift.End.Sub(shift.Start))
+ }
+ }
+}
+
+func TestWeeklyRestrictionSpansDays(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1", Name: "Weekend", RotationType: RotationWeekly, HandoffTime: "09:00", HandoffDay: 1,
+ RotationStart: "2026-06-01", UserIds: []int{4},
+ // Friday 17:00 -> Monday 09:00.
+ Restrictions: []models.OncallRestriction{{Type: RestrictionWeekly, StartDay: 5, StartTime: "17:00", EndDay: 1, EndTime: "09:00"}},
+ }
+ from := utc(t, "2026-06-01T00:00:00Z") // Monday
+ to := utc(t, "2026-06-09T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ // Tail of the prior weekend (Mon 00:00-09:00) + Fri 17:00 -> Mon 09:00... clamped at `to` Mon 00:00? No: to is Tue 00:00. 2026-06-05 is Friday.
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 4, utc(t, "2026-06-01T00:00:00Z"), utc(t, "2026-06-01T09:00:00Z"))
+ assertShift(t, shifts[1], 4, utc(t, "2026-06-05T17:00:00Z"), utc(t, "2026-06-08T09:00:00Z"))
+}
+
+// A weekly window that lands inside a spring-forward gap covers nothing that
+// week. Only that week may be lost: bailing out of the expansion would leave
+// the rest of the month with no coverage at all.
+func TestWeeklyRestrictionSurvivesADSTGapWeek(t *testing.T) {
+ tz := mustLoad(t, "America/New_York")
+ layer := &models.OncallLayer{
+ Id: "l_1", Name: "Sunday night", RotationType: RotationWeekly, HandoffTime: "09:00", HandoffDay: 1,
+ RotationStart: "2026-02-01", UserIds: []int{4},
+ // Sunday 01:30 -> 02:30 local; on 2026-03-08 that hour does not exist.
+ Restrictions: []models.OncallRestriction{{Type: RestrictionWeekly, StartDay: 7, StartTime: "01:30", EndDay: 7, EndTime: "02:30"}},
+ }
+ from := utc(t, "2026-03-01T00:00:00Z")
+ to := utc(t, "2026-04-01T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ // Five Sundays in the range, minus the one swallowed by the DST gap.
+ if len(shifts) != 4 {
+ t.Fatalf("expected 4 weekly windows (5 Sundays less the DST-gap week), got %d: %+v", len(shifts), shifts)
+ }
+}
+
+func TestHandoffInsideRestrictionChangesUser(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1", Name: "Days", RotationType: RotationDaily, HandoffTime: "12:00",
+ RotationStart: "2026-06-01", UserIds: []int{1, 2},
+ Restrictions: []models.OncallRestriction{{Type: RestrictionDaily, StartTime: "09:00", EndTime: "17:00"}},
+ }
+ from := utc(t, "2026-06-01T00:00:00Z")
+ to := utc(t, "2026-06-02T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts (split at the 12:00 handoff), got %d: %+v", len(shifts), shifts)
+ }
+ // The 09:00-12:00 stretch belongs to the period that began May 31 12:00
+ // (index -1, floor-mod -> user 2); the handoff at 12:00 rotates to user 1.
+ assertShift(t, shifts[0], 2, utc(t, "2026-06-01T09:00:00Z"), utc(t, "2026-06-01T12:00:00Z"))
+ assertShift(t, shifts[1], 1, utc(t, "2026-06-01T12:00:00Z"), utc(t, "2026-06-01T17:00:00Z"))
+}
+
+func TestLayerStackingLaterLayerWins(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ def := &models.OncallScheduleDefinition{
+ SchemaVersion: 1,
+ Layers: []models.OncallLayer{
+ baseLayer("l_base", []int{1}),
+ daytimeLayer("l_week", []int{2}),
+ },
+ }
+ from := utc(t, "2026-06-01T09:00:00Z") // Monday 09:00
+ to := utc(t, "2026-06-02T09:00:00Z")
+ shifts := ResolveRange(def, tz, nil, from, to)
+ // Weekday layer covers Mon 09:00-17:00, base fills 17:00-Tue 09:00.
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 2, utc(t, "2026-06-01T09:00:00Z"), utc(t, "2026-06-01T17:00:00Z"))
+ assertShift(t, shifts[1], 1, utc(t, "2026-06-01T17:00:00Z"), utc(t, "2026-06-02T09:00:00Z"))
+}
+
+func TestOverrideSplitsShift(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ def := &models.OncallScheduleDefinition{
+ SchemaVersion: 1,
+ Layers: []models.OncallLayer{baseLayer("l_base", []int{1})},
+ }
+ overrides := []*models.OncallOverride{{
+ Id: 1, UserId: 9,
+ StartAt: utc(t, "2026-06-01T12:00:00Z"),
+ EndAt: utc(t, "2026-06-01T14:00:00Z"),
+ CreatedAt: utc(t, "2026-05-30T00:00:00Z"),
+ }}
+ from := utc(t, "2026-06-01T09:00:00Z")
+ to := utc(t, "2026-06-01T17:00:00Z")
+ shifts := ResolveRange(def, tz, overrides, from, to)
+ if len(shifts) != 3 {
+ t.Fatalf("expected 3 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 1, utc(t, "2026-06-01T09:00:00Z"), utc(t, "2026-06-01T12:00:00Z"))
+ assertShift(t, shifts[1], 9, utc(t, "2026-06-01T12:00:00Z"), utc(t, "2026-06-01T14:00:00Z"))
+ assertShift(t, shifts[2], 1, utc(t, "2026-06-01T14:00:00Z"), utc(t, "2026-06-01T17:00:00Z"))
+ if !shifts[1].IsOverride {
+ t.Error("middle shift should be an override")
+ }
+}
+
+func TestOverlappingOverridesLatestCreatedWins(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ def := &models.OncallScheduleDefinition{
+ SchemaVersion: 1,
+ Layers: []models.OncallLayer{baseLayer("l_base", []int{1})},
+ }
+ overrides := []*models.OncallOverride{
+ {Id: 1, UserId: 8, StartAt: utc(t, "2026-06-01T10:00:00Z"), EndAt: utc(t, "2026-06-01T16:00:00Z"), CreatedAt: utc(t, "2026-05-29T00:00:00Z")},
+ {Id: 2, UserId: 9, StartAt: utc(t, "2026-06-01T12:00:00Z"), EndAt: utc(t, "2026-06-01T14:00:00Z"), CreatedAt: utc(t, "2026-05-30T00:00:00Z")},
+ }
+ from := utc(t, "2026-06-01T09:00:00Z")
+ to := utc(t, "2026-06-01T17:00:00Z")
+ shifts := ResolveRange(def, tz, overrides, from, to)
+ if len(shifts) != 5 {
+ t.Fatalf("expected 5 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[1], 8, utc(t, "2026-06-01T10:00:00Z"), utc(t, "2026-06-01T12:00:00Z"))
+ assertShift(t, shifts[2], 9, utc(t, "2026-06-01T12:00:00Z"), utc(t, "2026-06-01T14:00:00Z"))
+ assertShift(t, shifts[3], 8, utc(t, "2026-06-01T14:00:00Z"), utc(t, "2026-06-01T16:00:00Z"))
+}
+
+func TestEmptyDefinitionResolvesToNobody(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ def := &models.OncallScheduleDefinition{SchemaVersion: 1}
+ if shifts := ResolveRange(def, tz, nil, utc(t, "2026-06-01T00:00:00Z"), utc(t, "2026-06-02T00:00:00Z")); len(shifts) != 0 {
+ t.Errorf("expected no shifts, got %+v", shifts)
+ }
+ if users := ResolveAt(def, tz, nil, utc(t, "2026-06-01T00:00:00Z")); len(users) != 0 {
+ t.Errorf("expected nobody on call, got %v", users)
+ }
+}
+
+// ResolveAt must agree with the timeline ResolveRange renders: whoever the
+// stack puts on top, and nobody else. Anything looser pages the people an
+// override or a higher layer was meant to relieve.
+func TestResolveAtAppliesFullPrecedence(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ def := &models.OncallScheduleDefinition{
+ SchemaVersion: 1,
+ Layers: []models.OncallLayer{
+ baseLayer("l_base", []int{1}),
+ daytimeLayer("l_week", []int{2}),
+ },
+ }
+ overrides := []*models.OncallOverride{{
+ Id: 1, UserId: 9,
+ StartAt: utc(t, "2026-06-01T00:00:00Z"),
+ EndAt: utc(t, "2026-06-02T00:00:00Z"),
+ CreatedAt: utc(t, "2026-05-30T00:00:00Z"),
+ }}
+ // Monday 10:00: override beats the weekday layer, which beats the base.
+ users := ResolveAt(def, tz, overrides, utc(t, "2026-06-01T10:00:00Z"))
+ if len(users) != 1 || users[0] != 9 {
+ t.Errorf("expected only the override user [9], got %v", users)
+ }
+ // Same instant with no override: the weekday layer covers, not the base.
+ users = ResolveAt(def, tz, nil, utc(t, "2026-06-01T10:00:00Z"))
+ if len(users) != 1 || users[0] != 2 {
+ t.Errorf("expected only the weekday layer user [2], got %v", users)
+ }
+ // Monday 20:00: weekday layer restricted out, base takes over.
+ users = ResolveAt(def, tz, nil, utc(t, "2026-06-01T20:00:00Z"))
+ if len(users) != 1 || users[0] != 1 {
+ t.Errorf("expected [1], got %v", users)
+ }
+}
+
+func TestCurrentAndNext(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ def := &models.OncallScheduleDefinition{
+ SchemaVersion: 1,
+ Layers: []models.OncallLayer{{
+ Id: "l_1", Name: "Primary", RotationType: RotationDaily, HandoffTime: "09:00",
+ RotationStart: "2026-06-01", UserIds: []int{1, 2},
+ }},
+ }
+ now := utc(t, "2026-06-01T12:00:00Z")
+ current, next := CurrentAndNext(def, tz, nil, now)
+ if current == nil || current.UserId != 1 {
+ t.Fatalf("current = %+v, want user 1", current)
+ }
+ if next == nil || next.UserId != 2 || !next.Start.Equal(utc(t, "2026-06-02T09:00:00Z")) {
+ t.Fatalf("next = %+v, want user 2 starting 06-02 09:00", next)
+ }
+}
+
+func TestParseDefinitionValidation(t *testing.T) {
+ cases := []struct {
+ name string
+ json string
+ }{
+ {"bad json", `{`},
+ {"bad schema version", `{"schemaVersion": 99}`},
+ {"missing layer name", `{"layers":[{"rotationType":"daily","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1]}]}`},
+ {"bad rotation type", `{"layers":[{"name":"A","rotationType":"hourly","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1]}]}`},
+ {"bad handoff time", `{"layers":[{"name":"A","rotationType":"daily","handoffTime":"9am","rotationStart":"2026-06-01","userIds":[1]}]}`},
+ {"bad rotation start", `{"layers":[{"name":"A","rotationType":"daily","handoffTime":"09:00","rotationStart":"June 1","userIds":[1]}]}`},
+ {"no members", `{"layers":[{"name":"A","rotationType":"daily","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[]}]}`},
+ {"duplicate members", `{"layers":[{"name":"A","rotationType":"daily","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1,1]}]}`},
+ {"weekly without handoff day", `{"layers":[{"name":"A","rotationType":"weekly","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1]}]}`},
+ {"custom without interval", `{"layers":[{"name":"A","rotationType":"custom","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1]}]}`},
+ {"bad restriction type", `{"layers":[{"name":"A","rotationType":"daily","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1],"restrictions":[{"type":"monthly","startTime":"09:00","endTime":"17:00"}]}]}`},
+ {"weekly restriction bad day", `{"layers":[{"name":"A","rotationType":"daily","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1],"restrictions":[{"type":"weekly","startDay":0,"endDay":5,"startTime":"09:00","endTime":"17:00"}]}]}`},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ if _, err := ParseDefinition([]byte(tc.json)); err == nil {
+ t.Errorf("expected validation error for %s", tc.name)
+ }
+ })
+ }
+}
+
+func TestParseDefinitionAssignsLayerIds(t *testing.T) {
+ def, err := ParseDefinition([]byte(`{"layers":[{"name":"A","rotationType":"daily","handoffTime":"09:00","rotationStart":"2026-06-01","userIds":[1]}]}`))
+ if err != nil {
+ t.Fatalf("unexpected error: %v", err)
+ }
+ if def.Layers[0].Id == "" {
+ t.Error("expected a generated layer id")
+ }
+ if def.SchemaVersion != SchemaVersion {
+ t.Errorf("schemaVersion = %d, want %d", def.SchemaVersion, SchemaVersion)
+ }
+}
diff --git a/backend/app/oncall/restrictions.go b/backend/app/oncall/restrictions.go
new file mode 100644
index 00000000..f410af36
--- /dev/null
+++ b/backend/app/oncall/restrictions.go
@@ -0,0 +1,113 @@
+package oncall
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+const (
+ RestrictionDaily = "daily"
+ RestrictionWeekly = "weekly"
+)
+
+// expandRestrictions turns recurring restriction windows into concrete UTC
+// spans overlapping [from, to). Windows are computed in the schedule timezone.
+func expandRestrictions(restrictions []models.OncallRestriction, tz *time.Location, from, to time.Time) []span {
+ var windows []span
+ for _, restriction := range restrictions {
+ switch restriction.Type {
+ case RestrictionDaily:
+ windows = append(windows, expandDaily(restriction, tz, from, to)...)
+ case RestrictionWeekly:
+ windows = append(windows, expandWeekly(restriction, tz, from, to)...)
+ }
+ }
+ return mergeSpans(windows)
+}
+
+func expandDaily(restriction models.OncallRestriction, tz *time.Location, from, to time.Time) []span {
+ startHour, startMinute, ok := parseLocalTime(restriction.StartTime)
+ if !ok {
+ return nil
+ }
+ endHour, endMinute, ok := parseLocalTime(restriction.EndTime)
+ if !ok {
+ return nil
+ }
+
+ // Whether the window wraps past midnight is decided from the configured
+ // wall-clock values, not the normalized instants: on a DST spring-forward
+ // day a window inside the skipped hour collapses to a single instant,
+ // which must yield no coverage rather than a full wrapped day.
+ endDayOffset := 0
+ if !isTimeAfter(endHour, endMinute, startHour, startMinute) {
+ endDayOffset = 1
+ }
+
+ var windows []span
+ // Start one day early so a window that wraps past midnight into the range
+ // is not missed.
+ local := from.In(tz)
+ for day := -1; ; day++ {
+ start := time.Date(local.Year(), local.Month(), local.Day()+day, startHour, startMinute, 0, 0, tz)
+ end := time.Date(local.Year(), local.Month(), local.Day()+day+endDayOffset, endHour, endMinute, 0, 0, tz)
+ if start.After(to) {
+ break
+ }
+ if end.After(start) && end.After(from) {
+ windows = append(windows, span{maxTime(start.UTC(), from), minTime(end.UTC(), to)})
+ }
+ }
+ return windows
+}
+
+func expandWeekly(restriction models.OncallRestriction, tz *time.Location, from, to time.Time) []span {
+ startHour, startMinute, ok := parseLocalTime(restriction.StartTime)
+ if !ok {
+ return nil
+ }
+ endHour, endMinute, ok := parseLocalTime(restriction.EndTime)
+ if !ok {
+ return nil
+ }
+ if restriction.StartDay < 1 || restriction.StartDay > 7 || restriction.EndDay < 1 || restriction.EndDay > 7 {
+ return nil
+ }
+
+ // Anchor on the StartDay occurrence at or before `from`, minus one extra
+ // week so a window already in progress is included.
+ local := from.In(tz)
+ daysSinceStartDay := (int(local.Weekday()) - restriction.StartDay%7 + 7) % 7
+ anchorYear, anchorMonth, anchorDay := local.Year(), local.Month(), local.Day()-daysSinceStartDay-7
+
+ dayLength := (restriction.EndDay - restriction.StartDay + 7) % 7
+ wrapsWeek := dayLength == 0 && !isTimeAfter(endHour, endMinute, startHour, startMinute)
+ if wrapsWeek {
+ dayLength = 7
+ }
+
+ var windows []span
+ for week := 0; ; week++ {
+ start := time.Date(anchorYear, anchorMonth, anchorDay+week*7, startHour, startMinute, 0, 0, tz)
+ end := time.Date(anchorYear, anchorMonth, anchorDay+week*7+dayLength, endHour, endMinute, 0, 0, tz)
+ if start.After(to) {
+ break
+ }
+ // A window whose end normalizes back onto its start (a spring-forward
+ // gap swallowed it) covers nothing that week. Skip that week only:
+ // abandoning the loop would drop every later week too. expandDaily
+ // makes the same distinction.
+ if end.After(start) && end.After(from) {
+ windows = append(windows, span{maxTime(start.UTC(), from), minTime(end.UTC(), to)})
+ }
+ }
+ return windows
+}
+
+func isTimeAfter(aHour, aMinute, bHour, bMinute int) bool {
+ if aHour != bHour {
+ return aHour > bHour
+ }
+ return aMinute > bMinute
+}
diff --git a/backend/app/oncall/rotation.go b/backend/app/oncall/rotation.go
new file mode 100644
index 00000000..1e3e6a31
--- /dev/null
+++ b/backend/app/oncall/rotation.go
@@ -0,0 +1,109 @@
+package oncall
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+const (
+ RotationDaily = "daily"
+ RotationWeekly = "weekly"
+ RotationCustom = "custom"
+)
+
+// layerPeriods returns the raw rotation periods of a layer overlapping
+// [from, to), before restrictions. Period boundaries are computed in calendar
+// space (time.Date in the schedule timezone), so a handoff configured at 09:00
+// stays at 09:00 local time across DST transitions and transition days are
+// simply 23 or 25 hours long.
+func layerPeriods(layer *models.OncallLayer, tz *time.Location, from, to time.Time) []Shift {
+ year, month, day, ok := parseLocalDate(layer.RotationStart)
+ if !ok {
+ return nil
+ }
+ hour, minute, ok := parseLocalTime(layer.HandoffTime)
+ if !ok {
+ return nil
+ }
+
+ periodDays := 1
+ switch layer.RotationType {
+ case RotationDaily:
+ case RotationWeekly:
+ periodDays = 7
+ case RotationCustom:
+ if layer.IntervalDays < 1 {
+ return nil
+ }
+ periodDays = layer.IntervalDays
+ default:
+ return nil
+ }
+
+ if layer.RotationType == RotationWeekly && layer.HandoffDay >= 1 && layer.HandoffDay <= 7 {
+ // Shift the anchor back to the configured handoff weekday, so period 0
+ // is the week containing the rotation start. The weekday is read from
+ // the civil date, not a tz instant, which a DST gap could normalize
+ // onto the previous day.
+ startWeekday := int(time.Date(year, month, day, 12, 0, 0, 0, time.UTC).Weekday())
+ day -= (startWeekday - layer.HandoffDay%7 + 7) % 7
+ }
+
+ boundary := func(k int) time.Time {
+ return time.Date(year, month, day+k*periodDays, hour, minute, 0, 0, tz)
+ }
+ anchor := boundary(0)
+
+ // Locate the period containing `from` by day-count estimate, then correct
+ // so that boundary(k) <= from < boundary(k+1). Integer seconds rather than
+ // from.Sub(anchor), which saturates at ~292 years.
+ k := floorDiv(floorDiv(int(from.Unix()-anchor.Unix()), 86400), periodDays)
+ for boundary(k).After(from) {
+ k--
+ }
+ for !boundary(k + 1).After(from) {
+ k++
+ }
+
+ memberCount := len(layer.UserIds)
+ var shifts []Shift
+ for start := boundary(k); start.Before(to); k, start = k+1, boundary(k+1) {
+ end := boundary(k + 1)
+ shifts = append(shifts, Shift{
+ UserId: layer.UserIds[floorMod(k, memberCount)],
+ LayerId: layer.Id,
+ Start: start.UTC(),
+ End: end.UTC(),
+ })
+ }
+ return shifts
+}
+
+func floorMod(a, n int) int {
+ return ((a % n) + n) % n
+}
+
+func floorDiv(a, n int) int {
+ q := a / n
+ if a%n != 0 && (a < 0) != (n < 0) {
+ q--
+ }
+ return q
+}
+
+func parseLocalDate(s string) (year int, month time.Month, day int, ok bool) {
+ t, err := time.Parse("2006-01-02", s)
+ if err != nil {
+ return 0, 0, 0, false
+ }
+ return t.Year(), t.Month(), t.Day(), true
+}
+
+func parseLocalTime(s string) (hour, minute int, ok bool) {
+ t, err := time.Parse("15:04", s)
+ if err != nil {
+ return 0, 0, false
+ }
+ return t.Hour(), t.Minute(), true
+}
diff --git a/backend/app/oncall/rotation_test.go b/backend/app/oncall/rotation_test.go
new file mode 100644
index 00000000..a86d748e
--- /dev/null
+++ b/backend/app/oncall/rotation_test.go
@@ -0,0 +1,306 @@
+package oncall
+
+import (
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+func mustLoad(t *testing.T, name string) *time.Location {
+ t.Helper()
+ tz, err := time.LoadLocation(name)
+ if err != nil {
+ t.Fatalf("load tz %s: %v", name, err)
+ }
+ return tz
+}
+
+func utc(t *testing.T, value string) time.Time {
+ t.Helper()
+ parsed, err := time.Parse(time.RFC3339, value)
+ if err != nil {
+ t.Fatalf("parse %s: %v", value, err)
+ }
+ return parsed.UTC()
+}
+
+func local(t *testing.T, tz *time.Location, value string) time.Time {
+ t.Helper()
+ parsed, err := time.ParseInLocation("2006-01-02 15:04", value, tz)
+ if err != nil {
+ t.Fatalf("parse %s: %v", value, err)
+ }
+ return parsed
+}
+
+func assertShift(t *testing.T, shift Shift, userId int, start, end time.Time) {
+ t.Helper()
+ if shift.UserId != userId {
+ t.Errorf("shift user = %d, want %d (shift %v..%v)", shift.UserId, userId, shift.Start, shift.End)
+ }
+ if !shift.Start.Equal(start) {
+ t.Errorf("shift start = %v, want %v", shift.Start, start)
+ }
+ if !shift.End.Equal(end) {
+ t.Errorf("shift end = %v, want %v", shift.End, end)
+ }
+}
+
+func TestDailyRotation(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "09:00",
+ RotationStart: "2026-06-01",
+ UserIds: []int{1, 2, 3},
+ }
+ from := utc(t, "2026-06-01T09:00:00Z")
+ to := utc(t, "2026-06-04T09:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 3 {
+ t.Fatalf("expected 3 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 1, utc(t, "2026-06-01T09:00:00Z"), utc(t, "2026-06-02T09:00:00Z"))
+ assertShift(t, shifts[1], 2, utc(t, "2026-06-02T09:00:00Z"), utc(t, "2026-06-03T09:00:00Z"))
+ assertShift(t, shifts[2], 3, utc(t, "2026-06-03T09:00:00Z"), utc(t, "2026-06-04T09:00:00Z"))
+}
+
+func TestDailyRotationBeforeAnchor(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "09:00",
+ RotationStart: "2026-06-10",
+ UserIds: []int{1, 2, 3},
+ }
+ // Two days before the anchor: floor-mod should give user 2 (index -2 -> 1).
+ from := utc(t, "2026-06-08T09:00:00Z")
+ to := utc(t, "2026-06-09T09:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 1 {
+ t.Fatalf("expected 1 shift, got %d", len(shifts))
+ }
+ assertShift(t, shifts[0], 2, from, to)
+}
+
+func TestWeeklyRotationHandoffDay(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ // 2026-06-03 is a Wednesday; handoff day Monday (1) at 09:00.
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationWeekly,
+ HandoffTime: "09:00",
+ HandoffDay: 1,
+ RotationStart: "2026-06-03",
+ UserIds: []int{1, 2},
+ }
+ // Period 0 anchors back to Monday 2026-06-01 09:00.
+ from := utc(t, "2026-06-01T09:00:00Z")
+ to := utc(t, "2026-06-15T09:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 1, utc(t, "2026-06-01T09:00:00Z"), utc(t, "2026-06-08T09:00:00Z"))
+ assertShift(t, shifts[1], 2, utc(t, "2026-06-08T09:00:00Z"), utc(t, "2026-06-15T09:00:00Z"))
+}
+
+func TestCustomIntervalRotation(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationCustom,
+ HandoffTime: "00:00",
+ IntervalDays: 3,
+ RotationStart: "2026-06-01",
+ UserIds: []int{7, 8},
+ }
+ from := utc(t, "2026-06-01T00:00:00Z")
+ to := utc(t, "2026-06-07T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d", len(shifts))
+ }
+ assertShift(t, shifts[0], 7, utc(t, "2026-06-01T00:00:00Z"), utc(t, "2026-06-04T00:00:00Z"))
+ assertShift(t, shifts[1], 8, utc(t, "2026-06-04T00:00:00Z"), utc(t, "2026-06-07T00:00:00Z"))
+}
+
+func TestSingleUserLayerCoalesces(t *testing.T) {
+ tz := mustLoad(t, "UTC")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Solo",
+ RotationType: RotationDaily,
+ HandoffTime: "09:00",
+ RotationStart: "2026-06-01",
+ UserIds: []int{5},
+ }
+ from := utc(t, "2026-06-01T00:00:00Z")
+ to := utc(t, "2026-06-08T00:00:00Z")
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 1 {
+ t.Fatalf("expected a single coalesced shift, got %d: %+v", len(shifts), shifts)
+ }
+ assertShift(t, shifts[0], 5, from, to)
+}
+
+func TestDSTSpringForwardHandoffStaysLocal(t *testing.T) {
+ tz := mustLoad(t, "America/New_York")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "09:00",
+ RotationStart: "2026-03-06",
+ UserIds: []int{1, 2},
+ }
+ // DST starts 2026-03-08 in the US: 07 09:00 EST -> 08 09:00 EDT is 23h.
+ from := local(t, tz, "2026-03-07 09:00").UTC()
+ to := local(t, tz, "2026-03-09 09:00").UTC()
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ first := shifts[0].End.Sub(shifts[0].Start)
+ if first != 23*time.Hour {
+ t.Errorf("spring-forward day length = %v, want 23h", first)
+ }
+ for _, shift := range shifts {
+ if localTime := shift.Start.In(tz); localTime.Hour() != 9 || localTime.Minute() != 0 {
+ t.Errorf("handoff drifted to %02d:%02d local, want 09:00", localTime.Hour(), localTime.Minute())
+ }
+ }
+}
+
+func TestDSTFallBackDayIs25Hours(t *testing.T) {
+ tz := mustLoad(t, "America/New_York")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "09:00",
+ RotationStart: "2026-10-30",
+ UserIds: []int{1, 2},
+ }
+ // DST ends 2026-11-01: the 10-31 09:00 -> 11-01 09:00 period is 25h.
+ from := local(t, tz, "2026-10-31 09:00").UTC()
+ to := local(t, tz, "2026-11-01 09:00").UTC()
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 1 {
+ t.Fatalf("expected 1 shift, got %d", len(shifts))
+ }
+ if length := shifts[0].End.Sub(shifts[0].Start); length != 25*time.Hour {
+ t.Errorf("fall-back day length = %v, want 25h", length)
+ }
+}
+
+func TestDSTSkippedHandoffTimeNormalizesForward(t *testing.T) {
+ tz := mustLoad(t, "America/New_York")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "02:30",
+ RotationStart: "2026-03-07",
+ UserIds: []int{1, 2},
+ }
+ // 02:30 does not exist on 2026-03-08; time.Date resolves it to the same
+ // instant under one of the two offsets. The invariants that matter: the
+ // transition period is 23h, and the handoff is back at 02:30 local the
+ // next day.
+ from := local(t, tz, "2026-03-07 02:30").UTC()
+ to := local(t, tz, "2026-03-09 02:30").UTC()
+ shifts := ResolveLayerRange(layer, tz, from, to)
+ if len(shifts) != 2 {
+ t.Fatalf("expected 2 shifts, got %d: %+v", len(shifts), shifts)
+ }
+ if length := shifts[0].End.Sub(shifts[0].Start); length != 23*time.Hour {
+ t.Errorf("transition period length = %v, want 23h", length)
+ }
+ nextDay := shifts[1].End.In(tz)
+ if nextDay.Hour() != 2 || nextDay.Minute() != 30 {
+ t.Errorf("handoff after transition at %02d:%02d local, want 02:30", nextDay.Hour(), nextDay.Minute())
+ }
+}
+
+// A rotation start whose handoff time falls inside a DST gap must not shift
+// the handoff of every later period.
+func TestRotationStartInsideDSTGapKeepsHandoffTime(t *testing.T) {
+ tz := mustLoad(t, "America/New_York")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "02:30",
+ RotationStart: "2026-03-08", // 02:00 -> 03:00 that morning; 02:30 does not exist
+ UserIds: []int{1, 2, 3},
+ }
+ from := local(t, tz, "2026-06-10 00:00").UTC()
+ shifts := ResolveLayerRange(layer, tz, from, from.AddDate(0, 0, 5))
+ if len(shifts) == 0 {
+ t.Fatal("expected shifts")
+ }
+ for _, shift := range shifts[1:] {
+ start := shift.Start.In(tz)
+ if start.Hour() != 2 || start.Minute() != 30 {
+ t.Errorf("handoff at %s, want 02:30 local", start.Format("2006-01-02 15:04 MST"))
+ }
+ }
+}
+
+// Same defect on the weekly path: the handoff weekday must not move.
+func TestRotationStartInsideDSTGapKeepsHandoffWeekday(t *testing.T) {
+ tz := mustLoad(t, "America/Havana")
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationWeekly,
+ HandoffDay: 1, // Monday
+ HandoffTime: "00:00",
+ RotationStart: "2026-03-08", // local midnight does not exist that day
+ UserIds: []int{1, 2},
+ }
+ from := local(t, tz, "2026-06-01 00:00").UTC()
+ shifts := ResolveLayerRange(layer, tz, from, from.AddDate(0, 0, 28))
+ if len(shifts) == 0 {
+ t.Fatal("expected shifts")
+ }
+ for _, shift := range shifts[1:] {
+ start := shift.Start.In(tz)
+ if start.Weekday() != time.Monday {
+ t.Errorf("handoff on %s, want Monday", start.Format("Mon 2006-01-02 15:04 MST"))
+ }
+ }
+}
+
+// Far-future dates must resolve arithmetically, not by walking periods.
+func TestExtremeDatesResolveInConstantTime(t *testing.T) {
+ tz := mustLoad(t, "Europe/Berlin")
+ for _, rotationStart := range []string{"2026-01-05", "9999-12-31", "0001-01-01"} {
+ layer := &models.OncallLayer{
+ Id: "l_1",
+ Name: "Primary",
+ RotationType: RotationDaily,
+ HandoffTime: "09:00",
+ RotationStart: rotationStart,
+ UserIds: []int{1, 2},
+ }
+ for _, from := range []time.Time{
+ local(t, tz, "2026-06-01 00:00").UTC(),
+ time.Date(9999, 1, 1, 0, 0, 0, 0, time.UTC),
+ } {
+ start := time.Now()
+ ResolveLayerRange(layer, tz, from, from.AddDate(0, 0, MaxTimelineRangeDays))
+ if elapsed := time.Since(start); elapsed > 250*time.Millisecond {
+ t.Errorf("rotationStart=%s from=%s took %v", rotationStart, from.Format("2006-01-02"), elapsed)
+ }
+ }
+ }
+}
diff --git a/backend/app/oncall/rules_chain_test.go b/backend/app/oncall/rules_chain_test.go
new file mode 100644
index 00000000..b5138d1a
--- /dev/null
+++ b/backend/app/oncall/rules_chain_test.go
@@ -0,0 +1,448 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package oncall
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional/shared"
+)
+
+func TestResolveUrgencyMatrix(t *testing.T) {
+ cases := []struct {
+ policy string
+ severity string
+ want string
+ }{
+ {"", "critical", models.UrgencyHigh},
+ {"", "", models.UrgencyHigh},
+ {"", "warning", models.UrgencyLow},
+ {"", "info", models.UrgencyLow},
+ {"auto", "critical", models.UrgencyHigh},
+ {"auto", "info", models.UrgencyLow},
+ {"high", "info", models.UrgencyHigh},
+ {"low", "critical", models.UrgencyLow},
+ {"junk", "critical", models.UrgencyHigh},
+ }
+ for _, tc := range cases {
+ if got := ResolveUrgency(tc.policy, tc.severity); got != tc.want {
+ t.Errorf("ResolveUrgency(%q, %q) = %q, want %q", tc.policy, tc.severity, got, tc.want)
+ }
+ }
+}
+
+func TestPolicyUrgencyValidationAndParsing(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ valid := `{"steps":[{"targets":[{"type":"user","id":` + itoa(fixture.Alice) + `}],"delayMinutes":5}],"urgency":"high"}`
+ if _, err := ValidatePolicyDefinition(tx, fixture.OrgId, []byte(valid)); err != nil {
+ t.Errorf("high urgency should validate: %v", err)
+ }
+ bad := `{"steps":[{"targets":[{"type":"user","id":` + itoa(fixture.Alice) + `}],"delayMinutes":5}],"urgency":"shout"}`
+ if _, err := ValidatePolicyDefinition(tx, fixture.OrgId, []byte(bad)); err == nil {
+ t.Error("junk urgency should 422")
+ }
+ return struct{}{}, nil
+ })
+ if err != nil {
+ t.Fatalf("tx: %v", err)
+ }
+
+ // Old definitions without the field parse to auto.
+ definition, err := ParsePolicyDefinition([]byte(`{"schemaVersion":1,"steps":[],"repeatCount":0}`))
+ if err != nil {
+ t.Fatalf("parse: %v", err)
+ }
+ if definition.Urgency != "" {
+ t.Errorf("legacy definition urgency = %q, want empty (auto)", definition.Urgency)
+ }
+}
+
+func itoa(v int) string {
+ return strconv.Itoa(v)
+}
+
+func createContactMethod(t *testing.T, userId int, methodType string, config string, enabled bool, verified bool) int {
+ t.Helper()
+ id, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return transactional.UserContactMethodRepository.Create(tx, &models.UserContactMethod{
+ UserId: userId,
+ MethodType: methodType,
+ Config: models.JSONText(config),
+ Enabled: enabled,
+ Verified: verified,
+ CreatedAt: time.Now().UTC(),
+ })
+ })
+ if err != nil {
+ t.Fatalf("create contact method: %v", err)
+ }
+ return id
+}
+
+func setRules(t *testing.T, userId int, urgency string, steps []models.UserNotificationRule) {
+ t.Helper()
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ rules := make([]*models.UserNotificationRule, 0, len(steps))
+ for i := range steps {
+ rule := steps[i]
+ rule.UserId = userId
+ rule.Urgency = urgency
+ rule.Position = i
+ rule.CreatedAt = time.Now().UTC()
+ rules = append(rules, &rule)
+ }
+ return struct{}{}, transactional.UserNotificationRuleRepository.ReplaceForUser(tx, userId, rules)
+ })
+ if err != nil {
+ t.Fatalf("set rules: %v", err)
+ }
+}
+
+func outboxRowsForPage(t *testing.T, pageId int) []*models.OutboxDelivery {
+ t.Helper()
+ rows, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindCancellable(tx, outbox.PageCancelKey(pageId))
+ })
+ if err != nil {
+ t.Fatalf("load outbox rows: %v", err)
+ }
+ return rows
+}
+
+func singleStepPolicy(userId int) string {
+ return `{"schemaVersion":1,"steps":[{"targets":[{"type":"user","id":` + strconv.Itoa(userId) + `}],"delayMinutes":30}],"repeatCount":0}`
+}
+
+// enableTwilioForTest makes SMS a sendable method for the duration of one
+// test; without credentials the escalator skips SMS methods entirely.
+func enableTwilioForTest(t *testing.T) {
+ t.Helper()
+ previous := *config.Config
+ config.Config.TwilioAccountSID = "ACtest"
+ config.Config.TwilioAuthToken = "test-token"
+ config.Config.TwilioFromNumber = "+15005550006"
+ t.Cleanup(func() { *config.Config = previous })
+}
+
+func TestRuleChainEnqueuesStaggeredSteps(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ enableTwilioForTest(t)
+ emailId := createContactMethod(t, fixture.Alice, "email", `{}`, true, true)
+ smsId := createContactMethod(t, fixture.Alice, "sms", `{"phoneNumber":"+12025550123"}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{
+ {ContactMethodId: smsId, DelayMinutes: 0},
+ {ContactMethodId: emailId, DelayMinutes: 5},
+ })
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "chain1|/issues/abc")
+
+ now := time.Now().UTC()
+ runEscalatorTick(context.Background(), now)
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 2 {
+ t.Fatalf("expected 2 chain rows, got %d", len(rows))
+ }
+ if rows[0].MethodType != "sms" || rows[0].ScheduledFor == nil {
+ t.Errorf("first step = %+v, want immediate sms", rows[0])
+ }
+ if rows[1].MethodType != "email" || rows[1].ScheduledFor == nil {
+ t.Fatalf("second step = %+v, want delayed email", rows[1])
+ }
+ if delay := rows[1].ScheduledFor.Sub(now); delay < 4*time.Minute || delay > 6*time.Minute {
+ t.Errorf("email step scheduled in %v, want ~5m", delay)
+ }
+ if rows[0].AckTokenHash == "" || rows[1].AckTokenHash == "" || rows[0].AckTokenHash == rows[1].AckTokenHash {
+ t.Error("each delivery row should carry a distinct ack token hash")
+ }
+
+ outboxRows := outboxRowsForPage(t, page.Id)
+ if len(outboxRows) != 2 {
+ t.Fatalf("expected 2 outbox rows, got %d", len(outboxRows))
+ }
+ var delayed *models.OutboxDelivery
+ for _, row := range outboxRows {
+ if row.AdapterType == "email" {
+ delayed = row
+ }
+ }
+ if delayed == nil {
+ t.Fatal("expected an email outbox row")
+ }
+ if gap := delayed.NextAttemptAt.Sub(now); gap < 4*time.Minute || gap > 6*time.Minute {
+ t.Errorf("email outbox NotBefore in %v, want ~5m", gap)
+ }
+}
+
+func TestAckCancelsChainTail(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ emailId := createContactMethod(t, fixture.Alice, "email", `{}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{
+ {ContactMethodId: emailId, DelayMinutes: 0},
+ {ContactMethodId: emailId, DelayMinutes: 10},
+ })
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "chain2|/issues/abc")
+
+ now := time.Now().UTC()
+ tickAndDrain(t, now)
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 2 {
+ t.Fatalf("expected 2 rows, got %d", len(rows))
+ }
+ if rows[0].Status != models.PageNotificationSent {
+ t.Errorf("immediate step = %s, want sent", rows[0].Status)
+ }
+ if rows[1].Status != models.PageNotificationPending {
+ t.Errorf("delayed step = %s, want pending", rows[1].Status)
+ }
+
+ acknowledged, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return AcknowledgePage(tx, page.Id, &fixture.Alice, AckViaDashboard, now)
+ })
+ if err != nil || !acknowledged {
+ t.Fatalf("acknowledge: %v (%v)", err, acknowledged)
+ }
+ rows = pageNotifications(t, page.Id)
+ if rows[1].Status != models.PageNotificationCancelled {
+ t.Errorf("delayed step after ack = %s, want cancelled", rows[1].Status)
+ }
+ page = reloadPage(t, page.Id)
+ if page.AcknowledgedVia != AckViaDashboard {
+ t.Errorf("acknowledged_via = %q, want dashboard", page.AcknowledgedVia)
+ }
+}
+
+func TestRuleChainSkipsUnusableStepsAndFallsBack(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ disabledId := createContactMethod(t, fixture.Alice, "email", `{}`, false, true)
+ unverifiedSmsId := createContactMethod(t, fixture.Alice, "sms", `{"phoneNumber":"+12025550123"}`, true, false)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{
+ {ContactMethodId: disabledId, DelayMinutes: 0},
+ {ContactMethodId: unverifiedSmsId, DelayMinutes: 5},
+ })
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "chain3|/issues/abc")
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected 1 fallback row, got %d", len(rows))
+ }
+ if rows[0].MethodType != "email" || rows[0].TargetDesc != "alice@example.com (email)" {
+ t.Errorf("fallback should page the account email, got %+v", rows[0])
+ }
+}
+
+// Dropping the leading zero-delay step must not postpone the first page.
+func TestRuleChainRebasesWhenTheLeadingStepIsDropped(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ disabledId := createContactMethod(t, fixture.Alice, "email", `{"email":"first@example.com"}`, false, true)
+ secondId := createContactMethod(t, fixture.Alice, "email", `{"email":"second@example.com"}`, true, true)
+ thirdId := createContactMethod(t, fixture.Alice, "email", `{"email":"third@example.com"}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{
+ {ContactMethodId: disabledId, DelayMinutes: 0},
+ {ContactMethodId: secondId, DelayMinutes: 15},
+ {ContactMethodId: thirdId, DelayMinutes: 20},
+ })
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "rebase1|/issues/abc")
+ now := time.Now().UTC()
+ runEscalatorTick(context.Background(), now)
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 2 {
+ t.Fatalf("expected the 2 surviving steps, got %d", len(rows))
+ }
+ if rows[0].TargetDesc != "s***@example.com (email)" {
+ t.Errorf("first surviving step = %q, want no delay suffix", rows[0].TargetDesc)
+ }
+ if rows[0].ScheduledFor == nil || rows[0].ScheduledFor.After(now.Add(time.Minute)) {
+ t.Errorf("first surviving step scheduled for %v, want immediately", rows[0].ScheduledFor)
+ }
+ // 20m - 15m: the spacing between the surviving steps is preserved.
+ if rows[1].TargetDesc != "t***@example.com (email), +5m" {
+ t.Errorf("second surviving step = %q, want a 5 minute gap", rows[1].TargetDesc)
+ }
+}
+
+// An intact leading delay is kept; the rebase must not erase it.
+func TestRuleChainKeepsAnIntentionalLeadingDelay(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ firstId := createContactMethod(t, fixture.Alice, "email", `{"email":"first@example.com"}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{
+ {ContactMethodId: firstId, DelayMinutes: 10},
+ })
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "rebase2|/issues/abc")
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 || rows[0].TargetDesc != "f***@example.com (email), +10m" {
+ t.Errorf("intact leading delay should survive, got %+v", rows)
+ }
+}
+
+// An oversized target_desc must be clamped, not fail the claim insert.
+func TestOversizedTargetDescIsClamped(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ long := strings.Repeat("a", 400) + "@example.com"
+ methodId := createContactMethod(t, fixture.Alice, "email", `{"email":"`+long+`"}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{
+ {ContactMethodId: methodId, DelayMinutes: 0},
+ })
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "clamp1|/issues/abc")
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected 1 notification, got %d", len(rows))
+ }
+ if len(rows[0].TargetDesc) > maxTargetDescLength {
+ t.Errorf("target desc is %d chars, want at most %d", len(rows[0].TargetDesc), maxTargetDescLength)
+ }
+}
+
+// An SMS method can outlive its transport: the credentials are removed after
+// the method was verified. The page must not vanish into an unsendable
+// channel, so SMS is skipped and the account-email fallback still fires.
+func TestSmsMethodsSkippedWithoutTwilio(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ if config.Config.TwilioEnabled() {
+ t.Skip("Twilio configured in this environment")
+ }
+ smsId := createContactMethod(t, fixture.Alice, "sms", `{"phoneNumber":"+12025550123"}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{{ContactMethodId: smsId, DelayMinutes: 0}})
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "nosms|/issues/abc")
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 {
+ t.Fatalf("expected 1 fallback row, got %d", len(rows))
+ }
+ if rows[0].MethodType != "email" || rows[0].TargetDesc != "alice@example.com (email)" {
+ t.Errorf("unsendable sms should fall back to the account email, got %+v", rows[0])
+ }
+}
+
+func TestLowUrgencyPageRunsLowChain(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ emailId := createContactMethod(t, fixture.Alice, "email", `{"email":"low@example.com"}`, true, true)
+ smsId := createContactMethod(t, fixture.Alice, "sms", `{"phoneNumber":"+12025550123"}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{{ContactMethodId: smsId, DelayMinutes: 0}})
+ // setRules replaces the WHOLE rule set, so write both chains at once.
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ now := time.Now().UTC()
+ return struct{}{}, transactional.UserNotificationRuleRepository.ReplaceForUser(tx, fixture.Alice, []*models.UserNotificationRule{
+ {UserId: fixture.Alice, Urgency: models.UrgencyHigh, Position: 0, DelayMinutes: 0, ContactMethodId: smsId, CreatedAt: now},
+ {UserId: fixture.Alice, Urgency: models.UrgencyLow, Position: 0, DelayMinutes: 0, ContactMethodId: emailId, CreatedAt: now},
+ })
+ })
+ if err != nil {
+ t.Fatalf("set chains: %v", err)
+ }
+
+ // warning severity + auto urgency -> low chain (email), not sms.
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ if _, err := openPage(openPageParams{
+ PolicyId: policyId, ProjectId: fixture.ProjectId, RuleName: "r", RuleType: "new_error",
+ Subject: "warn", Severity: "warning", DedupKey: "chain4|/issues/w",
+ }); err != nil {
+ t.Fatalf("open page: %v", err)
+ }
+ page := findPageByDedupKey(t, "chain4|/issues/w")
+ if page.Urgency != models.UrgencyLow {
+ t.Fatalf("page urgency = %q, want low", page.Urgency)
+ }
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ rows := pageNotifications(t, page.Id)
+ if len(rows) != 1 || rows[0].MethodType != "email" || rows[0].TargetDesc != "l***@example.com (email)" {
+ t.Errorf("low chain should run the email step, got %+v", rows)
+ }
+}
+
+func TestAckTokenFlow(t *testing.T) {
+ fixture := setupEscalatorDB(t)
+ emailId := createContactMethod(t, fixture.Alice, "email", `{}`, true, true)
+ setRules(t, fixture.Alice, models.UrgencyHigh, []models.UserNotificationRule{{ContactMethodId: emailId, DelayMinutes: 0}})
+
+ policyId := createPolicy(t, fixture.OrgId, singleStepPolicy(fixture.Alice))
+ page := openTestPageForPolicy(t, fixture, policyId, "token1|/issues/abc")
+ runEscalatorTick(context.Background(), time.Now().UTC())
+
+ // The plaintext token exists only inside the outgoing message.
+ outboxRows := outboxRowsForPage(t, page.Id)
+ if len(outboxRows) != 1 {
+ t.Fatalf("expected 1 outbox row, got %d", len(outboxRows))
+ }
+ var msg models.NotificationMessage
+ if err := jsonUnmarshalForTest(outboxRows[0].Message, &msg); err != nil {
+ t.Fatalf("decode message: %v", err)
+ }
+ idx := strings.LastIndex(msg.URL, "/ack/")
+ if idx < 0 {
+ t.Fatalf("message URL carries no ack link: %q", msg.URL)
+ }
+ token := msg.URL[idx+len("/ack/"):]
+ if !strings.HasPrefix(token, "twk_") {
+ t.Fatalf("unexpected token shape %q", token)
+ }
+
+ // Token resolves to the delivery row, attributed to Alice.
+ notification, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.PageNotification, error) {
+ return transactional.PageNotificationRepository.FindByAckTokenHash(tx, shared.HashAuthToken(token))
+ })
+ if err != nil || notification == nil {
+ t.Fatalf("token lookup failed: %v (%v)", err, notification)
+ }
+ if notification.UserId == nil || *notification.UserId != fixture.Alice {
+ t.Fatalf("token attributed to %v, want alice", notification.UserId)
+ }
+
+ // Link-ack records via + attribution and stops the escalation.
+ acknowledged, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return AcknowledgePage(tx, page.Id, notification.UserId, AckViaLink, time.Now().UTC())
+ })
+ if err != nil || !acknowledged {
+ t.Fatalf("link ack failed: %v (%v)", err, acknowledged)
+ }
+ page = reloadPage(t, page.Id)
+ if page.AcknowledgedVia != AckViaLink || page.AcknowledgedBy == nil || *page.AcknowledgedBy != fixture.Alice {
+ t.Errorf("link ack not recorded: via=%q by=%v", page.AcknowledgedVia, page.AcknowledgedBy)
+ }
+
+ // Wrong token finds nothing.
+ missing, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.PageNotification, error) {
+ return transactional.PageNotificationRepository.FindByAckTokenHash(tx, shared.HashAuthToken("twk_bogus"))
+ })
+ if err != nil || missing != nil {
+ t.Errorf("bogus token should resolve to nothing, got %v (%v)", missing, err)
+ }
+}
+
+func jsonUnmarshalForTest(data []byte, v any) error {
+ return json.Unmarshal(data, v)
+}
diff --git a/backend/app/oncall/service.go b/backend/app/oncall/service.go
new file mode 100644
index 00000000..e360b60e
--- /dev/null
+++ b/backend/app/oncall/service.go
@@ -0,0 +1,189 @@
+package oncall
+
+import (
+ "database/sql"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ traceway "go.tracewayapp.com"
+)
+
+type ScheduleRef struct {
+ Id int `json:"id"`
+ Name string `json:"name"`
+}
+
+type OncallUser struct {
+ UserId int `json:"userId"`
+ Name string `json:"name"`
+ Email string `json:"email"`
+}
+
+// ProjectOnCall is the ownership seam: the team owning a project and who is on
+// call for it right now.
+type ProjectOnCall struct {
+ Team *models.Team `json:"team"`
+ Schedules []ScheduleRef `json:"schedules"`
+ Oncall []OncallUser `json:"oncall"`
+}
+
+// CurrentOnCallForSchedule resolves who is on call for a schedule at the given
+// instant, filtered to current members of the schedule's organization. A
+// missing schedule resolves to nobody rather than an error, so dangling
+// references never abort a caller.
+func CurrentOnCallForSchedule(tx *sql.Tx, scheduleId int, at time.Time) ([]int, error) {
+ schedule, err := transactional.OncallScheduleRepository.FindById(tx, scheduleId)
+ if err != nil {
+ return nil, err
+ }
+ if schedule == nil {
+ return nil, nil
+ }
+ userIds, err := resolveScheduleAt(tx, schedule, at)
+ if err != nil {
+ return nil, err
+ }
+ members, err := memberDetails(tx, schedule.OrganizationId)
+ if err != nil {
+ return nil, err
+ }
+ var filtered []int
+ for _, userId := range userIds {
+ if _, ok := members[userId]; ok {
+ filtered = append(filtered, userId)
+ }
+ }
+ return filtered, nil
+}
+
+// RemoveUserFromOrgSchedules scrubs a removed member from the organization's
+// on-call data: their overrides are deleted and they are dropped from every
+// schedule layer (a layer left with no members is dropped with them, keeping
+// stored definitions valid). Resolution already filters to current org
+// members; this keeps schedules and timelines from showing phantom coverage.
+func RemoveUserFromOrgSchedules(tx *sql.Tx, organizationId int, userId int) error {
+ if err := transactional.OncallOverrideRepository.DeleteByOrganizationAndUser(tx, organizationId, userId); err != nil {
+ return err
+ }
+ schedules, err := transactional.OncallScheduleRepository.ListByOrganization(tx, organizationId)
+ if err != nil {
+ return err
+ }
+ now := time.Now().UTC()
+ for _, schedule := range schedules {
+ def, err := ParseDefinition(schedule.Definition)
+ if err != nil {
+ // An unparseable definition cannot be scrubbed; resolution filters
+ // removed members out regardless.
+ traceway.CaptureException(fmt.Errorf("cannot scrub user %d from schedule %d: %w", userId, schedule.Id, err))
+ continue
+ }
+ changed := false
+ layers := make([]models.OncallLayer, 0, len(def.Layers))
+ for _, layer := range def.Layers {
+ userIds := make([]int, 0, len(layer.UserIds))
+ for _, id := range layer.UserIds {
+ if id == userId {
+ changed = true
+ continue
+ }
+ userIds = append(userIds, id)
+ }
+ if len(userIds) == 0 {
+ continue
+ }
+ layer.UserIds = userIds
+ layers = append(layers, layer)
+ }
+ if !changed {
+ continue
+ }
+ def.Layers = layers
+ raw, err := MarshalDefinition(def)
+ if err != nil {
+ return err
+ }
+ schedule.Definition = raw
+ schedule.UpdatedAt = now
+ if err := transactional.OncallScheduleRepository.Update(tx, schedule); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// CurrentOnCallForProject resolves the owning team's current on-call across
+// all of its schedules (schedule creation order), filtered to current org
+// members. Returns nil when the project has no owning team.
+func CurrentOnCallForProject(tx *sql.Tx, projectId uuid.UUID, at time.Time) (*ProjectOnCall, error) {
+ team, err := transactional.TeamRepository.FindTeamForProject(tx, projectId)
+ if err != nil {
+ return nil, err
+ }
+ if team == nil {
+ return nil, nil
+ }
+ schedules, err := transactional.OncallScheduleRepository.ListByTeam(tx, team.Id)
+ if err != nil {
+ return nil, err
+ }
+ members, err := memberDetails(tx, team.OrganizationId)
+ if err != nil {
+ return nil, err
+ }
+
+ result := &ProjectOnCall{Team: team, Schedules: []ScheduleRef{}, Oncall: []OncallUser{}}
+ seen := map[int]bool{}
+ for _, schedule := range schedules {
+ result.Schedules = append(result.Schedules, ScheduleRef{Id: schedule.Id, Name: schedule.Name})
+ userIds, err := resolveScheduleAt(tx, schedule, at)
+ if err != nil {
+ return nil, err
+ }
+ for _, userId := range userIds {
+ member, ok := members[userId]
+ if !ok || seen[userId] {
+ continue
+ }
+ seen[userId] = true
+ result.Oncall = append(result.Oncall, OncallUser{UserId: userId, Name: member.Name, Email: member.Email})
+ }
+ }
+ return result, nil
+}
+
+func resolveScheduleAt(tx *sql.Tx, schedule *models.OncallSchedule, at time.Time) ([]int, error) {
+ tz, err := time.LoadLocation(schedule.Timezone)
+ if err != nil {
+ tz = time.UTC
+ }
+ def, err := ParseDefinition(schedule.Definition)
+ if err != nil {
+ // A stored definition that no longer parses must not wedge a page's
+ // escalation (the claim would fail and retry forever): treat it like a
+ // dangling target — report it and resolve to nobody, so other targets
+ // and later levels still run.
+ traceway.CaptureException(fmt.Errorf("schedule %d definition no longer parses, resolving to nobody on call: %w", schedule.Id, err))
+ return nil, nil
+ }
+ overrides, err := transactional.OncallOverrideRepository.ListForRange(tx, schedule.Id, at, at.Add(time.Nanosecond))
+ if err != nil {
+ return nil, err
+ }
+ return ResolveAt(def, tz, overrides, at), nil
+}
+
+func memberDetails(tx *sql.Tx, organizationId int) (map[int]*models.OrganizationMember, error) {
+ members, err := transactional.OrganizationRepository.GetMembersWithDetails(tx, organizationId)
+ if err != nil {
+ return nil, err
+ }
+ byId := make(map[int]*models.OrganizationMember, len(members))
+ for _, member := range members {
+ byId[member.Id] = member
+ }
+ return byId, nil
+}
diff --git a/backend/app/oncall/validate.go b/backend/app/oncall/validate.go
new file mode 100644
index 00000000..eb82acfb
--- /dev/null
+++ b/backend/app/oncall/validate.go
@@ -0,0 +1,142 @@
+package oncall
+
+import (
+ "crypto/rand"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+const SchemaVersion = 1
+
+const (
+ MaxLayersPerSchedule = 20
+ MaxUsersPerLayer = 50
+ MaxRestrictionsPerLayer = 10
+ MaxCustomIntervalDays = 365
+ MaxTimelineRangeDays = 62
+ MaxOverrideDurationDays = 30
+)
+
+// ParseDefinition parses and validates a schedule definition, assigning ids to
+// layers that lack one. Validation errors are user-facing (422) messages.
+func ParseDefinition(raw []byte) (*models.OncallScheduleDefinition, error) {
+ def := &models.OncallScheduleDefinition{}
+ if len(raw) > 0 {
+ if err := json.Unmarshal(raw, def); err != nil {
+ return nil, errors.New("The schedule definition is not valid JSON.")
+ }
+ }
+ if def.SchemaVersion != 0 && def.SchemaVersion != SchemaVersion {
+ return nil, fmt.Errorf("Unsupported schedule schemaVersion %d, this server supports schemaVersion %d.", def.SchemaVersion, SchemaVersion)
+ }
+ def.SchemaVersion = SchemaVersion
+ if def.Layers == nil {
+ def.Layers = []models.OncallLayer{}
+ }
+ if len(def.Layers) > MaxLayersPerSchedule {
+ return nil, fmt.Errorf("A schedule can have at most %d layers.", MaxLayersPerSchedule)
+ }
+ for i := range def.Layers {
+ if err := validateLayer(&def.Layers[i], i); err != nil {
+ return nil, err
+ }
+ }
+ return def, nil
+}
+
+func validateLayer(layer *models.OncallLayer, index int) error {
+ if layer.Id == "" {
+ layer.Id = NewLayerId()
+ }
+ if layer.Name == "" {
+ return fmt.Errorf("Layer %d needs a name.", index+1)
+ }
+ switch layer.RotationType {
+ case RotationDaily:
+ case RotationWeekly:
+ if layer.HandoffDay < 1 || layer.HandoffDay > 7 {
+ return fmt.Errorf("Layer %q needs a handoff day between Monday and Sunday.", layer.Name)
+ }
+ case RotationCustom:
+ if layer.IntervalDays < 1 || layer.IntervalDays > MaxCustomIntervalDays {
+ return fmt.Errorf("Layer %q needs a rotation interval between 1 and %d days.", layer.Name, MaxCustomIntervalDays)
+ }
+ default:
+ return fmt.Errorf("Layer %q has an unknown rotation type %q.", layer.Name, layer.RotationType)
+ }
+ if _, _, ok := parseLocalTime(layer.HandoffTime); !ok {
+ return fmt.Errorf("Layer %q needs a handoff time in HH:MM format.", layer.Name)
+ }
+ if _, _, _, ok := parseLocalDate(layer.RotationStart); !ok {
+ return fmt.Errorf("Layer %q needs a rotation start date in YYYY-MM-DD format.", layer.Name)
+ }
+ if len(layer.UserIds) == 0 {
+ return fmt.Errorf("Layer %q needs at least one member.", layer.Name)
+ }
+ if len(layer.UserIds) > MaxUsersPerLayer {
+ return fmt.Errorf("Layer %q can have at most %d members.", layer.Name, MaxUsersPerLayer)
+ }
+ seen := map[int]bool{}
+ for _, userId := range layer.UserIds {
+ if seen[userId] {
+ return fmt.Errorf("Layer %q lists the same member twice.", layer.Name)
+ }
+ seen[userId] = true
+ }
+ if len(layer.Restrictions) > MaxRestrictionsPerLayer {
+ return fmt.Errorf("Layer %q can have at most %d restrictions.", layer.Name, MaxRestrictionsPerLayer)
+ }
+ for _, restriction := range layer.Restrictions {
+ if err := validateRestriction(layer.Name, restriction); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func validateRestriction(layerName string, restriction models.OncallRestriction) error {
+ if restriction.Type != RestrictionDaily && restriction.Type != RestrictionWeekly {
+ return fmt.Errorf("Layer %q has a restriction with unknown type %q.", layerName, restriction.Type)
+ }
+ if _, _, ok := parseLocalTime(restriction.StartTime); !ok {
+ return fmt.Errorf("Layer %q has a restriction start time that is not HH:MM.", layerName)
+ }
+ if _, _, ok := parseLocalTime(restriction.EndTime); !ok {
+ return fmt.Errorf("Layer %q has a restriction end time that is not HH:MM.", layerName)
+ }
+ if restriction.Type == RestrictionWeekly {
+ if restriction.StartDay < 1 || restriction.StartDay > 7 || restriction.EndDay < 1 || restriction.EndDay > 7 {
+ return fmt.Errorf("Layer %q has a weekly restriction with days outside Monday..Sunday.", layerName)
+ }
+ }
+ return nil
+}
+
+// LoadTimezone validates an IANA timezone name, returning a user-facing error.
+func LoadTimezone(name string) (*time.Location, error) {
+ if name == "" {
+ return nil, errors.New("A timezone is required.")
+ }
+ tz, err := time.LoadLocation(name)
+ if err != nil {
+ return nil, fmt.Errorf("Unknown timezone %q.", name)
+ }
+ return tz, nil
+}
+
+func NewLayerId() string {
+ b := make([]byte, 4)
+ if _, err := rand.Read(b); err != nil {
+ panic(err)
+ }
+ return "l_" + hex.EncodeToString(b)
+}
+
+func MarshalDefinition(def *models.OncallScheduleDefinition) ([]byte, error) {
+ return json.Marshal(def)
+}
diff --git a/backend/app/outbox/drain.go b/backend/app/outbox/drain.go
new file mode 100644
index 00000000..6bb0a89d
--- /dev/null
+++ b/backend/app/outbox/drain.go
@@ -0,0 +1,253 @@
+package outbox
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ traceway "go.tracewayapp.com"
+)
+
+const (
+ drainAdvisoryLockId = 824737003 // escalator: 824737002, backfill: 824737001
+ drainBatchSize = 50
+ drainConcurrency = 8
+ sendTimeout = 30 * time.Second
+ sendAbandonGrace = 5 * time.Second
+ staleSendingAge = 5 * time.Minute
+ maxAttempts = 5
+)
+
+// backoffSchedule[attempts-1] = delay before the next attempt (attempts was
+// already incremented at claim time). attempts >= maxAttempts is terminal.
+var backoffSchedule = []time.Duration{1 * time.Minute, 5 * time.Minute, 15 * time.Minute, 60 * time.Minute}
+
+var (
+ sentTotal uint64
+ terminalFailuresTotal uint64
+)
+
+func drainPollInterval() time.Duration {
+ return config.PollSeconds(config.Config.OutboxPollSeconds, 15)
+}
+
+// StartDrain runs the outbox drain loop: claim due rows in a transaction
+// (marking them sending), deliver after commit, finalize each row in its own
+// small transaction. All state lives in notification_outbox, so a crash at any
+// point is recovered: unclaimed rows stay due, stale sending rows are
+// reclaimed, and the guarded status transitions make cancellation win races.
+func StartDrain(ctx context.Context) {
+ go func() {
+ defer traceway.Recover()
+
+ ticker := time.NewTicker(drainPollInterval())
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ case <-wakeCh:
+ }
+ DrainOnce(ctx, time.Now().UTC())
+ }
+ }()
+}
+
+// DrainOnce runs a single drain tick: reclaim stale sending rows, claim due
+// rows, deliver them, finalize. Exported so tests (and tooling) can drive the
+// outbox deterministically.
+func DrainOnce(ctx context.Context, now time.Time) {
+ dueCount := 0
+ claimed, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.OutboxDelivery, error) {
+ if !db.IsSQLite() {
+ if _, err := tx.Exec(fmt.Sprintf("SELECT pg_advisory_xact_lock(%d)", drainAdvisoryLockId)); err != nil {
+ return nil, fmt.Errorf("failed to take outbox drain lock: %w", err)
+ }
+ }
+ if err := transactional.OutboxRepository.ReclaimStaleSending(tx, now.Add(-staleSendingAge), now); err != nil {
+ return nil, err
+ }
+ due, err := transactional.OutboxRepository.FindDue(tx, now, drainBatchSize)
+ if err != nil {
+ return nil, err
+ }
+ dueCount = len(due)
+ claimed := make([]*models.OutboxDelivery, 0, len(due))
+ for _, row := range due {
+ won, err := transactional.OutboxRepository.MarkSending(tx, row.Id, now)
+ if err != nil {
+ return nil, err
+ }
+ if !won {
+ // Cancelled (acknowledge/resolve) since the due list was read;
+ // the row must not be sent.
+ continue
+ }
+ row.Attempts++
+ claimed = append(claimed, row)
+ }
+ return claimed, nil
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("outbox drain tick failed: %w", err))
+ return
+ }
+
+ var wg sync.WaitGroup
+ sem := make(chan struct{}, drainConcurrency)
+ for _, row := range claimed {
+ wg.Add(1)
+ sem <- struct{}{}
+ go func(row *models.OutboxDelivery) {
+ defer traceway.Recover()
+ defer wg.Done()
+ defer func() { <-sem }()
+ sendRow(ctx, row)
+ }(row)
+ }
+ wg.Wait()
+
+ if dueCount == drainBatchSize {
+ // A full batch means more rows may already be due.
+ Wake()
+ }
+}
+
+func sendRow(ctx context.Context, row *models.OutboxDelivery) {
+ if sender == nil {
+ finalizeRow(row, errors.New("outbox sender not registered"))
+ return
+ }
+ var msg models.NotificationMessage
+ if err := json.Unmarshal(row.Message, &msg); err != nil {
+ // Poison row: retrying cannot fix an unreadable payload.
+ finalizeTerminal(row, "unreadable message payload: "+err.Error())
+ return
+ }
+ finalizeRow(row, sendWithDeadline(ctx, row, msg))
+}
+
+// sendWithDeadline bounds one attempt in wall-clock time; an adapter that
+// ignores its context deadline is abandoned and the row retries (at-least-once).
+func sendWithDeadline(ctx context.Context, row *models.OutboxDelivery, msg models.NotificationMessage) error {
+ sendCtx, cancel := context.WithTimeout(ctx, sendTimeout)
+ defer cancel()
+
+ done := make(chan error, 1)
+ go func() {
+ defer func() {
+ if r := recover(); r != nil {
+ done <- fmt.Errorf("adapter %q panicked: %v", row.AdapterType, r)
+ }
+ }()
+ done <- sender(sendCtx, row.AdapterType, json.RawMessage(row.AdapterConfig), msg)
+ }()
+
+ abandonAfter := time.NewTimer(sendTimeout + sendAbandonGrace)
+ defer abandonAfter.Stop()
+ select {
+ case err := <-done:
+ return err
+ case <-abandonAfter.C:
+ return fmt.Errorf("adapter %q ignored its %s deadline and was abandoned", row.AdapterType, sendTimeout)
+ }
+}
+
+func finalizeRow(row *models.OutboxDelivery, sendErr error) {
+ if sendErr != nil && row.Attempts >= maxAttempts {
+ finalizeTerminal(row, sendErr.Error())
+ return
+ }
+ now := time.Now().UTC()
+ finalized, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ if sendErr == nil {
+ sent, err := transactional.OutboxRepository.MarkSent(tx, row.Id, now)
+ if err != nil {
+ return false, err
+ }
+ if !sent {
+ // Cancelled while the send was in flight (the documented
+ // at-least-once window): the cancel already mirrored the
+ // page_notifications row, which must not be rewritten.
+ return false, nil
+ }
+ if row.PageNotificationId != nil {
+ if err := transactional.PageNotificationRepository.MarkSent(tx, *row.PageNotificationId, now); err != nil {
+ return false, err
+ }
+ }
+ return true, nil
+ }
+ next := now.Add(backoffSchedule[backoffIndex(row.Attempts)])
+ return transactional.OutboxRepository.MarkFailedWithBackoff(tx, row.Id, sendErr.Error(), &next, now)
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to finalize outbox row %d: %w", row.Id, err))
+ return
+ }
+ if sendErr != nil {
+ // Retryable failures must be visible before they turn terminal.
+ traceway.CaptureException(fmt.Errorf("outbox delivery %d (%s via %s) attempt %d failed, will retry: %w", row.Id, row.Kind, row.AdapterType, row.Attempts, sendErr))
+ return
+ }
+ if finalized {
+ atomic.AddUint64(&sentTotal, 1)
+ if terminalHook != nil {
+ terminalHook(row, models.OutboxSent, "")
+ }
+ }
+}
+
+func finalizeTerminal(row *models.OutboxDelivery, errorMsg string) {
+ now := time.Now().UTC()
+ finalized, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ failed, err := transactional.OutboxRepository.MarkFailedWithBackoff(tx, row.Id, errorMsg, nil, now)
+ if err != nil {
+ return false, err
+ }
+ if !failed {
+ // Cancelled while the send was in flight; the cancel already
+ // mirrored the page_notifications row.
+ return false, nil
+ }
+ if row.PageNotificationId != nil {
+ if err := transactional.PageNotificationRepository.MarkFailed(tx, *row.PageNotificationId, errorMsg, now); err != nil {
+ return false, err
+ }
+ }
+ return true, nil
+ })
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to finalize outbox row %d: %w", row.Id, err))
+ return
+ }
+ if !finalized {
+ return
+ }
+ atomic.AddUint64(&terminalFailuresTotal, 1)
+ traceway.CaptureException(fmt.Errorf("outbox delivery %d (%s via %s) permanently failed after %d attempts: %s", row.Id, row.Kind, row.AdapterType, row.Attempts, errorMsg))
+ if terminalHook != nil {
+ terminalHook(row, models.OutboxFailed, errorMsg)
+ }
+}
+
+func backoffIndex(attempts int) int {
+ index := attempts - 1
+ if index < 0 {
+ index = 0
+ }
+ if index >= len(backoffSchedule) {
+ index = len(backoffSchedule) - 1
+ }
+ return index
+}
diff --git a/backend/app/outbox/health.go b/backend/app/outbox/health.go
new file mode 100644
index 00000000..8da5a9fb
--- /dev/null
+++ b/backend/app/outbox/health.go
@@ -0,0 +1,58 @@
+package outbox
+
+import (
+ "database/sql"
+ "sync/atomic"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+type HealthStats struct {
+ Pending int `json:"pending"`
+ Sending int `json:"sending"`
+ OldestPendingAgeSec int64 `json:"oldestPendingAgeSec"`
+ FailedRows int `json:"failedRows"`
+ SentTotal uint64 `json:"sentTotal"`
+ TerminalFailuresTotal uint64 `json:"terminalFailuresTotal"`
+}
+
+// HealthSnapshot powers /api/health/deep. OldestPendingAgeSec measures due-age
+// (from next_attempt_at), so a scheduled future delivery does not look stuck.
+func HealthSnapshot() (*HealthStats, error) {
+ type snapshot struct {
+ Counts *models.OutboxHealthCounts
+ Oldest *models.OutboxDelivery
+ }
+ loaded, err := db.ExecuteTransaction(func(tx *sql.Tx) (snapshot, error) {
+ counts, err := transactional.OutboxRepository.CountsForHealth(tx)
+ if err != nil {
+ return snapshot{}, err
+ }
+ oldest, err := transactional.OutboxRepository.OldestPending(tx)
+ if err != nil {
+ return snapshot{}, err
+ }
+ return snapshot{Counts: counts, Oldest: oldest}, nil
+ })
+ if err != nil {
+ return nil, err
+ }
+ stats := &HealthStats{
+ SentTotal: atomic.LoadUint64(&sentTotal),
+ TerminalFailuresTotal: atomic.LoadUint64(&terminalFailuresTotal),
+ }
+ if loaded.Counts != nil {
+ stats.Pending = loaded.Counts.PendingCount
+ stats.Sending = loaded.Counts.SendingCount
+ stats.FailedRows = loaded.Counts.FailedCount
+ }
+ if loaded.Oldest != nil {
+ if age := time.Since(loaded.Oldest.NextAttemptAt); age > 0 {
+ stats.OldestPendingAgeSec = int64(age.Seconds())
+ }
+ }
+ return stats, nil
+}
diff --git a/backend/app/outbox/outbox.go b/backend/app/outbox/outbox.go
new file mode 100644
index 00000000..965af372
--- /dev/null
+++ b/backend/app/outbox/outbox.go
@@ -0,0 +1,150 @@
+package outbox
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "strconv"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+// Delivery is one intended notification send. Enqueue persists it; the drain
+// worker delivers it with retries. AdapterConfig is a snapshot: it is never
+// re-derived at send time, so later channel/contact-method edits do not affect
+// rows already queued.
+type Delivery struct {
+ Kind string
+ AdapterType string
+ AdapterConfig json.RawMessage
+ Message models.NotificationMessage
+ NotBefore *time.Time
+ CancelKey string
+ PageNotificationId *int
+ RuleId *int
+ ProjectId *uuid.UUID
+ ChannelName string
+}
+
+// Enqueue inserts a pending outbox row in the caller's transaction. The commit
+// of that transaction is the durable "someone will be notified" promise.
+// Callers should Wake() after their transaction commits.
+func Enqueue(tx *sql.Tx, d Delivery) (int, error) {
+ messageJSON, err := json.Marshal(d.Message)
+ if err != nil {
+ return 0, err
+ }
+ now := time.Now().UTC()
+ next := now
+ if d.NotBefore != nil {
+ next = d.NotBefore.UTC()
+ }
+ cfg := d.AdapterConfig
+ if len(cfg) == 0 {
+ cfg = json.RawMessage("{}")
+ }
+ row := &models.OutboxDelivery{
+ Kind: d.Kind,
+ Status: models.OutboxPending,
+ AdapterType: d.AdapterType,
+ AdapterConfig: models.JSONText(cfg),
+ Message: models.JSONText(messageJSON),
+ NextAttemptAt: next,
+ CancelKey: d.CancelKey,
+ PageNotificationId: d.PageNotificationId,
+ RuleId: d.RuleId,
+ ProjectId: d.ProjectId,
+ ChannelName: d.ChannelName,
+ CreatedAt: now,
+ }
+ return transactional.OutboxRepository.Enqueue(tx, row)
+}
+
+// CancelByKey flips every pending/sending row under the key to cancelled and
+// mirrors their linked page_notifications rows. Runs in the caller's
+// transaction. An in-flight send may still deliver once (at-least-once), but
+// the row's final state stays cancelled: MarkSent/MarkFailedWithBackoff are
+// guarded on status = 'sending'.
+func CancelByKey(tx *sql.Tx, cancelKey string) error {
+ now := time.Now().UTC()
+ rows, err := transactional.OutboxRepository.FindCancellable(tx, cancelKey)
+ if err != nil {
+ return err
+ }
+ // notification_outbox first, matching the drain's finalize order; the
+ // reverse order can deadlock against a concurrent finalize on Postgres.
+ if err := transactional.OutboxRepository.CancelByKey(tx, cancelKey, now); err != nil {
+ return err
+ }
+ for _, row := range rows {
+ if row.PageNotificationId != nil {
+ if err := transactional.PageNotificationRepository.MarkCancelled(tx, *row.PageNotificationId, now); err != nil {
+ return err
+ }
+ }
+ }
+ return nil
+}
+
+func PageCancelKey(pageId int) string {
+ return "page:" + strconv.Itoa(pageId)
+}
+
+// CancelForProject cancels everything still queued for a project (which has no
+// foreign key from notification_outbox), in the caller's transaction.
+func CancelForProject(tx *sql.Tx, projectId uuid.UUID) error {
+ pages, err := transactional.PageRepository.FindByProject(tx, projectId, "", projectCancelPageLimit, 0)
+ if err != nil {
+ return err
+ }
+ for _, page := range pages {
+ if err := CancelByKey(tx, PageCancelKey(page.Id)); err != nil {
+ return err
+ }
+ }
+ return transactional.OutboxRepository.CancelByProject(tx, projectId, time.Now().UTC())
+}
+
+const projectCancelPageLimit = 5000
+
+// VerificationCancelKey scopes a contact method's verification sends so a new
+// code supersedes the one before it.
+func VerificationCancelKey(methodId int) string {
+ return "verify:" + strconv.Itoa(methodId)
+}
+
+var wakeCh = make(chan struct{}, 1)
+
+// Wake nudges the drain worker so a freshly enqueued delivery goes out
+// immediately instead of waiting for the next poll. Non-blocking.
+func Wake() {
+ select {
+ case wakeCh <- struct{}{}:
+ default:
+ }
+}
+
+// SendFunc performs one delivery attempt. Registered from cmd/run.go with the
+// notifications-package implementation; the indirection exists because this
+// package cannot import notifications (notifications imports it to enqueue).
+type SendFunc func(ctx context.Context, adapterType string, adapterConfig json.RawMessage, msg models.NotificationMessage) error
+
+var sender SendFunc
+
+func RegisterSender(fn SendFunc) {
+ sender = fn
+}
+
+// TerminalHook observes terminal outcomes (models.OutboxSent or
+// models.OutboxFailed) for audit bookkeeping. Called outside any transaction;
+// must not block.
+type TerminalHook func(row *models.OutboxDelivery, status string, errorMsg string)
+
+var terminalHook TerminalHook
+
+func RegisterTerminalHook(fn TerminalHook) {
+ terminalHook = fn
+}
diff --git a/backend/app/outbox/outbox_sqlite_test.go b/backend/app/outbox/outbox_sqlite_test.go
new file mode 100644
index 00000000..8f86392e
--- /dev/null
+++ b/backend/app/outbox/outbox_sqlite_test.go
@@ -0,0 +1,391 @@
+//go:build !transactional_pg && !telemetry_ch && !telemetry_duckdb
+
+package outbox
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "errors"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func setupOutboxDB(t *testing.T) {
+ t.Helper()
+ dbtest.SetupSQLite(t)
+ t.Cleanup(func() {
+ sender = nil
+ terminalHook = nil
+ })
+}
+
+type fakeSender struct {
+ mu sync.Mutex
+ calls []fakeCall
+ fail error
+}
+
+type fakeCall struct {
+ AdapterType string
+ Config string
+ Message models.NotificationMessage
+}
+
+func (f *fakeSender) fn(ctx context.Context, adapterType string, adapterConfig json.RawMessage, msg models.NotificationMessage) error {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.calls = append(f.calls, fakeCall{AdapterType: adapterType, Config: string(adapterConfig), Message: msg})
+ return f.fail
+}
+
+func (f *fakeSender) callCount() int {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return len(f.calls)
+}
+
+func enqueueTest(t *testing.T, d Delivery) int {
+ t.Helper()
+ id, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return Enqueue(tx, d)
+ })
+ if err != nil {
+ t.Fatalf("enqueue: %v", err)
+ }
+ return id
+}
+
+func reloadRow(t *testing.T, id int) *models.OutboxDelivery {
+ t.Helper()
+ row, err := db.ExecuteTransaction(func(tx *sql.Tx) (*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindById(tx, id)
+ })
+ if err != nil || row == nil {
+ t.Fatalf("reload outbox row %d: %v", id, err)
+ }
+ return row
+}
+
+func testMessage(subject string) models.NotificationMessage {
+ return models.NotificationMessage{Subject: subject, Body: "body", Severity: models.NotificationSeverityCritical}
+}
+
+func TestEnqueueSetsPendingAndDue(t *testing.T) {
+ setupOutboxDB(t)
+ now := time.Now().UTC()
+
+ immediate := enqueueTest(t, Delivery{Kind: models.OutboxKindRule, AdapterType: "slack", AdapterConfig: json.RawMessage(`{"webhookUrl":"x"}`), Message: testMessage("a")})
+ future := now.Add(30 * time.Minute)
+ scheduled := enqueueTest(t, Delivery{Kind: models.OutboxKindPage, AdapterType: "email", Message: testMessage("b"), NotBefore: &future})
+
+ due, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindDue(tx, now.Add(time.Second), 10)
+ })
+ if err != nil {
+ t.Fatalf("find due: %v", err)
+ }
+ if len(due) != 1 || due[0].Id != immediate {
+ t.Fatalf("expected only the immediate row due, got %+v", due)
+ }
+
+ dueAtFuture, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindDue(tx, future.Add(time.Second), 10)
+ })
+ if err != nil {
+ t.Fatalf("find due future: %v", err)
+ }
+ if len(dueAtFuture) != 2 {
+ t.Fatalf("expected both rows due at NotBefore, got %d", len(dueAtFuture))
+ }
+ if row := reloadRow(t, scheduled); row.Status != models.OutboxPending {
+ t.Errorf("scheduled row status = %s, want pending", row.Status)
+ }
+}
+
+func TestDrainTickSendsViaRegisteredSender(t *testing.T) {
+ setupOutboxDB(t)
+ fake := &fakeSender{}
+ RegisterSender(fake.fn)
+
+ id := enqueueTest(t, Delivery{Kind: models.OutboxKindRule, AdapterType: "slack", AdapterConfig: json.RawMessage(`{"webhookUrl":"https://example.com"}`), Message: testMessage("hello")})
+ DrainOnce(context.Background(), time.Now().UTC())
+
+ if fake.callCount() != 1 {
+ t.Fatalf("sender calls = %d, want 1", fake.callCount())
+ }
+ call := fake.calls[0]
+ if call.AdapterType != "slack" || call.Config != `{"webhookUrl":"https://example.com"}` || call.Message.Subject != "hello" {
+ t.Errorf("sender got %+v", call)
+ }
+ row := reloadRow(t, id)
+ if row.Status != models.OutboxSent || row.SentAt == nil {
+ t.Errorf("row after send = %s (sent_at %v), want sent", row.Status, row.SentAt)
+ }
+}
+
+func TestBackoffProgressionToTerminal(t *testing.T) {
+ setupOutboxDB(t)
+ fake := &fakeSender{fail: errors.New("boom")}
+ RegisterSender(fake.fn)
+ var hookStatus string
+ var hookCalls int
+ RegisterTerminalHook(func(row *models.OutboxDelivery, status string, errorMsg string) {
+ hookCalls++
+ hookStatus = status
+ })
+
+ id := enqueueTest(t, Delivery{Kind: models.OutboxKindRule, AdapterType: "slack", Message: testMessage("x")})
+
+ now := time.Now().UTC()
+ expectedDelays := []time.Duration{1 * time.Minute, 5 * time.Minute, 15 * time.Minute, 60 * time.Minute}
+ for attempt := 1; attempt <= 4; attempt++ {
+ DrainOnce(context.Background(), now)
+ row := reloadRow(t, id)
+ if row.Status != models.OutboxPending {
+ t.Fatalf("after attempt %d status = %s, want pending", attempt, row.Status)
+ }
+ if row.Attempts != attempt {
+ t.Fatalf("after attempt %d attempts = %d", attempt, row.Attempts)
+ }
+ delay := row.NextAttemptAt.Sub(time.Now().UTC())
+ want := expectedDelays[attempt-1]
+ if delay < want-time.Minute || delay > want+time.Minute {
+ t.Errorf("attempt %d backoff = %v, want ~%v", attempt, delay, want)
+ }
+ now = row.NextAttemptAt.Add(time.Second)
+ }
+
+ DrainOnce(context.Background(), now)
+ row := reloadRow(t, id)
+ if row.Status != models.OutboxFailed {
+ t.Fatalf("after attempt 5 status = %s, want failed", row.Status)
+ }
+ if row.LastError == "" {
+ t.Error("terminal row should keep last_error")
+ }
+ if hookCalls != 1 || hookStatus != models.OutboxFailed {
+ t.Errorf("terminal hook calls = %d status = %s, want 1 failed", hookCalls, hookStatus)
+ }
+}
+
+func TestCrashAfterClaimIsReclaimed(t *testing.T) {
+ setupOutboxDB(t)
+ fake := &fakeSender{}
+ RegisterSender(fake.fn)
+
+ id := enqueueTest(t, Delivery{Kind: models.OutboxKindRule, AdapterType: "slack", Message: testMessage("x")})
+
+ // Simulate a crash between claim-commit and send: claim without finalize.
+ now := time.Now().UTC()
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := transactional.OutboxRepository.MarkSending(tx, id, now)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatalf("mark sending: %v", err)
+ }
+
+ // Within the stale window nothing happens.
+ DrainOnce(context.Background(), now.Add(1*time.Minute))
+ if fake.callCount() != 0 {
+ t.Fatal("row should not be re-sent inside the stale-sending window")
+ }
+
+ // Past the window the row is reclaimed and delivered; attempts kept.
+ DrainOnce(context.Background(), now.Add(staleSendingAge+time.Second))
+ if fake.callCount() != 1 {
+ t.Fatalf("sender calls = %d, want 1 after reclaim", fake.callCount())
+ }
+ row := reloadRow(t, id)
+ if row.Status != models.OutboxSent {
+ t.Errorf("status = %s, want sent", row.Status)
+ }
+ if row.Attempts != 2 {
+ t.Errorf("attempts = %d, want 2 (claim before crash + reclaimed attempt)", row.Attempts)
+ }
+}
+
+// seedPage creates the minimal org -> project -> page chain so FK-constrained
+// page_notifications rows can exist.
+func seedPage(t *testing.T) int {
+ t.Helper()
+ pageId, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ org, err := transactional.OrganizationRepository.Create(tx, "Acme", "UTC")
+ if err != nil {
+ return 0, err
+ }
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "api", "gin", org.Id)
+ if err != nil {
+ return 0, err
+ }
+ now := time.Now().UTC()
+ return transactional.PageRepository.Create(tx, &models.Page{
+ OrganizationId: org.Id, ProjectId: project.Id,
+ PolicySnapshot: models.JSONText("{}"), Subject: "s", Status: models.PageStatusOpen,
+ DedupKey: "test", EventCount: 1, LastEventAt: now, EscalationLevel: -1,
+ CreatedAt: now, UpdatedAt: now,
+ })
+ })
+ if err != nil {
+ t.Fatalf("seed page: %v", err)
+ }
+ return pageId
+}
+
+func TestCancelByKeyCancelsPendingAndMirrors(t *testing.T) {
+ setupOutboxDB(t)
+ pageId := seedPage(t)
+
+ notificationId, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return transactional.PageNotificationRepository.Create(tx, &models.PageNotification{
+ PageId: pageId, Level: 0, TargetDesc: "x", MethodType: "email",
+ Status: models.PageNotificationPending, CreatedAt: time.Now().UTC(),
+ })
+ })
+ if err != nil {
+ t.Fatalf("create page notification: %v", err)
+ }
+ id := enqueueTest(t, Delivery{Kind: models.OutboxKindPage, AdapterType: "email", Message: testMessage("x"), CancelKey: PageCancelKey(pageId), PageNotificationId: ¬ificationId})
+ sentId := enqueueTest(t, Delivery{Kind: models.OutboxKindPage, AdapterType: "email", Message: testMessage("y"), CancelKey: PageCancelKey(pageId)})
+ now := time.Now().UTC()
+ _, err = db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ if _, err := transactional.OutboxRepository.MarkSending(tx, sentId, now); err != nil {
+ return struct{}{}, err
+ }
+ _, err := transactional.OutboxRepository.MarkSent(tx, sentId, now)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatalf("pre-send row: %v", err)
+ }
+
+ _, err = db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ return struct{}{}, CancelByKey(tx, PageCancelKey(1))
+ })
+ if err != nil {
+ t.Fatalf("cancel: %v", err)
+ }
+
+ if row := reloadRow(t, id); row.Status != models.OutboxCancelled {
+ t.Errorf("pending row status = %s, want cancelled", row.Status)
+ }
+ if row := reloadRow(t, sentId); row.Status != models.OutboxSent {
+ t.Errorf("already-sent row status = %s, want sent (untouched)", row.Status)
+ }
+ notifications, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.PageNotification, error) {
+ return transactional.PageNotificationRepository.FindByPage(tx, pageId)
+ })
+ if err != nil || len(notifications) != 1 {
+ t.Fatalf("load notifications: %v (%d)", err, len(notifications))
+ }
+ if notifications[0].Status != models.PageNotificationCancelled {
+ t.Errorf("mirrored notification status = %s, want cancelled", notifications[0].Status)
+ }
+}
+
+func TestCancelBeatsInFlightSend(t *testing.T) {
+ setupOutboxDB(t)
+ id := enqueueTest(t, Delivery{Kind: models.OutboxKindPage, AdapterType: "email", Message: testMessage("x"), CancelKey: PageCancelKey(2)})
+ now := time.Now().UTC()
+
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ _, err := transactional.OutboxRepository.MarkSending(tx, id, now)
+ return struct{}{}, err
+ })
+ if err != nil {
+ t.Fatalf("mark sending: %v", err)
+ }
+ _, err = db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ return struct{}{}, CancelByKey(tx, PageCancelKey(2))
+ })
+ if err != nil {
+ t.Fatalf("cancel: %v", err)
+ }
+ // The in-flight send finishes and tries to mark sent: the guard loses.
+ sent, err := db.ExecuteTransaction(func(tx *sql.Tx) (bool, error) {
+ return transactional.OutboxRepository.MarkSent(tx, id, now)
+ })
+ if err != nil {
+ t.Fatalf("mark sent: %v", err)
+ }
+ if sent {
+ t.Error("MarkSent reported a win against a cancelled row")
+ }
+ if row := reloadRow(t, id); row.Status != models.OutboxCancelled {
+ t.Errorf("status = %s, want cancelled to win the race", row.Status)
+ }
+}
+
+func TestLastEnqueuedPerRuleAndHealthAndPrune(t *testing.T) {
+ setupOutboxDB(t)
+ ruleId := 7
+ enqueueTest(t, Delivery{Kind: models.OutboxKindRule, AdapterType: "slack", Message: testMessage("x"), RuleId: &ruleId})
+
+ last, err := db.ExecuteTransaction(func(tx *sql.Tx) (map[int]time.Time, error) {
+ return transactional.OutboxRepository.LastEnqueuedPerRule(tx)
+ })
+ if err != nil {
+ t.Fatalf("last enqueued: %v", err)
+ }
+ if _, ok := last[ruleId]; !ok {
+ t.Errorf("expected rule %d in seeding map, got %v", ruleId, last)
+ }
+
+ stats, err := HealthSnapshot()
+ if err != nil {
+ t.Fatalf("health: %v", err)
+ }
+ if stats.Pending != 1 {
+ t.Errorf("health pending = %d, want 1", stats.Pending)
+ }
+
+ // Prune: sent row 8 days old pruned; failed row 8 days old kept; failed 31d pruned; pending never.
+ old := time.Now().UTC().AddDate(0, 0, -8)
+ veryOld := time.Now().UTC().AddDate(0, 0, -31)
+ mkRow := func(status string, createdAt time.Time) int {
+ id, err := db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
+ return transactional.OutboxRepository.Enqueue(tx, &models.OutboxDelivery{
+ Kind: models.OutboxKindRule, Status: status, AdapterType: "slack",
+ AdapterConfig: models.JSONText("{}"), Message: models.JSONText("{}"),
+ NextAttemptAt: createdAt, CreatedAt: createdAt,
+ })
+ })
+ if err != nil {
+ t.Fatalf("insert %s row: %v", status, err)
+ }
+ return id
+ }
+ sentOld := mkRow(models.OutboxSent, old)
+ failedOld := mkRow(models.OutboxFailed, old)
+ failedVeryOld := mkRow(models.OutboxFailed, veryOld)
+
+ pruned, err := db.ExecuteTransaction(func(tx *sql.Tx) (int64, error) {
+ now := time.Now().UTC()
+ return transactional.OutboxRepository.PruneTerminal(tx, now.AddDate(0, 0, -7), now.AddDate(0, 0, -30))
+ })
+ if err != nil {
+ t.Fatalf("prune: %v", err)
+ }
+ if pruned != 2 {
+ t.Errorf("pruned = %d, want 2 (old sent + very old failed)", pruned)
+ }
+ if row, _ := db.ExecuteTransaction(func(tx *sql.Tx) (*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindById(tx, failedOld)
+ }); row == nil {
+ t.Error("8-day-old failed row should be kept (30d retention)")
+ }
+ for _, gone := range []int{sentOld, failedVeryOld} {
+ if row, _ := db.ExecuteTransaction(func(tx *sql.Tx) (*models.OutboxDelivery, error) {
+ return transactional.OutboxRepository.FindById(tx, gone)
+ }); row != nil {
+ t.Errorf("row %d should be pruned", gone)
+ }
+ }
+}
diff --git a/backend/app/repositories/transactional/pg/escalation_policy.repository.go b/backend/app/repositories/transactional/pg/escalation_policy.repository.go
new file mode 100644
index 00000000..c49f5b12
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/escalation_policy.repository.go
@@ -0,0 +1,54 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type escalationPolicyRepository struct{}
+
+const escalationPolicyColumns = "id, organization_id, name, definition, created_by, created_at, updated_at"
+
+func (r *escalationPolicyRepository) FindById(tx *sql.Tx, id int) (*models.EscalationPolicy, error) {
+ return lit.SelectSingleNamed[models.EscalationPolicy](
+ tx,
+ "SELECT "+escalationPolicyColumns+" FROM escalation_policies WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *escalationPolicyRepository) FindByOrganization(tx *sql.Tx, organizationId int) ([]*models.EscalationPolicy, error) {
+ return lit.SelectNamed[models.EscalationPolicy](
+ tx,
+ "SELECT "+escalationPolicyColumns+" FROM escalation_policies WHERE organization_id = :organization_id ORDER BY name ASC, id ASC",
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *escalationPolicyRepository) FindByOrganizationAndName(tx *sql.Tx, organizationId int, name string) (*models.EscalationPolicy, error) {
+ return lit.SelectSingleNamed[models.EscalationPolicy](
+ tx,
+ "SELECT "+escalationPolicyColumns+" FROM escalation_policies WHERE organization_id = :organization_id AND LOWER(name) = LOWER(:name)",
+ lit.P{"organization_id": organizationId, "name": name},
+ )
+}
+
+func (r *escalationPolicyRepository) Create(tx *sql.Tx, policy *models.EscalationPolicy) (int, error) {
+ return lit.Insert[models.EscalationPolicy](tx, policy)
+}
+
+func (r *escalationPolicyRepository) Update(tx *sql.Tx, policy *models.EscalationPolicy) error {
+ return lit.UpdateNamed(tx, policy, "id = :id", lit.P{"id": policy.Id})
+}
+
+func (r *escalationPolicyRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM escalation_policies WHERE id = :id", lit.P{"id": id})
+}
+
+var EscalationPolicyRepository = escalationPolicyRepository{}
diff --git a/backend/app/repositories/transactional/pg/notification_channel.repository.go b/backend/app/repositories/transactional/pg/notification_channel.repository.go
index 1cf6d1d4..2b1097c0 100644
--- a/backend/app/repositories/transactional/pg/notification_channel.repository.go
+++ b/backend/app/repositories/transactional/pg/notification_channel.repository.go
@@ -22,6 +22,17 @@ func (r *notificationChannelRepository) FindByProject(tx *sql.Tx, projectId uuid
)
}
+// FindEscalationByOrganization returns every escalation channel across the
+// organization's projects in one query; callers match policy ids from the
+// config JSON in Go.
+func (r *notificationChannelRepository) FindEscalationByOrganization(tx *sql.Tx, organizationId int) ([]*models.NotificationChannel, error) {
+ return lit.SelectNamed[models.NotificationChannel](
+ tx,
+ "SELECT nc.id, nc.project_id, nc.name, nc.channel_type, nc.config, nc.enabled, nc.created_by, nc.created_at, nc.updated_at FROM notification_channels nc JOIN projects p ON p.id = nc.project_id WHERE p.organization_id = :organization_id AND nc.channel_type = 'escalation' ORDER BY nc.name ASC, nc.id ASC",
+ lit.P{"organization_id": organizationId},
+ )
+}
+
func (r *notificationChannelRepository) FindById(tx *sql.Tx, id int) (*models.NotificationChannel, error) {
return lit.SelectSingleNamed[models.NotificationChannel](
tx,
diff --git a/backend/app/repositories/transactional/pg/oncall_override.repository.go b/backend/app/repositories/transactional/pg/oncall_override.repository.go
new file mode 100644
index 00000000..d62712b7
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/oncall_override.repository.go
@@ -0,0 +1,69 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type oncallOverrideRepository struct{}
+
+const oncallOverrideColumns = "id, schedule_id, user_id, start_at, end_at, created_by, created_at"
+
+func (r *oncallOverrideRepository) FindById(tx *sql.Tx, id int) (*models.OncallOverride, error) {
+ return lit.SelectSingleNamed[models.OncallOverride](
+ tx,
+ "SELECT "+oncallOverrideColumns+" FROM oncall_overrides WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *oncallOverrideRepository) ListForRange(tx *sql.Tx, scheduleId int, from time.Time, to time.Time) ([]*models.OncallOverride, error) {
+ return lit.SelectNamed[models.OncallOverride](
+ tx,
+ `SELECT `+oncallOverrideColumns+` FROM oncall_overrides
+ WHERE schedule_id = :schedule_id AND end_at > :from AND start_at < :to
+ ORDER BY created_at ASC, id ASC`,
+ lit.P{"schedule_id": scheduleId, "from": from.UTC(), "to": to.UTC()},
+ )
+}
+
+// ListForRangeByOrganization returns every override intersecting [from, to)
+// across the organization's schedules in one query; callers group by
+// ScheduleId.
+func (r *oncallOverrideRepository) ListForRangeByOrganization(tx *sql.Tx, organizationId int, from time.Time, to time.Time) ([]*models.OncallOverride, error) {
+ return lit.SelectNamed[models.OncallOverride](
+ tx,
+ `SELECT `+oncallOverrideColumns+` FROM oncall_overrides
+ WHERE schedule_id IN (SELECT id FROM oncall_schedules WHERE organization_id = :organization_id) AND end_at > :from AND start_at < :to
+ ORDER BY created_at ASC, id ASC`,
+ lit.P{"organization_id": organizationId, "from": from.UTC(), "to": to.UTC()},
+ )
+}
+
+func (r *oncallOverrideRepository) Create(tx *sql.Tx, override *models.OncallOverride) (int, error) {
+ return lit.Insert[models.OncallOverride](tx, override)
+}
+
+func (r *oncallOverrideRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM oncall_overrides WHERE id = :id", lit.P{"id": id})
+}
+
+// DeleteByOrganizationAndUser removes a user's overrides across every schedule
+// in the organization; called when the user is removed from the organization.
+func (r *oncallOverrideRepository) DeleteByOrganizationAndUser(tx *sql.Tx, organizationId int, userId int) error {
+ return lit.DeleteNamed(
+ db.Driver,
+ tx,
+ "DELETE FROM oncall_overrides WHERE user_id = :user_id AND schedule_id IN (SELECT id FROM oncall_schedules WHERE organization_id = :organization_id)",
+ lit.P{"user_id": userId, "organization_id": organizationId},
+ )
+}
+
+var OncallOverrideRepository = oncallOverrideRepository{}
diff --git a/backend/app/repositories/transactional/pg/oncall_schedule.repository.go b/backend/app/repositories/transactional/pg/oncall_schedule.repository.go
new file mode 100644
index 00000000..e8a529b4
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/oncall_schedule.repository.go
@@ -0,0 +1,62 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type oncallScheduleRepository struct{}
+
+const oncallScheduleColumns = "id, organization_id, team_id, name, description, timezone, definition, created_by, created_at, updated_at"
+
+func (r *oncallScheduleRepository) FindById(tx *sql.Tx, id int) (*models.OncallSchedule, error) {
+ return lit.SelectSingleNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *oncallScheduleRepository) FindByOrganizationAndName(tx *sql.Tx, organizationId int, name string) (*models.OncallSchedule, error) {
+ return lit.SelectSingleNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE organization_id = :organization_id AND LOWER(name) = LOWER(:name)",
+ lit.P{"organization_id": organizationId, "name": name},
+ )
+}
+
+func (r *oncallScheduleRepository) ListByOrganization(tx *sql.Tx, organizationId int) ([]*models.OncallSchedule, error) {
+ return lit.SelectNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE organization_id = :organization_id ORDER BY name ASC, id ASC",
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *oncallScheduleRepository) ListByTeam(tx *sql.Tx, teamId int) ([]*models.OncallSchedule, error) {
+ return lit.SelectNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE team_id = :team_id ORDER BY created_at ASC, id ASC",
+ lit.P{"team_id": teamId},
+ )
+}
+
+func (r *oncallScheduleRepository) Create(tx *sql.Tx, schedule *models.OncallSchedule) (int, error) {
+ return lit.Insert[models.OncallSchedule](tx, schedule)
+}
+
+func (r *oncallScheduleRepository) Update(tx *sql.Tx, schedule *models.OncallSchedule) error {
+ return lit.UpdateNamed(tx, schedule, "id = :id", lit.P{"id": schedule.Id})
+}
+
+func (r *oncallScheduleRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM oncall_schedules WHERE id = :id", lit.P{"id": id})
+}
+
+var OncallScheduleRepository = oncallScheduleRepository{}
diff --git a/backend/app/repositories/transactional/pg/outbox.repository.go b/backend/app/repositories/transactional/pg/outbox.repository.go
new file mode 100644
index 00000000..e170d9c3
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/outbox.repository.go
@@ -0,0 +1,215 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type outboxRepository struct{}
+
+const outboxColumns = "id, kind, status, adapter_type, adapter_config, message, attempts, next_attempt_at, claimed_at, cancel_key, page_notification_id, rule_id, project_id, channel_name, last_error, created_at, sent_at"
+
+// Enqueue inserts a pending row in the caller's transaction and returns its id.
+// The commit of that transaction is the durable "someone will be notified"
+// promise.
+func (r *outboxRepository) Enqueue(tx *sql.Tx, row *models.OutboxDelivery) (int, error) {
+ return lit.Insert[models.OutboxDelivery](tx, row)
+}
+
+func (r *outboxRepository) FindById(tx *sql.Tx, id int) (*models.OutboxDelivery, error) {
+ return lit.SelectSingleNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+// FindDue returns pending rows whose next_attempt_at has passed: pages first,
+// then oldest first.
+func (r *outboxRepository) FindDue(tx *sql.Tx, now time.Time, limit int) ([]*models.OutboxDelivery, error) {
+ return lit.SelectNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE status = 'pending' AND next_attempt_at <= :now ORDER BY CASE WHEN kind = 'page' THEN 0 ELSE 1 END, next_attempt_at ASC, id ASC LIMIT :limit",
+ lit.P{"now": now.UTC(), "limit": limit},
+ )
+}
+
+// MarkSending claims one row: pending -> sending, attempts+1. The status guard
+// makes the claim lose against a concurrent cancel; returns whether the claim
+// won, and callers must not send when it lost.
+func (r *outboxRepository) MarkSending(tx *sql.Tx, id int, now time.Time) (bool, error) {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'sending', attempts = attempts + 1, claimed_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+}
+
+// MarkSent finalizes a delivered row. The status guard loses to a concurrent
+// cancel: a cancelled row stays cancelled even when the last send landed.
+// Returns whether the row was still sending.
+func (r *outboxRepository) MarkSent(tx *sql.Tx, id int, now time.Time) (bool, error) {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'sent', sent_at = :now, last_error = '' WHERE id = :id AND status = 'sending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+}
+
+// MarkFailedWithBackoff records a failed attempt. nextAttemptAt == nil is
+// terminal (status failed); otherwise the row returns to pending, scheduled at
+// nextAttemptAt. Guarded on status = 'sending' so cancel wins races; returns
+// whether the row was still sending.
+func (r *outboxRepository) MarkFailedWithBackoff(tx *sql.Tx, id int, errorMsg string, nextAttemptAt *time.Time, now time.Time) (bool, error) {
+ if nextAttemptAt == nil {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'failed', last_error = :error_msg, sent_at = :now WHERE id = :id AND status = 'sending'",
+ lit.P{"error_msg": errorMsg, "now": now.UTC(), "id": id},
+ )
+ }
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'pending', last_error = :error_msg, next_attempt_at = :next_attempt_at, claimed_at = NULL WHERE id = :id AND status = 'sending'",
+ lit.P{"error_msg": errorMsg, "next_attempt_at": nextAttemptAt.UTC(), "id": id},
+ )
+}
+
+// guardedStatusUpdate runs a status-guarded UPDATE and reports whether it
+// matched: zero rows means a concurrent transition (usually a cancel or a
+// lost ack/resolve race) won.
+func guardedStatusUpdate(tx *sql.Tx, namedQuery string, params lit.P) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(db.Driver, namedQuery, params)
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ if err != nil {
+ return false, err
+ }
+ return affected > 0, nil
+}
+
+// FindCancellable returns the pending/sending rows for a cancel key, so their
+// linked page_notifications can be mirrored before the status flip.
+func (r *outboxRepository) FindCancellable(tx *sql.Tx, cancelKey string) ([]*models.OutboxDelivery, error) {
+ return lit.SelectNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE cancel_key <> '' AND cancel_key = :cancel_key AND status IN ('pending', 'sending') ORDER BY id ASC",
+ lit.P{"cancel_key": cancelKey},
+ )
+}
+
+// CancelByKey flips every pending/sending row holding the key to cancelled.
+func (r *outboxRepository) CancelByKey(tx *sql.Tx, cancelKey string, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE notification_outbox SET status = 'cancelled', sent_at = :now WHERE cancel_key <> '' AND cancel_key = :cancel_key AND status IN ('pending', 'sending')",
+ lit.P{"now": now.UTC(), "cancel_key": cancelKey},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// CancelByProject cancels a deleted project's queued rule deliveries; page
+// rows carry no project id and are cancelled by key.
+func (r *outboxRepository) CancelByProject(tx *sql.Tx, projectId uuid.UUID, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE notification_outbox SET status = 'cancelled', sent_at = :now WHERE project_id = :project_id AND status IN ('pending', 'sending')",
+ lit.P{"now": now.UTC(), "project_id": projectId},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// ReclaimStaleSending returns rows claimed before the cutoff (a run died
+// between claim-commit and result-commit) to pending, due immediately.
+// Attempts are NOT reset, so crash loops still reach terminal failure.
+func (r *outboxRepository) ReclaimStaleSending(tx *sql.Tx, cutoff time.Time, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE notification_outbox SET status = 'pending', next_attempt_at = :now, claimed_at = NULL WHERE status = 'sending' AND claimed_at < :cutoff",
+ lit.P{"now": now.UTC(), "cutoff": cutoff.UTC()},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// LastEnqueuedPerRule backstops cooldown seeding at boot: the newest outbox
+// row per rule regardless of status (fired_notifications only exists once an
+// outcome is terminal). The max is folded in Go because SQLite loses column
+// type affinity on aggregates; the table is small (terminal rows are pruned).
+func (r *outboxRepository) LastEnqueuedPerRule(tx *sql.Tx) (map[int]time.Time, error) {
+ rows, err := lit.SelectNamed[models.OutboxRuleEnqueue](
+ tx,
+ "SELECT rule_id, created_at AS last_enqueued_at FROM notification_outbox WHERE rule_id IS NOT NULL",
+ lit.P{},
+ )
+ if err != nil {
+ return nil, err
+ }
+ result := make(map[int]time.Time, len(rows))
+ for _, row := range rows {
+ if existing, ok := result[row.RuleId]; !ok || row.LastEnqueuedAt.After(existing) {
+ result[row.RuleId] = row.LastEnqueuedAt
+ }
+ }
+ return result, nil
+}
+
+func (r *outboxRepository) CountsForHealth(tx *sql.Tx) (*models.OutboxHealthCounts, error) {
+ return lit.SelectSingleNamed[models.OutboxHealthCounts](
+ tx,
+ "SELECT COALESCE(SUM(CASE WHEN status = 'pending' THEN 1 ELSE 0 END), 0) AS pending_count, COALESCE(SUM(CASE WHEN status = 'sending' THEN 1 ELSE 0 END), 0) AS sending_count, COALESCE(SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END), 0) AS failed_count FROM notification_outbox",
+ lit.P{},
+ )
+}
+
+// OldestPending returns the pending row with the earliest next_attempt_at, or
+// nil. A plain-column select, because SQLite loses type affinity on
+// timestamp aggregates.
+func (r *outboxRepository) OldestPending(tx *sql.Tx) (*models.OutboxDelivery, error) {
+ return lit.SelectSingleNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE status = 'pending' ORDER BY next_attempt_at ASC, id ASC LIMIT 1",
+ lit.P{},
+ )
+}
+
+// PruneTerminal deletes finished rows past retention.
+func (r *outboxRepository) PruneTerminal(tx *sql.Tx, sentCutoff time.Time, failedCutoff time.Time) (int64, error) {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "DELETE FROM notification_outbox WHERE (status IN ('sent', 'cancelled') AND created_at < :sent_cutoff) OR (status = 'failed' AND created_at < :failed_cutoff)",
+ lit.P{"sent_cutoff": sentCutoff.UTC(), "failed_cutoff": failedCutoff.UTC()},
+ )
+ if err != nil {
+ return 0, err
+ }
+ res, err := tx.Exec(query, args...)
+ if err != nil {
+ return 0, err
+ }
+ return res.RowsAffected()
+}
+
+var OutboxRepository = outboxRepository{}
diff --git a/backend/app/repositories/transactional/pg/page.repository.go b/backend/app/repositories/transactional/pg/page.repository.go
new file mode 100644
index 00000000..4293e5ce
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/page.repository.go
@@ -0,0 +1,157 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type pageRepository struct{}
+
+const pageColumns = "id, organization_id, project_id, policy_id, policy_snapshot, rule_id, rule_name, rule_type, subject, body, url, severity, urgency, status, dedup_key, event_count, last_event_at, escalation_level, repeat_iteration, next_escalation_at, last_escalated_at, acknowledged_by, acknowledged_via, acknowledged_at, resolved_by, resolved_at, created_at, updated_at"
+
+func (r *pageRepository) FindById(tx *sql.Tx, id int) (*models.Page, error) {
+ return lit.SelectSingleNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *pageRepository) FindUnresolvedByDedupKey(tx *sql.Tx, dedupKey string) (*models.Page, error) {
+ return lit.SelectSingleNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE dedup_key = :dedup_key AND status <> 'resolved'",
+ lit.P{"dedup_key": dedupKey},
+ )
+}
+
+func (r *pageRepository) BumpEvent(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE pages SET event_count = event_count + 1, last_event_at = :now, updated_at = :now WHERE id = :id",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// FindDueById re-fetches one page only if it is still due, so a per-page claim
+// transaction can skip pages acknowledged, resolved, or claimed by a
+// concurrent escalator since the due list was read.
+func (r *pageRepository) FindDueById(tx *sql.Tx, id int, now time.Time) (*models.Page, error) {
+ return lit.SelectSingleNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE id = :id AND status = 'open' AND next_escalation_at IS NOT NULL AND next_escalation_at <= :now",
+ lit.P{"id": id, "now": now.UTC()},
+ )
+}
+
+// FindDue returns open pages whose next escalation is due, oldest first.
+func (r *pageRepository) FindDue(tx *sql.Tx, now time.Time) ([]*models.Page, error) {
+ return lit.SelectNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE status = 'open' AND next_escalation_at IS NOT NULL AND next_escalation_at <= :now ORDER BY next_escalation_at ASC, id ASC",
+ lit.P{"now": now.UTC()},
+ )
+}
+
+func (r *pageRepository) FindByProject(tx *sql.Tx, projectId uuid.UUID, status string, limit int, offset int) ([]*models.Page, error) {
+ return lit.SelectNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE project_id = :project_id AND ("+statusCondition(status)+") ORDER BY created_at DESC, id DESC LIMIT :limit OFFSET :offset",
+ lit.P{"project_id": projectId, "limit": limit, "offset": offset},
+ )
+}
+
+func (r *pageRepository) CountByProject(tx *sql.Tx, projectId uuid.UUID, status string) (int, error) {
+ result, err := lit.SelectSingleNamed[models.CountResult](
+ tx,
+ "SELECT COUNT(*) as count FROM pages WHERE project_id = :project_id AND ("+statusCondition(status)+")",
+ lit.P{"project_id": projectId},
+ )
+ if err != nil {
+ return 0, err
+ }
+ if result == nil {
+ return 0, nil
+ }
+ return result.Count, nil
+}
+
+func (r *pageRepository) CountOpenByProject(tx *sql.Tx, projectId uuid.UUID) (int, error) {
+ return r.CountByProject(tx, projectId, models.PageStatusOpen)
+}
+
+// statusCondition maps a status filter to a fixed SQL condition; values are
+// from a closed set, never user input.
+func statusCondition(status string) string {
+ switch status {
+ case models.PageStatusOpen:
+ return "status = 'open'"
+ case models.PageStatusAcknowledged:
+ return "status = 'acknowledged'"
+ case models.PageStatusResolved:
+ return "status = 'resolved'"
+ case "active":
+ return "status = 'open' OR status = 'acknowledged'"
+ default:
+ return "1 = 1"
+ }
+}
+
+func (r *pageRepository) Create(tx *sql.Tx, page *models.Page) (int, error) {
+ return lit.Insert[models.Page](tx, page)
+}
+
+// UpdateEscalationState advances the escalation clock. Guarded on
+// status = 'open' so a claim racing a concurrent acknowledge/resolve loses:
+// returns false when the page is no longer open, and the caller must roll the
+// claim back (its inserted deliveries were invisible to the ack's cancel).
+func (r *pageRepository) UpdateEscalationState(tx *sql.Tx, id int, level int, iteration int, nextEscalationAt *time.Time, now time.Time) (bool, error) {
+ var nextValue any
+ if nextEscalationAt != nil {
+ nextValue = nextEscalationAt.UTC()
+ }
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE pages SET escalation_level = :level, repeat_iteration = :iteration, next_escalation_at = :next_at, last_escalated_at = :now, updated_at = :now WHERE id = :id AND status = 'open'",
+ lit.P{"level": level, "iteration": iteration, "next_at": nextValue, "now": now.UTC(), "id": id},
+ )
+}
+
+// Acknowledge transitions open -> acknowledged. Returns false when the page
+// was not open (lost race or wrong state). userId is nil for anonymous
+// link-acks; via is 'dashboard' or 'link'.
+func (r *pageRepository) Acknowledge(tx *sql.Tx, id int, userId *int, via string, now time.Time) (bool, error) {
+ var userValue any
+ if userId != nil {
+ userValue = *userId
+ }
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE pages SET status = 'acknowledged', acknowledged_by = :user_id, acknowledged_via = :via, acknowledged_at = :now, next_escalation_at = NULL, updated_at = :now WHERE id = :id AND status = 'open'",
+ lit.P{"user_id": userValue, "via": via, "now": now.UTC(), "id": id},
+ )
+}
+
+// Resolve transitions open/acknowledged -> resolved. Returns false when the
+// page was already resolved.
+func (r *pageRepository) Resolve(tx *sql.Tx, id int, userId int, now time.Time) (bool, error) {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE pages SET status = 'resolved', resolved_by = :user_id, resolved_at = :now, next_escalation_at = NULL, updated_at = :now WHERE id = :id AND status <> 'resolved'",
+ lit.P{"user_id": userId, "now": now.UTC(), "id": id},
+ )
+}
+
+var PageRepository = pageRepository{}
diff --git a/backend/app/repositories/transactional/pg/page_notification.repository.go b/backend/app/repositories/transactional/pg/page_notification.repository.go
new file mode 100644
index 00000000..528a3c39
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/page_notification.repository.go
@@ -0,0 +1,85 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type pageNotificationRepository struct{}
+
+const pageNotificationColumns = "id, page_id, level, iteration, user_id, target_desc, method_type, status, error_msg, scheduled_for, ack_token_hash, created_at, sent_at"
+
+func (r *pageNotificationRepository) FindByPage(tx *sql.Tx, pageId int) ([]*models.PageNotification, error) {
+ return lit.SelectNamed[models.PageNotification](
+ tx,
+ "SELECT "+pageNotificationColumns+" FROM page_notifications WHERE page_id = :page_id ORDER BY created_at ASC, id ASC",
+ lit.P{"page_id": pageId},
+ )
+}
+
+// FindByAckTokenHash resolves a delivery ack token. The non-empty guard means
+// a row without a token (channel deliveries) can never match, even if a caller
+// ever hashes an empty input.
+func (r *pageNotificationRepository) FindByAckTokenHash(tx *sql.Tx, hash string) (*models.PageNotification, error) {
+ return lit.SelectSingleNamed[models.PageNotification](
+ tx,
+ "SELECT "+pageNotificationColumns+" FROM page_notifications WHERE ack_token_hash = :hash AND ack_token_hash <> ''",
+ lit.P{"hash": hash},
+ )
+}
+
+func (r *pageNotificationRepository) Create(tx *sql.Tx, notification *models.PageNotification) (int, error) {
+ return lit.Insert[models.PageNotification](tx, notification)
+}
+
+// MarkSent finalizes a delivered row. Guarded on status = 'pending' so a
+// terminal state (cancelled/failed) is never rewritten; the drain's mirror
+// call tolerates matching nothing.
+func (r *pageNotificationRepository) MarkSent(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE page_notifications SET status = 'sent', sent_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// MarkCancelled flips a not-yet-delivered row to cancelled; already-sent or
+// failed rows are left untouched.
+func (r *pageNotificationRepository) MarkCancelled(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE page_notifications SET status = 'cancelled', sent_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// MarkFailed records a terminal delivery failure. Guarded on
+// status = 'pending' so a cancelled row is never resurrected to failed.
+func (r *pageNotificationRepository) MarkFailed(tx *sql.Tx, id int, errorMsg string, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE page_notifications SET status = 'failed', error_msg = :error_msg, sent_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"error_msg": errorMsg, "now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+var PageNotificationRepository = pageNotificationRepository{}
diff --git a/backend/app/repositories/transactional/pg/team.repository.go b/backend/app/repositories/transactional/pg/team.repository.go
new file mode 100644
index 00000000..d71d1c6f
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/team.repository.go
@@ -0,0 +1,171 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type teamRepository struct{}
+
+const teamColumns = "id, organization_id, name, description, created_at, updated_at"
+
+func (r *teamRepository) FindById(tx *sql.Tx, id int) (*models.Team, error) {
+ return lit.SelectSingleNamed[models.Team](
+ tx,
+ "SELECT "+teamColumns+" FROM teams WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *teamRepository) FindByOrganizationAndName(tx *sql.Tx, organizationId int, name string) (*models.Team, error) {
+ return lit.SelectSingleNamed[models.Team](
+ tx,
+ "SELECT "+teamColumns+" FROM teams WHERE organization_id = :organization_id AND LOWER(name) = LOWER(:name)",
+ lit.P{"organization_id": organizationId, "name": name},
+ )
+}
+
+func (r *teamRepository) ListByOrganization(tx *sql.Tx, organizationId int) ([]*models.TeamWithCounts, error) {
+ return lit.SelectNamed[models.TeamWithCounts](
+ tx,
+ `SELECT t.id, t.organization_id, t.name, t.description, t.created_at, t.updated_at,
+ (SELECT COUNT(*) FROM team_members tm WHERE tm.team_id = t.id) as member_count,
+ (SELECT COUNT(*) FROM project_teams pt WHERE pt.team_id = t.id) as project_count,
+ (SELECT COUNT(*) FROM oncall_schedules s WHERE s.team_id = t.id) as schedule_count
+ FROM teams t
+ WHERE t.organization_id = :organization_id
+ ORDER BY t.name ASC, t.id ASC`,
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) Create(tx *sql.Tx, team *models.Team) (int, error) {
+ return lit.Insert[models.Team](tx, team)
+}
+
+func (r *teamRepository) Update(tx *sql.Tx, team *models.Team) error {
+ return lit.UpdateNamed(tx, team, "id = :id", lit.P{"id": team.Id})
+}
+
+func (r *teamRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM teams WHERE id = :id", lit.P{"id": id})
+}
+
+func (r *teamRepository) SetMembers(tx *sql.Tx, teamId int, orderedUserIds []int) error {
+ if err := lit.DeleteNamed(db.Driver, tx, "DELETE FROM team_members WHERE team_id = :team_id", lit.P{"team_id": teamId}); err != nil {
+ return err
+ }
+ now := time.Now().UTC()
+ for position, userId := range orderedUserIds {
+ member := &models.TeamMember{
+ TeamId: teamId,
+ UserId: userId,
+ Position: position,
+ CreatedAt: now,
+ }
+ if _, err := lit.Insert[models.TeamMember](tx, member); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (r *teamRepository) ListMembersWithUsersByOrganization(tx *sql.Tx, organizationId int) ([]*models.TeamMemberWithUser, error) {
+ return lit.SelectNamed[models.TeamMemberWithUser](
+ tx,
+ `SELECT tm.team_id, tm.user_id, tm.position, u.name, u.email
+ FROM team_members tm
+ JOIN teams t ON t.id = tm.team_id
+ JOIN users u ON u.id = tm.user_id
+ WHERE t.organization_id = :organization_id
+ ORDER BY tm.team_id ASC, tm.position ASC, tm.id ASC`,
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) FindMemberUserIds(tx *sql.Tx, teamId int) ([]int, error) {
+ members, err := lit.SelectNamed[models.TeamMember](
+ tx,
+ "SELECT id, team_id, user_id, position, created_at FROM team_members WHERE team_id = :team_id ORDER BY position ASC, id ASC",
+ lit.P{"team_id": teamId},
+ )
+ if err != nil {
+ return nil, err
+ }
+ userIds := make([]int, 0, len(members))
+ for _, member := range members {
+ userIds = append(userIds, member.UserId)
+ }
+ return userIds, nil
+}
+
+func (r *teamRepository) RemoveUserFromOrgTeams(tx *sql.Tx, organizationId int, userId int) error {
+ return lit.DeleteNamed(
+ db.Driver,
+ tx,
+ `DELETE FROM team_members
+ WHERE user_id = :user_id
+ AND team_id IN (SELECT id FROM teams WHERE organization_id = :organization_id)`,
+ lit.P{"user_id": userId, "organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) SetProjects(tx *sql.Tx, teamId int, projectIds []uuid.UUID) error {
+ if err := lit.DeleteNamed(db.Driver, tx, "DELETE FROM project_teams WHERE team_id = :team_id", lit.P{"team_id": teamId}); err != nil {
+ return err
+ }
+ now := time.Now().UTC()
+ for _, projectId := range projectIds {
+ link := &models.ProjectTeam{
+ ProjectId: projectId,
+ TeamId: teamId,
+ CreatedAt: now,
+ }
+ if _, err := lit.Insert[models.ProjectTeam](tx, link); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (r *teamRepository) ListProjectsByOrganization(tx *sql.Tx, organizationId int) ([]*models.TeamProjectRow, error) {
+ return lit.SelectNamed[models.TeamProjectRow](
+ tx,
+ `SELECT pt.team_id, pt.project_id, p.name
+ FROM project_teams pt
+ JOIN teams t ON t.id = pt.team_id
+ JOIN projects p ON p.id = pt.project_id
+ WHERE t.organization_id = :organization_id
+ ORDER BY pt.team_id ASC, p.name ASC`,
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) FindProjectTeam(tx *sql.Tx, projectId uuid.UUID) (*models.ProjectTeam, error) {
+ return lit.SelectSingleNamed[models.ProjectTeam](
+ tx,
+ "SELECT id, project_id, team_id, created_at FROM project_teams WHERE project_id = :project_id",
+ lit.P{"project_id": projectId},
+ )
+}
+
+func (r *teamRepository) FindTeamForProject(tx *sql.Tx, projectId uuid.UUID) (*models.Team, error) {
+ return lit.SelectSingleNamed[models.Team](
+ tx,
+ `SELECT t.id, t.organization_id, t.name, t.description, t.created_at, t.updated_at
+ FROM teams t
+ JOIN project_teams pt ON pt.team_id = t.id
+ WHERE pt.project_id = :project_id`,
+ lit.P{"project_id": projectId},
+ )
+}
+
+var TeamRepository = teamRepository{}
diff --git a/backend/app/repositories/transactional/pg/user_contact_method.repository.go b/backend/app/repositories/transactional/pg/user_contact_method.repository.go
new file mode 100644
index 00000000..c0107623
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/user_contact_method.repository.go
@@ -0,0 +1,106 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type userContactMethodRepository struct{}
+
+const userContactMethodColumns = "id, user_id, method_type, config, enabled, verified, verification_code_hash, verification_expires_at, verification_attempts, created_at"
+
+func (r *userContactMethodRepository) FindById(tx *sql.Tx, id int) (*models.UserContactMethod, error) {
+ return lit.SelectSingleNamed[models.UserContactMethod](
+ tx,
+ "SELECT "+userContactMethodColumns+" FROM user_contact_methods WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *userContactMethodRepository) FindByUser(tx *sql.Tx, userId int) ([]*models.UserContactMethod, error) {
+ return lit.SelectNamed[models.UserContactMethod](
+ tx,
+ "SELECT "+userContactMethodColumns+" FROM user_contact_methods WHERE user_id = :user_id ORDER BY created_at ASC, id ASC",
+ lit.P{"user_id": userId},
+ )
+}
+
+// FindEnabledByUser returns enabled AND verified methods: unverified numbers
+// are never paged.
+func (r *userContactMethodRepository) FindEnabledByUser(tx *sql.Tx, userId int) ([]*models.UserContactMethod, error) {
+ return lit.SelectNamed[models.UserContactMethod](
+ tx,
+ "SELECT "+userContactMethodColumns+" FROM user_contact_methods WHERE user_id = :user_id AND enabled = :enabled AND verified = :verified ORDER BY created_at ASC, id ASC",
+ lit.P{"user_id": userId, "enabled": true, "verified": true},
+ )
+}
+
+// SetVerification stores a fresh hashed code and flips the method back to
+// unverified with zero attempts.
+func (r *userContactMethodRepository) SetVerification(tx *sql.Tx, id int, codeHash string, expiresAt time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE user_contact_methods SET verified = :verified, verification_code_hash = :code_hash, verification_expires_at = :expires_at, verification_attempts = 0 WHERE id = :id",
+ lit.P{"verified": false, "code_hash": codeHash, "expires_at": expiresAt.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+func (r *userContactMethodRepository) MarkVerified(tx *sql.Tx, id int) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE user_contact_methods SET verified = :verified, verification_code_hash = '', verification_expires_at = NULL, verification_attempts = 0 WHERE id = :id",
+ lit.P{"verified": true, "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// IncrementVerificationAttempts consumes one verification attempt, guarded in
+// SQL so concurrent requests cannot exceed the cap (a check-then-increment in
+// Go would race). Returns false when the attempt budget is already spent.
+func (r *userContactMethodRepository) IncrementVerificationAttempts(tx *sql.Tx, id int, maxAttempts int) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE user_contact_methods SET verification_attempts = verification_attempts + 1 WHERE id = :id AND verification_attempts < :max_attempts",
+ lit.P{"id": id, "max_attempts": maxAttempts},
+ )
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ if err != nil {
+ return false, err
+ }
+ return affected > 0, nil
+}
+
+func (r *userContactMethodRepository) Create(tx *sql.Tx, method *models.UserContactMethod) (int, error) {
+ return lit.Insert[models.UserContactMethod](tx, method)
+}
+
+func (r *userContactMethodRepository) Update(tx *sql.Tx, method *models.UserContactMethod) error {
+ return lit.UpdateNamed(tx, method, "id = :id", lit.P{"id": method.Id})
+}
+
+func (r *userContactMethodRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM user_contact_methods WHERE id = :id", lit.P{"id": id})
+}
+
+var UserContactMethodRepository = userContactMethodRepository{}
diff --git a/backend/app/repositories/transactional/pg/user_notification_rule.repository.go b/backend/app/repositories/transactional/pg/user_notification_rule.repository.go
new file mode 100644
index 00000000..75520a3a
--- /dev/null
+++ b/backend/app/repositories/transactional/pg/user_notification_rule.repository.go
@@ -0,0 +1,48 @@
+//go:build transactional_pg
+
+package pg
+
+import (
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type userNotificationRuleRepository struct{}
+
+const userNotificationRuleColumns = "id, user_id, urgency, position, delay_minutes, contact_method_id, created_at"
+
+func (r *userNotificationRuleRepository) FindByUser(tx *sql.Tx, userId int) ([]*models.UserNotificationRule, error) {
+ return lit.SelectNamed[models.UserNotificationRule](
+ tx,
+ "SELECT "+userNotificationRuleColumns+" FROM user_notification_rules WHERE user_id = :user_id ORDER BY urgency ASC, position ASC, id ASC",
+ lit.P{"user_id": userId},
+ )
+}
+
+func (r *userNotificationRuleRepository) FindByUserAndUrgency(tx *sql.Tx, userId int, urgency string) ([]*models.UserNotificationRule, error) {
+ return lit.SelectNamed[models.UserNotificationRule](
+ tx,
+ "SELECT "+userNotificationRuleColumns+" FROM user_notification_rules WHERE user_id = :user_id AND urgency = :urgency ORDER BY position ASC, id ASC",
+ lit.P{"user_id": userId, "urgency": urgency},
+ )
+}
+
+// ReplaceForUser swaps the user's entire rule set in one transaction, so
+// positions are correct by construction.
+func (r *userNotificationRuleRepository) ReplaceForUser(tx *sql.Tx, userId int, rules []*models.UserNotificationRule) error {
+ if err := lit.DeleteNamed(db.Driver, tx, "DELETE FROM user_notification_rules WHERE user_id = :user_id", lit.P{"user_id": userId}); err != nil {
+ return err
+ }
+ for _, rule := range rules {
+ if _, err := lit.Insert[models.UserNotificationRule](tx, rule); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+var UserNotificationRuleRepository = userNotificationRuleRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/escalation_policy.repository.go b/backend/app/repositories/transactional/sqlite/escalation_policy.repository.go
new file mode 100644
index 00000000..104e161a
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/escalation_policy.repository.go
@@ -0,0 +1,54 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type escalationPolicyRepository struct{}
+
+const escalationPolicyColumns = "id, organization_id, name, definition, created_by, created_at, updated_at"
+
+func (r *escalationPolicyRepository) FindById(tx *sql.Tx, id int) (*models.EscalationPolicy, error) {
+ return lit.SelectSingleNamed[models.EscalationPolicy](
+ tx,
+ "SELECT "+escalationPolicyColumns+" FROM escalation_policies WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *escalationPolicyRepository) FindByOrganization(tx *sql.Tx, organizationId int) ([]*models.EscalationPolicy, error) {
+ return lit.SelectNamed[models.EscalationPolicy](
+ tx,
+ "SELECT "+escalationPolicyColumns+" FROM escalation_policies WHERE organization_id = :organization_id ORDER BY name ASC, id ASC",
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *escalationPolicyRepository) FindByOrganizationAndName(tx *sql.Tx, organizationId int, name string) (*models.EscalationPolicy, error) {
+ return lit.SelectSingleNamed[models.EscalationPolicy](
+ tx,
+ "SELECT "+escalationPolicyColumns+" FROM escalation_policies WHERE organization_id = :organization_id AND LOWER(name) = LOWER(:name)",
+ lit.P{"organization_id": organizationId, "name": name},
+ )
+}
+
+func (r *escalationPolicyRepository) Create(tx *sql.Tx, policy *models.EscalationPolicy) (int, error) {
+ return lit.Insert[models.EscalationPolicy](tx, policy)
+}
+
+func (r *escalationPolicyRepository) Update(tx *sql.Tx, policy *models.EscalationPolicy) error {
+ return lit.UpdateNamed(tx, policy, "id = :id", lit.P{"id": policy.Id})
+}
+
+func (r *escalationPolicyRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM escalation_policies WHERE id = :id", lit.P{"id": id})
+}
+
+var EscalationPolicyRepository = escalationPolicyRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/notification_channel.repository.go b/backend/app/repositories/transactional/sqlite/notification_channel.repository.go
index 50e2ee99..44fdd7fb 100644
--- a/backend/app/repositories/transactional/sqlite/notification_channel.repository.go
+++ b/backend/app/repositories/transactional/sqlite/notification_channel.repository.go
@@ -22,6 +22,17 @@ func (r *notificationChannelRepository) FindByProject(tx *sql.Tx, projectId uuid
)
}
+// FindEscalationByOrganization returns every escalation channel across the
+// organization's projects in one query; callers match policy ids from the
+// config JSON in Go.
+func (r *notificationChannelRepository) FindEscalationByOrganization(tx *sql.Tx, organizationId int) ([]*models.NotificationChannel, error) {
+ return lit.SelectNamed[models.NotificationChannel](
+ tx,
+ "SELECT nc.id, nc.project_id, nc.name, nc.channel_type, nc.config, nc.enabled, nc.created_by, nc.created_at, nc.updated_at FROM notification_channels nc JOIN projects p ON p.id = nc.project_id WHERE p.organization_id = :organization_id AND nc.channel_type = 'escalation' ORDER BY nc.name ASC, nc.id ASC",
+ lit.P{"organization_id": organizationId},
+ )
+}
+
func (r *notificationChannelRepository) FindById(tx *sql.Tx, id int) (*models.NotificationChannel, error) {
return lit.SelectSingleNamed[models.NotificationChannel](
tx,
diff --git a/backend/app/repositories/transactional/sqlite/oncall_override.repository.go b/backend/app/repositories/transactional/sqlite/oncall_override.repository.go
new file mode 100644
index 00000000..8695b677
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/oncall_override.repository.go
@@ -0,0 +1,69 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type oncallOverrideRepository struct{}
+
+const oncallOverrideColumns = "id, schedule_id, user_id, start_at, end_at, created_by, created_at"
+
+func (r *oncallOverrideRepository) FindById(tx *sql.Tx, id int) (*models.OncallOverride, error) {
+ return lit.SelectSingleNamed[models.OncallOverride](
+ tx,
+ "SELECT "+oncallOverrideColumns+" FROM oncall_overrides WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *oncallOverrideRepository) ListForRange(tx *sql.Tx, scheduleId int, from time.Time, to time.Time) ([]*models.OncallOverride, error) {
+ return lit.SelectNamed[models.OncallOverride](
+ tx,
+ `SELECT `+oncallOverrideColumns+` FROM oncall_overrides
+ WHERE schedule_id = :schedule_id AND end_at > :from AND start_at < :to
+ ORDER BY created_at ASC, id ASC`,
+ lit.P{"schedule_id": scheduleId, "from": from.UTC(), "to": to.UTC()},
+ )
+}
+
+// ListForRangeByOrganization returns every override intersecting [from, to)
+// across the organization's schedules in one query; callers group by
+// ScheduleId.
+func (r *oncallOverrideRepository) ListForRangeByOrganization(tx *sql.Tx, organizationId int, from time.Time, to time.Time) ([]*models.OncallOverride, error) {
+ return lit.SelectNamed[models.OncallOverride](
+ tx,
+ `SELECT `+oncallOverrideColumns+` FROM oncall_overrides
+ WHERE schedule_id IN (SELECT id FROM oncall_schedules WHERE organization_id = :organization_id) AND end_at > :from AND start_at < :to
+ ORDER BY created_at ASC, id ASC`,
+ lit.P{"organization_id": organizationId, "from": from.UTC(), "to": to.UTC()},
+ )
+}
+
+func (r *oncallOverrideRepository) Create(tx *sql.Tx, override *models.OncallOverride) (int, error) {
+ return lit.Insert[models.OncallOverride](tx, override)
+}
+
+func (r *oncallOverrideRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM oncall_overrides WHERE id = :id", lit.P{"id": id})
+}
+
+// DeleteByOrganizationAndUser removes a user's overrides across every schedule
+// in the organization; called when the user is removed from the organization.
+func (r *oncallOverrideRepository) DeleteByOrganizationAndUser(tx *sql.Tx, organizationId int, userId int) error {
+ return lit.DeleteNamed(
+ db.Driver,
+ tx,
+ "DELETE FROM oncall_overrides WHERE user_id = :user_id AND schedule_id IN (SELECT id FROM oncall_schedules WHERE organization_id = :organization_id)",
+ lit.P{"user_id": userId, "organization_id": organizationId},
+ )
+}
+
+var OncallOverrideRepository = oncallOverrideRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/oncall_schedule.repository.go b/backend/app/repositories/transactional/sqlite/oncall_schedule.repository.go
new file mode 100644
index 00000000..81dc3b59
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/oncall_schedule.repository.go
@@ -0,0 +1,62 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type oncallScheduleRepository struct{}
+
+const oncallScheduleColumns = "id, organization_id, team_id, name, description, timezone, definition, created_by, created_at, updated_at"
+
+func (r *oncallScheduleRepository) FindById(tx *sql.Tx, id int) (*models.OncallSchedule, error) {
+ return lit.SelectSingleNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *oncallScheduleRepository) FindByOrganizationAndName(tx *sql.Tx, organizationId int, name string) (*models.OncallSchedule, error) {
+ return lit.SelectSingleNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE organization_id = :organization_id AND LOWER(name) = LOWER(:name)",
+ lit.P{"organization_id": organizationId, "name": name},
+ )
+}
+
+func (r *oncallScheduleRepository) ListByOrganization(tx *sql.Tx, organizationId int) ([]*models.OncallSchedule, error) {
+ return lit.SelectNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE organization_id = :organization_id ORDER BY name ASC, id ASC",
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *oncallScheduleRepository) ListByTeam(tx *sql.Tx, teamId int) ([]*models.OncallSchedule, error) {
+ return lit.SelectNamed[models.OncallSchedule](
+ tx,
+ "SELECT "+oncallScheduleColumns+" FROM oncall_schedules WHERE team_id = :team_id ORDER BY created_at ASC, id ASC",
+ lit.P{"team_id": teamId},
+ )
+}
+
+func (r *oncallScheduleRepository) Create(tx *sql.Tx, schedule *models.OncallSchedule) (int, error) {
+ return lit.Insert[models.OncallSchedule](tx, schedule)
+}
+
+func (r *oncallScheduleRepository) Update(tx *sql.Tx, schedule *models.OncallSchedule) error {
+ return lit.UpdateNamed(tx, schedule, "id = :id", lit.P{"id": schedule.Id})
+}
+
+func (r *oncallScheduleRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM oncall_schedules WHERE id = :id", lit.P{"id": id})
+}
+
+var OncallScheduleRepository = oncallScheduleRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/outbox.repository.go b/backend/app/repositories/transactional/sqlite/outbox.repository.go
new file mode 100644
index 00000000..17001925
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/outbox.repository.go
@@ -0,0 +1,215 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type outboxRepository struct{}
+
+const outboxColumns = "id, kind, status, adapter_type, adapter_config, message, attempts, next_attempt_at, claimed_at, cancel_key, page_notification_id, rule_id, project_id, channel_name, last_error, created_at, sent_at"
+
+// Enqueue inserts a pending row in the caller's transaction and returns its id.
+// The commit of that transaction is the durable "someone will be notified"
+// promise.
+func (r *outboxRepository) Enqueue(tx *sql.Tx, row *models.OutboxDelivery) (int, error) {
+ return lit.Insert[models.OutboxDelivery](tx, row)
+}
+
+func (r *outboxRepository) FindById(tx *sql.Tx, id int) (*models.OutboxDelivery, error) {
+ return lit.SelectSingleNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+// FindDue returns pending rows whose next_attempt_at has passed: pages first,
+// then oldest first.
+func (r *outboxRepository) FindDue(tx *sql.Tx, now time.Time, limit int) ([]*models.OutboxDelivery, error) {
+ return lit.SelectNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE status = 'pending' AND next_attempt_at <= :now ORDER BY CASE WHEN kind = 'page' THEN 0 ELSE 1 END, next_attempt_at ASC, id ASC LIMIT :limit",
+ lit.P{"now": now.UTC(), "limit": limit},
+ )
+}
+
+// MarkSending claims one row: pending -> sending, attempts+1. The status guard
+// makes the claim lose against a concurrent cancel; returns whether the claim
+// won, and callers must not send when it lost.
+func (r *outboxRepository) MarkSending(tx *sql.Tx, id int, now time.Time) (bool, error) {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'sending', attempts = attempts + 1, claimed_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+}
+
+// MarkSent finalizes a delivered row. The status guard loses to a concurrent
+// cancel: a cancelled row stays cancelled even when the last send landed.
+// Returns whether the row was still sending.
+func (r *outboxRepository) MarkSent(tx *sql.Tx, id int, now time.Time) (bool, error) {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'sent', sent_at = :now, last_error = '' WHERE id = :id AND status = 'sending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+}
+
+// MarkFailedWithBackoff records a failed attempt. nextAttemptAt == nil is
+// terminal (status failed); otherwise the row returns to pending, scheduled at
+// nextAttemptAt. Guarded on status = 'sending' so cancel wins races; returns
+// whether the row was still sending.
+func (r *outboxRepository) MarkFailedWithBackoff(tx *sql.Tx, id int, errorMsg string, nextAttemptAt *time.Time, now time.Time) (bool, error) {
+ if nextAttemptAt == nil {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'failed', last_error = :error_msg, sent_at = :now WHERE id = :id AND status = 'sending'",
+ lit.P{"error_msg": errorMsg, "now": now.UTC(), "id": id},
+ )
+ }
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE notification_outbox SET status = 'pending', last_error = :error_msg, next_attempt_at = :next_attempt_at, claimed_at = NULL WHERE id = :id AND status = 'sending'",
+ lit.P{"error_msg": errorMsg, "next_attempt_at": nextAttemptAt.UTC(), "id": id},
+ )
+}
+
+// guardedStatusUpdate runs a status-guarded UPDATE and reports whether it
+// matched: zero rows means a concurrent transition (usually a cancel or a
+// lost ack/resolve race) won.
+func guardedStatusUpdate(tx *sql.Tx, namedQuery string, params lit.P) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(db.Driver, namedQuery, params)
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ if err != nil {
+ return false, err
+ }
+ return affected > 0, nil
+}
+
+// FindCancellable returns the pending/sending rows for a cancel key, so their
+// linked page_notifications can be mirrored before the status flip.
+func (r *outboxRepository) FindCancellable(tx *sql.Tx, cancelKey string) ([]*models.OutboxDelivery, error) {
+ return lit.SelectNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE cancel_key <> '' AND cancel_key = :cancel_key AND status IN ('pending', 'sending') ORDER BY id ASC",
+ lit.P{"cancel_key": cancelKey},
+ )
+}
+
+// CancelByKey flips every pending/sending row holding the key to cancelled.
+func (r *outboxRepository) CancelByKey(tx *sql.Tx, cancelKey string, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE notification_outbox SET status = 'cancelled', sent_at = :now WHERE cancel_key <> '' AND cancel_key = :cancel_key AND status IN ('pending', 'sending')",
+ lit.P{"now": now.UTC(), "cancel_key": cancelKey},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// CancelByProject cancels a deleted project's queued rule deliveries; page
+// rows carry no project id and are cancelled by key.
+func (r *outboxRepository) CancelByProject(tx *sql.Tx, projectId uuid.UUID, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE notification_outbox SET status = 'cancelled', sent_at = :now WHERE project_id = :project_id AND status IN ('pending', 'sending')",
+ lit.P{"now": now.UTC(), "project_id": projectId},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// ReclaimStaleSending returns rows claimed before the cutoff (a run died
+// between claim-commit and result-commit) to pending, due immediately.
+// Attempts are NOT reset, so crash loops still reach terminal failure.
+func (r *outboxRepository) ReclaimStaleSending(tx *sql.Tx, cutoff time.Time, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE notification_outbox SET status = 'pending', next_attempt_at = :now, claimed_at = NULL WHERE status = 'sending' AND claimed_at < :cutoff",
+ lit.P{"now": now.UTC(), "cutoff": cutoff.UTC()},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// LastEnqueuedPerRule backstops cooldown seeding at boot: the newest outbox
+// row per rule regardless of status (fired_notifications only exists once an
+// outcome is terminal). The max is folded in Go because SQLite loses column
+// type affinity on aggregates; the table is small (terminal rows are pruned).
+func (r *outboxRepository) LastEnqueuedPerRule(tx *sql.Tx) (map[int]time.Time, error) {
+ rows, err := lit.SelectNamed[models.OutboxRuleEnqueue](
+ tx,
+ "SELECT rule_id, created_at AS last_enqueued_at FROM notification_outbox WHERE rule_id IS NOT NULL",
+ lit.P{},
+ )
+ if err != nil {
+ return nil, err
+ }
+ result := make(map[int]time.Time, len(rows))
+ for _, row := range rows {
+ if existing, ok := result[row.RuleId]; !ok || row.LastEnqueuedAt.After(existing) {
+ result[row.RuleId] = row.LastEnqueuedAt
+ }
+ }
+ return result, nil
+}
+
+func (r *outboxRepository) CountsForHealth(tx *sql.Tx) (*models.OutboxHealthCounts, error) {
+ return lit.SelectSingleNamed[models.OutboxHealthCounts](
+ tx,
+ "SELECT COALESCE(SUM(CASE WHEN status = 'pending' THEN 1 ELSE 0 END), 0) AS pending_count, COALESCE(SUM(CASE WHEN status = 'sending' THEN 1 ELSE 0 END), 0) AS sending_count, COALESCE(SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END), 0) AS failed_count FROM notification_outbox",
+ lit.P{},
+ )
+}
+
+// OldestPending returns the pending row with the earliest next_attempt_at, or
+// nil. A plain-column select, because SQLite loses type affinity on
+// timestamp aggregates.
+func (r *outboxRepository) OldestPending(tx *sql.Tx) (*models.OutboxDelivery, error) {
+ return lit.SelectSingleNamed[models.OutboxDelivery](
+ tx,
+ "SELECT "+outboxColumns+" FROM notification_outbox WHERE status = 'pending' ORDER BY next_attempt_at ASC, id ASC LIMIT 1",
+ lit.P{},
+ )
+}
+
+// PruneTerminal deletes finished rows past retention.
+func (r *outboxRepository) PruneTerminal(tx *sql.Tx, sentCutoff time.Time, failedCutoff time.Time) (int64, error) {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "DELETE FROM notification_outbox WHERE (status IN ('sent', 'cancelled') AND created_at < :sent_cutoff) OR (status = 'failed' AND created_at < :failed_cutoff)",
+ lit.P{"sent_cutoff": sentCutoff.UTC(), "failed_cutoff": failedCutoff.UTC()},
+ )
+ if err != nil {
+ return 0, err
+ }
+ res, err := tx.Exec(query, args...)
+ if err != nil {
+ return 0, err
+ }
+ return res.RowsAffected()
+}
+
+var OutboxRepository = outboxRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/page.repository.go b/backend/app/repositories/transactional/sqlite/page.repository.go
new file mode 100644
index 00000000..ac3ea207
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/page.repository.go
@@ -0,0 +1,157 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type pageRepository struct{}
+
+const pageColumns = "id, organization_id, project_id, policy_id, policy_snapshot, rule_id, rule_name, rule_type, subject, body, url, severity, urgency, status, dedup_key, event_count, last_event_at, escalation_level, repeat_iteration, next_escalation_at, last_escalated_at, acknowledged_by, acknowledged_via, acknowledged_at, resolved_by, resolved_at, created_at, updated_at"
+
+func (r *pageRepository) FindById(tx *sql.Tx, id int) (*models.Page, error) {
+ return lit.SelectSingleNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *pageRepository) FindUnresolvedByDedupKey(tx *sql.Tx, dedupKey string) (*models.Page, error) {
+ return lit.SelectSingleNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE dedup_key = :dedup_key AND status <> 'resolved'",
+ lit.P{"dedup_key": dedupKey},
+ )
+}
+
+func (r *pageRepository) BumpEvent(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE pages SET event_count = event_count + 1, last_event_at = :now, updated_at = :now WHERE id = :id",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// FindDueById re-fetches one page only if it is still due, so a per-page claim
+// transaction can skip pages acknowledged, resolved, or claimed by a
+// concurrent escalator since the due list was read.
+func (r *pageRepository) FindDueById(tx *sql.Tx, id int, now time.Time) (*models.Page, error) {
+ return lit.SelectSingleNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE id = :id AND status = 'open' AND next_escalation_at IS NOT NULL AND next_escalation_at <= :now",
+ lit.P{"id": id, "now": now.UTC()},
+ )
+}
+
+// FindDue returns open pages whose next escalation is due, oldest first.
+func (r *pageRepository) FindDue(tx *sql.Tx, now time.Time) ([]*models.Page, error) {
+ return lit.SelectNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE status = 'open' AND next_escalation_at IS NOT NULL AND next_escalation_at <= :now ORDER BY next_escalation_at ASC, id ASC",
+ lit.P{"now": now.UTC()},
+ )
+}
+
+func (r *pageRepository) FindByProject(tx *sql.Tx, projectId uuid.UUID, status string, limit int, offset int) ([]*models.Page, error) {
+ return lit.SelectNamed[models.Page](
+ tx,
+ "SELECT "+pageColumns+" FROM pages WHERE project_id = :project_id AND ("+statusCondition(status)+") ORDER BY created_at DESC, id DESC LIMIT :limit OFFSET :offset",
+ lit.P{"project_id": projectId, "limit": limit, "offset": offset},
+ )
+}
+
+func (r *pageRepository) CountByProject(tx *sql.Tx, projectId uuid.UUID, status string) (int, error) {
+ result, err := lit.SelectSingleNamed[models.CountResult](
+ tx,
+ "SELECT COUNT(*) as count FROM pages WHERE project_id = :project_id AND ("+statusCondition(status)+")",
+ lit.P{"project_id": projectId},
+ )
+ if err != nil {
+ return 0, err
+ }
+ if result == nil {
+ return 0, nil
+ }
+ return result.Count, nil
+}
+
+func (r *pageRepository) CountOpenByProject(tx *sql.Tx, projectId uuid.UUID) (int, error) {
+ return r.CountByProject(tx, projectId, models.PageStatusOpen)
+}
+
+// statusCondition maps a status filter to a fixed SQL condition; values are
+// from a closed set, never user input.
+func statusCondition(status string) string {
+ switch status {
+ case models.PageStatusOpen:
+ return "status = 'open'"
+ case models.PageStatusAcknowledged:
+ return "status = 'acknowledged'"
+ case models.PageStatusResolved:
+ return "status = 'resolved'"
+ case "active":
+ return "status = 'open' OR status = 'acknowledged'"
+ default:
+ return "1 = 1"
+ }
+}
+
+func (r *pageRepository) Create(tx *sql.Tx, page *models.Page) (int, error) {
+ return lit.Insert[models.Page](tx, page)
+}
+
+// UpdateEscalationState advances the escalation clock. Guarded on
+// status = 'open' so a claim racing a concurrent acknowledge/resolve loses:
+// returns false when the page is no longer open, and the caller must roll the
+// claim back (its inserted deliveries were invisible to the ack's cancel).
+func (r *pageRepository) UpdateEscalationState(tx *sql.Tx, id int, level int, iteration int, nextEscalationAt *time.Time, now time.Time) (bool, error) {
+ var nextValue any
+ if nextEscalationAt != nil {
+ nextValue = nextEscalationAt.UTC()
+ }
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE pages SET escalation_level = :level, repeat_iteration = :iteration, next_escalation_at = :next_at, last_escalated_at = :now, updated_at = :now WHERE id = :id AND status = 'open'",
+ lit.P{"level": level, "iteration": iteration, "next_at": nextValue, "now": now.UTC(), "id": id},
+ )
+}
+
+// Acknowledge transitions open -> acknowledged. Returns false when the page
+// was not open (lost race or wrong state). userId is nil for anonymous
+// link-acks; via is 'dashboard' or 'link'.
+func (r *pageRepository) Acknowledge(tx *sql.Tx, id int, userId *int, via string, now time.Time) (bool, error) {
+ var userValue any
+ if userId != nil {
+ userValue = *userId
+ }
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE pages SET status = 'acknowledged', acknowledged_by = :user_id, acknowledged_via = :via, acknowledged_at = :now, next_escalation_at = NULL, updated_at = :now WHERE id = :id AND status = 'open'",
+ lit.P{"user_id": userValue, "via": via, "now": now.UTC(), "id": id},
+ )
+}
+
+// Resolve transitions open/acknowledged -> resolved. Returns false when the
+// page was already resolved.
+func (r *pageRepository) Resolve(tx *sql.Tx, id int, userId int, now time.Time) (bool, error) {
+ return guardedStatusUpdate(
+ tx,
+ "UPDATE pages SET status = 'resolved', resolved_by = :user_id, resolved_at = :now, next_escalation_at = NULL, updated_at = :now WHERE id = :id AND status <> 'resolved'",
+ lit.P{"user_id": userId, "now": now.UTC(), "id": id},
+ )
+}
+
+var PageRepository = pageRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/page_notification.repository.go b/backend/app/repositories/transactional/sqlite/page_notification.repository.go
new file mode 100644
index 00000000..939d9705
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/page_notification.repository.go
@@ -0,0 +1,85 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type pageNotificationRepository struct{}
+
+const pageNotificationColumns = "id, page_id, level, iteration, user_id, target_desc, method_type, status, error_msg, scheduled_for, ack_token_hash, created_at, sent_at"
+
+func (r *pageNotificationRepository) FindByPage(tx *sql.Tx, pageId int) ([]*models.PageNotification, error) {
+ return lit.SelectNamed[models.PageNotification](
+ tx,
+ "SELECT "+pageNotificationColumns+" FROM page_notifications WHERE page_id = :page_id ORDER BY created_at ASC, id ASC",
+ lit.P{"page_id": pageId},
+ )
+}
+
+// FindByAckTokenHash resolves a delivery ack token. The non-empty guard means
+// a row without a token (channel deliveries) can never match, even if a caller
+// ever hashes an empty input.
+func (r *pageNotificationRepository) FindByAckTokenHash(tx *sql.Tx, hash string) (*models.PageNotification, error) {
+ return lit.SelectSingleNamed[models.PageNotification](
+ tx,
+ "SELECT "+pageNotificationColumns+" FROM page_notifications WHERE ack_token_hash = :hash AND ack_token_hash <> ''",
+ lit.P{"hash": hash},
+ )
+}
+
+func (r *pageNotificationRepository) Create(tx *sql.Tx, notification *models.PageNotification) (int, error) {
+ return lit.Insert[models.PageNotification](tx, notification)
+}
+
+// MarkSent finalizes a delivered row. Guarded on status = 'pending' so a
+// terminal state (cancelled/failed) is never rewritten; the drain's mirror
+// call tolerates matching nothing.
+func (r *pageNotificationRepository) MarkSent(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE page_notifications SET status = 'sent', sent_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// MarkCancelled flips a not-yet-delivered row to cancelled; already-sent or
+// failed rows are left untouched.
+func (r *pageNotificationRepository) MarkCancelled(tx *sql.Tx, id int, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE page_notifications SET status = 'cancelled', sent_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// MarkFailed records a terminal delivery failure. Guarded on
+// status = 'pending' so a cancelled row is never resurrected to failed.
+func (r *pageNotificationRepository) MarkFailed(tx *sql.Tx, id int, errorMsg string, now time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE page_notifications SET status = 'failed', error_msg = :error_msg, sent_at = :now WHERE id = :id AND status = 'pending'",
+ lit.P{"error_msg": errorMsg, "now": now.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+var PageNotificationRepository = pageNotificationRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/team.repository.go b/backend/app/repositories/transactional/sqlite/team.repository.go
new file mode 100644
index 00000000..84037402
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/team.repository.go
@@ -0,0 +1,171 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+)
+
+type teamRepository struct{}
+
+const teamColumns = "id, organization_id, name, description, created_at, updated_at"
+
+func (r *teamRepository) FindById(tx *sql.Tx, id int) (*models.Team, error) {
+ return lit.SelectSingleNamed[models.Team](
+ tx,
+ "SELECT "+teamColumns+" FROM teams WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *teamRepository) FindByOrganizationAndName(tx *sql.Tx, organizationId int, name string) (*models.Team, error) {
+ return lit.SelectSingleNamed[models.Team](
+ tx,
+ "SELECT "+teamColumns+" FROM teams WHERE organization_id = :organization_id AND LOWER(name) = LOWER(:name)",
+ lit.P{"organization_id": organizationId, "name": name},
+ )
+}
+
+func (r *teamRepository) ListByOrganization(tx *sql.Tx, organizationId int) ([]*models.TeamWithCounts, error) {
+ return lit.SelectNamed[models.TeamWithCounts](
+ tx,
+ `SELECT t.id, t.organization_id, t.name, t.description, t.created_at, t.updated_at,
+ (SELECT COUNT(*) FROM team_members tm WHERE tm.team_id = t.id) as member_count,
+ (SELECT COUNT(*) FROM project_teams pt WHERE pt.team_id = t.id) as project_count,
+ (SELECT COUNT(*) FROM oncall_schedules s WHERE s.team_id = t.id) as schedule_count
+ FROM teams t
+ WHERE t.organization_id = :organization_id
+ ORDER BY t.name ASC, t.id ASC`,
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) Create(tx *sql.Tx, team *models.Team) (int, error) {
+ return lit.Insert[models.Team](tx, team)
+}
+
+func (r *teamRepository) Update(tx *sql.Tx, team *models.Team) error {
+ return lit.UpdateNamed(tx, team, "id = :id", lit.P{"id": team.Id})
+}
+
+func (r *teamRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM teams WHERE id = :id", lit.P{"id": id})
+}
+
+func (r *teamRepository) SetMembers(tx *sql.Tx, teamId int, orderedUserIds []int) error {
+ if err := lit.DeleteNamed(db.Driver, tx, "DELETE FROM team_members WHERE team_id = :team_id", lit.P{"team_id": teamId}); err != nil {
+ return err
+ }
+ now := time.Now().UTC()
+ for position, userId := range orderedUserIds {
+ member := &models.TeamMember{
+ TeamId: teamId,
+ UserId: userId,
+ Position: position,
+ CreatedAt: now,
+ }
+ if _, err := lit.Insert[models.TeamMember](tx, member); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (r *teamRepository) ListMembersWithUsersByOrganization(tx *sql.Tx, organizationId int) ([]*models.TeamMemberWithUser, error) {
+ return lit.SelectNamed[models.TeamMemberWithUser](
+ tx,
+ `SELECT tm.team_id, tm.user_id, tm.position, u.name, u.email
+ FROM team_members tm
+ JOIN teams t ON t.id = tm.team_id
+ JOIN users u ON u.id = tm.user_id
+ WHERE t.organization_id = :organization_id
+ ORDER BY tm.team_id ASC, tm.position ASC, tm.id ASC`,
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) FindMemberUserIds(tx *sql.Tx, teamId int) ([]int, error) {
+ members, err := lit.SelectNamed[models.TeamMember](
+ tx,
+ "SELECT id, team_id, user_id, position, created_at FROM team_members WHERE team_id = :team_id ORDER BY position ASC, id ASC",
+ lit.P{"team_id": teamId},
+ )
+ if err != nil {
+ return nil, err
+ }
+ userIds := make([]int, 0, len(members))
+ for _, member := range members {
+ userIds = append(userIds, member.UserId)
+ }
+ return userIds, nil
+}
+
+func (r *teamRepository) RemoveUserFromOrgTeams(tx *sql.Tx, organizationId int, userId int) error {
+ return lit.DeleteNamed(
+ db.Driver,
+ tx,
+ `DELETE FROM team_members
+ WHERE user_id = :user_id
+ AND team_id IN (SELECT id FROM teams WHERE organization_id = :organization_id)`,
+ lit.P{"user_id": userId, "organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) SetProjects(tx *sql.Tx, teamId int, projectIds []uuid.UUID) error {
+ if err := lit.DeleteNamed(db.Driver, tx, "DELETE FROM project_teams WHERE team_id = :team_id", lit.P{"team_id": teamId}); err != nil {
+ return err
+ }
+ now := time.Now().UTC()
+ for _, projectId := range projectIds {
+ link := &models.ProjectTeam{
+ ProjectId: projectId,
+ TeamId: teamId,
+ CreatedAt: now,
+ }
+ if _, err := lit.Insert[models.ProjectTeam](tx, link); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func (r *teamRepository) ListProjectsByOrganization(tx *sql.Tx, organizationId int) ([]*models.TeamProjectRow, error) {
+ return lit.SelectNamed[models.TeamProjectRow](
+ tx,
+ `SELECT pt.team_id, pt.project_id, p.name
+ FROM project_teams pt
+ JOIN teams t ON t.id = pt.team_id
+ JOIN projects p ON p.id = pt.project_id
+ WHERE t.organization_id = :organization_id
+ ORDER BY pt.team_id ASC, p.name ASC`,
+ lit.P{"organization_id": organizationId},
+ )
+}
+
+func (r *teamRepository) FindProjectTeam(tx *sql.Tx, projectId uuid.UUID) (*models.ProjectTeam, error) {
+ return lit.SelectSingleNamed[models.ProjectTeam](
+ tx,
+ "SELECT id, project_id, team_id, created_at FROM project_teams WHERE project_id = :project_id",
+ lit.P{"project_id": projectId},
+ )
+}
+
+func (r *teamRepository) FindTeamForProject(tx *sql.Tx, projectId uuid.UUID) (*models.Team, error) {
+ return lit.SelectSingleNamed[models.Team](
+ tx,
+ `SELECT t.id, t.organization_id, t.name, t.description, t.created_at, t.updated_at
+ FROM teams t
+ JOIN project_teams pt ON pt.team_id = t.id
+ WHERE pt.project_id = :project_id`,
+ lit.P{"project_id": projectId},
+ )
+}
+
+var TeamRepository = teamRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/user_contact_method.repository.go b/backend/app/repositories/transactional/sqlite/user_contact_method.repository.go
new file mode 100644
index 00000000..2a20c2ed
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/user_contact_method.repository.go
@@ -0,0 +1,106 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type userContactMethodRepository struct{}
+
+const userContactMethodColumns = "id, user_id, method_type, config, enabled, verified, verification_code_hash, verification_expires_at, verification_attempts, created_at"
+
+func (r *userContactMethodRepository) FindById(tx *sql.Tx, id int) (*models.UserContactMethod, error) {
+ return lit.SelectSingleNamed[models.UserContactMethod](
+ tx,
+ "SELECT "+userContactMethodColumns+" FROM user_contact_methods WHERE id = :id",
+ lit.P{"id": id},
+ )
+}
+
+func (r *userContactMethodRepository) FindByUser(tx *sql.Tx, userId int) ([]*models.UserContactMethod, error) {
+ return lit.SelectNamed[models.UserContactMethod](
+ tx,
+ "SELECT "+userContactMethodColumns+" FROM user_contact_methods WHERE user_id = :user_id ORDER BY created_at ASC, id ASC",
+ lit.P{"user_id": userId},
+ )
+}
+
+// FindEnabledByUser returns enabled AND verified methods: unverified numbers
+// are never paged.
+func (r *userContactMethodRepository) FindEnabledByUser(tx *sql.Tx, userId int) ([]*models.UserContactMethod, error) {
+ return lit.SelectNamed[models.UserContactMethod](
+ tx,
+ "SELECT "+userContactMethodColumns+" FROM user_contact_methods WHERE user_id = :user_id AND enabled = :enabled AND verified = :verified ORDER BY created_at ASC, id ASC",
+ lit.P{"user_id": userId, "enabled": true, "verified": true},
+ )
+}
+
+// SetVerification stores a fresh hashed code and flips the method back to
+// unverified with zero attempts.
+func (r *userContactMethodRepository) SetVerification(tx *sql.Tx, id int, codeHash string, expiresAt time.Time) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE user_contact_methods SET verified = :verified, verification_code_hash = :code_hash, verification_expires_at = :expires_at, verification_attempts = 0 WHERE id = :id",
+ lit.P{"verified": false, "code_hash": codeHash, "expires_at": expiresAt.UTC(), "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+func (r *userContactMethodRepository) MarkVerified(tx *sql.Tx, id int) error {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE user_contact_methods SET verified = :verified, verification_code_hash = '', verification_expires_at = NULL, verification_attempts = 0 WHERE id = :id",
+ lit.P{"verified": true, "id": id},
+ )
+ if err != nil {
+ return err
+ }
+ return lit.UpdateNative(tx, query, args...)
+}
+
+// IncrementVerificationAttempts consumes one verification attempt, guarded in
+// SQL so concurrent requests cannot exceed the cap (a check-then-increment in
+// Go would race). Returns false when the attempt budget is already spent.
+func (r *userContactMethodRepository) IncrementVerificationAttempts(tx *sql.Tx, id int, maxAttempts int) (bool, error) {
+ query, args, err := lit.ParseNamedQuery(
+ db.Driver,
+ "UPDATE user_contact_methods SET verification_attempts = verification_attempts + 1 WHERE id = :id AND verification_attempts < :max_attempts",
+ lit.P{"id": id, "max_attempts": maxAttempts},
+ )
+ if err != nil {
+ return false, err
+ }
+ result, err := tx.Exec(query, args...)
+ if err != nil {
+ return false, err
+ }
+ affected, err := result.RowsAffected()
+ if err != nil {
+ return false, err
+ }
+ return affected > 0, nil
+}
+
+func (r *userContactMethodRepository) Create(tx *sql.Tx, method *models.UserContactMethod) (int, error) {
+ return lit.Insert[models.UserContactMethod](tx, method)
+}
+
+func (r *userContactMethodRepository) Update(tx *sql.Tx, method *models.UserContactMethod) error {
+ return lit.UpdateNamed(tx, method, "id = :id", lit.P{"id": method.Id})
+}
+
+func (r *userContactMethodRepository) Delete(tx *sql.Tx, id int) error {
+ return lit.DeleteNamed(db.Driver, tx, "DELETE FROM user_contact_methods WHERE id = :id", lit.P{"id": id})
+}
+
+var UserContactMethodRepository = userContactMethodRepository{}
diff --git a/backend/app/repositories/transactional/sqlite/user_notification_rule.repository.go b/backend/app/repositories/transactional/sqlite/user_notification_rule.repository.go
new file mode 100644
index 00000000..2b8a1e16
--- /dev/null
+++ b/backend/app/repositories/transactional/sqlite/user_notification_rule.repository.go
@@ -0,0 +1,48 @@
+//go:build !transactional_pg
+
+package sqlite
+
+import (
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+
+ "github.com/tracewayapp/lit/v2"
+)
+
+type userNotificationRuleRepository struct{}
+
+const userNotificationRuleColumns = "id, user_id, urgency, position, delay_minutes, contact_method_id, created_at"
+
+func (r *userNotificationRuleRepository) FindByUser(tx *sql.Tx, userId int) ([]*models.UserNotificationRule, error) {
+ return lit.SelectNamed[models.UserNotificationRule](
+ tx,
+ "SELECT "+userNotificationRuleColumns+" FROM user_notification_rules WHERE user_id = :user_id ORDER BY urgency ASC, position ASC, id ASC",
+ lit.P{"user_id": userId},
+ )
+}
+
+func (r *userNotificationRuleRepository) FindByUserAndUrgency(tx *sql.Tx, userId int, urgency string) ([]*models.UserNotificationRule, error) {
+ return lit.SelectNamed[models.UserNotificationRule](
+ tx,
+ "SELECT "+userNotificationRuleColumns+" FROM user_notification_rules WHERE user_id = :user_id AND urgency = :urgency ORDER BY position ASC, id ASC",
+ lit.P{"user_id": userId, "urgency": urgency},
+ )
+}
+
+// ReplaceForUser swaps the user's entire rule set in one transaction, so
+// positions are correct by construction.
+func (r *userNotificationRuleRepository) ReplaceForUser(tx *sql.Tx, userId int, rules []*models.UserNotificationRule) error {
+ if err := lit.DeleteNamed(db.Driver, tx, "DELETE FROM user_notification_rules WHERE user_id = :user_id", lit.P{"user_id": userId}); err != nil {
+ return err
+ }
+ for _, rule := range rules {
+ if _, err := lit.Insert[models.UserNotificationRule](tx, rule); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+var UserNotificationRuleRepository = userNotificationRuleRepository{}
diff --git a/backend/app/repositories/transactional/transactional_pg.go b/backend/app/repositories/transactional/transactional_pg.go
index dfc69b46..ea01e64b 100644
--- a/backend/app/repositories/transactional/transactional_pg.go
+++ b/backend/app/repositories/transactional/transactional_pg.go
@@ -5,20 +5,29 @@ package transactional
import pgrepo "github.com/tracewayapp/traceway/backend/app/repositories/transactional/pg"
var (
- AuthorizationCodeRepository = pgrepo.AuthorizationCodeRepository
- DashboardRepository = pgrepo.DashboardRepository
- DashboardTemplateRepository = pgrepo.DashboardTemplateRepository
- DeviceAuthorizationRepository = pgrepo.DeviceAuthorizationRepository
- InvitationRepository = pgrepo.InvitationRepository
- MetricRegistryRepository = pgrepo.MetricRegistryRepository
- NotificationChannelRepository = pgrepo.NotificationChannelRepository
- NotificationRuleRepository = pgrepo.NotificationRuleRepository
- OAuthSessionRepository = pgrepo.OAuthSessionRepository
- OauthClientRepository = pgrepo.OauthClientRepository
- OrganizationRepository = pgrepo.OrganizationRepository
- PersonalAccessTokenRepository = pgrepo.PersonalAccessTokenRepository
- ProjectRepository = pgrepo.ProjectRepository
- ProjectUserRoleRepository = pgrepo.ProjectUserRoleRepository
- RefreshTokenRepository = pgrepo.RefreshTokenRepository
- UserRepository = pgrepo.UserRepository
+ AuthorizationCodeRepository = pgrepo.AuthorizationCodeRepository
+ DashboardRepository = pgrepo.DashboardRepository
+ DashboardTemplateRepository = pgrepo.DashboardTemplateRepository
+ DeviceAuthorizationRepository = pgrepo.DeviceAuthorizationRepository
+ EscalationPolicyRepository = pgrepo.EscalationPolicyRepository
+ InvitationRepository = pgrepo.InvitationRepository
+ MetricRegistryRepository = pgrepo.MetricRegistryRepository
+ NotificationChannelRepository = pgrepo.NotificationChannelRepository
+ NotificationRuleRepository = pgrepo.NotificationRuleRepository
+ OncallOverrideRepository = pgrepo.OncallOverrideRepository
+ OncallScheduleRepository = pgrepo.OncallScheduleRepository
+ OAuthSessionRepository = pgrepo.OAuthSessionRepository
+ OauthClientRepository = pgrepo.OauthClientRepository
+ OrganizationRepository = pgrepo.OrganizationRepository
+ OutboxRepository = pgrepo.OutboxRepository
+ PageNotificationRepository = pgrepo.PageNotificationRepository
+ PageRepository = pgrepo.PageRepository
+ PersonalAccessTokenRepository = pgrepo.PersonalAccessTokenRepository
+ ProjectRepository = pgrepo.ProjectRepository
+ ProjectUserRoleRepository = pgrepo.ProjectUserRoleRepository
+ RefreshTokenRepository = pgrepo.RefreshTokenRepository
+ TeamRepository = pgrepo.TeamRepository
+ UserContactMethodRepository = pgrepo.UserContactMethodRepository
+ UserNotificationRuleRepository = pgrepo.UserNotificationRuleRepository
+ UserRepository = pgrepo.UserRepository
)
diff --git a/backend/app/repositories/transactional/transactional_sqlite.go b/backend/app/repositories/transactional/transactional_sqlite.go
index ec70bc54..a797335b 100644
--- a/backend/app/repositories/transactional/transactional_sqlite.go
+++ b/backend/app/repositories/transactional/transactional_sqlite.go
@@ -5,20 +5,29 @@ package transactional
import sqliterepo "github.com/tracewayapp/traceway/backend/app/repositories/transactional/sqlite"
var (
- AuthorizationCodeRepository = sqliterepo.AuthorizationCodeRepository
- DashboardRepository = sqliterepo.DashboardRepository
- DashboardTemplateRepository = sqliterepo.DashboardTemplateRepository
- DeviceAuthorizationRepository = sqliterepo.DeviceAuthorizationRepository
- InvitationRepository = sqliterepo.InvitationRepository
- MetricRegistryRepository = sqliterepo.MetricRegistryRepository
- NotificationChannelRepository = sqliterepo.NotificationChannelRepository
- NotificationRuleRepository = sqliterepo.NotificationRuleRepository
- OAuthSessionRepository = sqliterepo.OAuthSessionRepository
- OauthClientRepository = sqliterepo.OauthClientRepository
- OrganizationRepository = sqliterepo.OrganizationRepository
- PersonalAccessTokenRepository = sqliterepo.PersonalAccessTokenRepository
- ProjectRepository = sqliterepo.ProjectRepository
- ProjectUserRoleRepository = sqliterepo.ProjectUserRoleRepository
- RefreshTokenRepository = sqliterepo.RefreshTokenRepository
- UserRepository = sqliterepo.UserRepository
+ AuthorizationCodeRepository = sqliterepo.AuthorizationCodeRepository
+ DashboardRepository = sqliterepo.DashboardRepository
+ DashboardTemplateRepository = sqliterepo.DashboardTemplateRepository
+ DeviceAuthorizationRepository = sqliterepo.DeviceAuthorizationRepository
+ EscalationPolicyRepository = sqliterepo.EscalationPolicyRepository
+ InvitationRepository = sqliterepo.InvitationRepository
+ MetricRegistryRepository = sqliterepo.MetricRegistryRepository
+ NotificationChannelRepository = sqliterepo.NotificationChannelRepository
+ NotificationRuleRepository = sqliterepo.NotificationRuleRepository
+ OncallOverrideRepository = sqliterepo.OncallOverrideRepository
+ OncallScheduleRepository = sqliterepo.OncallScheduleRepository
+ OAuthSessionRepository = sqliterepo.OAuthSessionRepository
+ OauthClientRepository = sqliterepo.OauthClientRepository
+ OrganizationRepository = sqliterepo.OrganizationRepository
+ OutboxRepository = sqliterepo.OutboxRepository
+ PageNotificationRepository = sqliterepo.PageNotificationRepository
+ PageRepository = sqliterepo.PageRepository
+ PersonalAccessTokenRepository = sqliterepo.PersonalAccessTokenRepository
+ ProjectRepository = sqliterepo.ProjectRepository
+ ProjectUserRoleRepository = sqliterepo.ProjectUserRoleRepository
+ RefreshTokenRepository = sqliterepo.RefreshTokenRepository
+ TeamRepository = sqliterepo.TeamRepository
+ UserContactMethodRepository = sqliterepo.UserContactMethodRepository
+ UserNotificationRuleRepository = sqliterepo.UserNotificationRuleRepository
+ UserRepository = sqliterepo.UserRepository
)
diff --git a/backend/app/retention/outbox.go b/backend/app/retention/outbox.go
new file mode 100644
index 00000000..534109f1
--- /dev/null
+++ b/backend/app/retention/outbox.go
@@ -0,0 +1,22 @@
+package retention
+
+import (
+ "context"
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+const outboxPruneInterval = 24 * time.Hour
+
+// Terminal outbox rows are debugging artifacts: the durable audit lives in
+// fired_notifications and page_notifications. Sent/cancelled rows keep a week;
+// failed rows keep a month so operators can still find why an alert never
+// arrived. Pending/sending rows are never pruned.
+func startOutboxPrune(ctx context.Context) {
+ startDBPruneWorker(ctx, "notification_outbox", outboxPruneInterval, func(tx *sql.Tx) (int64, error) {
+ now := time.Now().UTC()
+ return transactional.OutboxRepository.PruneTerminal(tx, now.AddDate(0, 0, -7), now.AddDate(0, 0, -30))
+ })
+}
diff --git a/backend/app/retention/prune_worker.go b/backend/app/retention/prune_worker.go
index d64e4022..ee2827bb 100644
--- a/backend/app/retention/prune_worker.go
+++ b/backend/app/retention/prune_worker.go
@@ -13,7 +13,8 @@ import (
// startDBPruneWorker runs prune once at startup and then every interval until
// ctx is cancelled, inside a panic-guarded goroutine. It is the shared
-// scaffolding behind the main-DB retention workers (oauth_sessions, auth_tokens).
+// scaffolding behind the main-DB retention workers (oauth_sessions,
+// auth_tokens, notification_outbox).
func startDBPruneWorker(ctx context.Context, name string, interval time.Duration, prune func(tx *sql.Tx) (int64, error)) {
config.Logf("Starting %s prune worker (interval: %s)", name, interval)
diff --git a/backend/app/retention/retention.go b/backend/app/retention/retention.go
index c714cf61..17377e00 100644
--- a/backend/app/retention/retention.go
+++ b/backend/app/retention/retention.go
@@ -22,6 +22,7 @@ func Start(ctx context.Context) {
startProfileArchiveDiskCleanup(ctx, parseRetentionDays(cfg.ProfileRetentionDays))
startOAuthSessionsPrune(ctx)
startAuthTokensPrune(ctx)
+ startOutboxPrune(ctx)
}
func parseRetentionDays(value string) int {
diff --git a/backend/app/services/email.service.go b/backend/app/services/email.service.go
index 35a6c91f..74097019 100644
--- a/backend/app/services/email.service.go
+++ b/backend/app/services/email.service.go
@@ -141,6 +141,9 @@ func (e *emailService) sendMail(to []string, msg []byte) error {
return fmt.Errorf("SMTP dial failed: %w", err)
}
+ // Must be set before smtp.NewClient, which blocks reading the server greeting.
+ conn.SetDeadline(time.Now().Add(10 * time.Second))
+
client, err := smtp.NewClient(conn, e.host)
if err != nil {
conn.Close()
@@ -148,8 +151,6 @@ func (e *emailService) sendMail(to []string, msg []byte) error {
}
defer client.Close()
- conn.SetDeadline(time.Now().Add(10 * time.Second))
-
if ok, _ := client.Extension("STARTTLS"); ok {
if err := client.StartTLS(&tls.Config{ServerName: e.host}); err != nil {
return fmt.Errorf("SMTP STARTTLS failed: %w", err)
diff --git a/backend/cmd/run.go b/backend/cmd/run.go
index 40eb9695..13044411 100644
--- a/backend/cmd/run.go
+++ b/backend/cmd/run.go
@@ -21,6 +21,8 @@ import (
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/monitoring"
"github.com/tracewayapp/traceway/backend/app/notifications"
+ "github.com/tracewayapp/traceway/backend/app/oncall"
+ "github.com/tracewayapp/traceway/backend/app/outbox"
"github.com/tracewayapp/traceway/backend/app/recordings"
"github.com/tracewayapp/traceway/backend/app/retention"
"github.com/tracewayapp/traceway/backend/app/services"
@@ -146,6 +148,7 @@ func Run(opts ...Option) {
middleware.InitRequireWriteAccess()
middleware.InitRequireProjectAccess()
middleware.InitRequireAdminAccess()
+ middleware.InitRequireOrganizationAccess()
middleware.InitUseSourceMapAuth()
services.InitEmail()
@@ -156,6 +159,11 @@ func Run(opts ...Option) {
hook(ctx)
}
+ outbox.RegisterSender(notifications.AdapterSend)
+ outbox.RegisterTerminalHook(notifications.OnOutboxTerminal)
+ notifications.RegisterPageOpener(oncall.OpenPageFromDispatch)
+ outbox.StartDrain(ctx)
+ oncall.StartEscalator(ctx)
notifications.StartEvaluator(ctx)
retention.Start(ctx)
recordings.Start(ctx)
@@ -192,6 +200,7 @@ func Run(opts ...Option) {
monitoring.StartClickHouseReporter(ctx)
monitoring.StartBackendReporter(ctx)
monitoring.StartTelemetryDBReporter(ctx)
+ monitoring.StartOutboxReporter(ctx)
}
router.GET("/health", func(c *gin.Context) {
@@ -270,11 +279,26 @@ func Run(opts ...Option) {
}
}
+// applyEnvOverrides forwards env vars to a config the embedded Run(opts...)
+// path built without LoadFromEnv; new Cfg fields belong in this table.
func applyEnvOverrides(cfg *config.Cfg) {
for _, m := range []struct {
envVar string
dest *string
}{
+ {"SMTP_ENABLED", &cfg.SMTPEnabled},
+ {"SMTP_HOST", &cfg.SMTPHost},
+ {"SMTP_PORT", &cfg.SMTPPort},
+ {"SMTP_USERNAME", &cfg.SMTPUsername},
+ {"SMTP_PASSWORD", &cfg.SMTPPassword},
+ {"SMTP_FROM", &cfg.SMTPFrom},
+ {"ONCALL_POLL_SECONDS", &cfg.OncallPollSeconds},
+ {"OUTBOX_POLL_SECONDS", &cfg.OutboxPollSeconds},
+ {"TWILIO_ACCOUNT_SID", &cfg.TwilioAccountSID},
+ {"TWILIO_AUTH_TOKEN", &cfg.TwilioAuthToken},
+ {"TWILIO_FROM_NUMBER", &cfg.TwilioFromNumber},
+ {"TWILIO_MESSAGING_SERVICE_SID", &cfg.TwilioMessagingServiceSID},
+ {"ALLOW_PRIVATE_NOTIFICATION_TARGETS", &cfg.AllowPrivateNotificationTargets},
{"OAUTH_SESSION_SECRET", &cfg.OAuthSessionSecret},
{"GOOGLE_CLIENT_ID", &cfg.GoogleClientID},
{"GOOGLE_CLIENT_SECRET", &cfg.GoogleClientSecret},
diff --git a/diagrams/docs/on-call/how-the-pieces-fit-together.dot b/diagrams/docs/on-call/how-the-pieces-fit-together.dot
new file mode 100644
index 00000000..9f39453e
--- /dev/null
+++ b/diagrams/docs/on-call/how-the-pieces-fit-together.dot
@@ -0,0 +1,41 @@
+// On-Call: alert rule -> page -> escalation -> a human's phone. Render:
+// dot -Tsvg how-the-pieces-fit-together.dot -o how-the-pieces-fit-together.svg
+// Output belongs in docs/public/on-call/.
+// Palette matches the Traceway dark theme (app/globals.css).
+digraph oncall_pieces {
+ bgcolor="transparent";
+ rankdir=TB;
+ nodesep=0.32;
+ ranksep=0.42;
+ pad=0.2;
+ fontname="Helvetica";
+
+ node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=14,
+ penwidth=1.3, margin="0.30,0.18",
+ color="#283041", fillcolor="#151a24", fontcolor="#f4f6fb"];
+ edge [color="#5a6374", penwidth=1.4, arrowsize=0.85,
+ fontname="Helvetica", fontsize=10, fontcolor="#8a93a6"];
+
+ rule [label=<an alert rule fires error rate · latency · metric threshold · missing data >,
+ fillcolor="#0f131c"];
+ // Subtitles stay in the default face: graphviz has no metrics for Menlo and
+ // sizes the box from a much narrower estimate, so a mono subtitle wider than
+ // its title spills out of the border.
+ channel [label=<escalation channel channel type “escalation”, pointing at one policy >];
+ page [label=<a page opens one incident: re-fires bump its count, nobody is notified twice >,
+ color="#f0a020", fillcolor="#1c1710"];
+ step [label=<policy step level 1, level 2, … each with a delay >];
+ target [label=<target schedule · team · user · notification channel >];
+ person [label=<the person on call resolved when the step runs, overrides included >];
+ chain [label=<notification rule chain their own steps, for this page’s urgency >];
+ method [label=<contact method email · Slack · Pushover · Telegram · SMS >,
+ color="#7c5cff", fillcolor="#15121f"];
+ ack [label=<acknowledged escalation stops, queued deliveries are cancelled >,
+ color="#22e0a8", fillcolor="#0c1714"];
+
+ rule -> channel -> page -> step -> target -> person -> chain -> method -> ack;
+
+ method -> step [constraint=false, tailport=e, headport=e,
+ style=dashed, color="#ff5a5f", fontcolor="#ff8a8d",
+ label=" nobody acknowledged:\l next level after\l the step's delay\l"];
+}
diff --git a/docs/pages/learn/_meta.json b/docs/pages/learn/_meta.json
index 2f28ea92..5f8672a1 100644
--- a/docs/pages/learn/_meta.json
+++ b/docs/pages/learn/_meta.json
@@ -15,6 +15,7 @@
"dashboards": "Dashboards",
"logs": "Logs",
"alerts": "Alerts",
+ "on-call": "On-Call",
"sso": "SSO",
"cli": "CLI",
"cli-auth": "CLI Authentication",
diff --git a/docs/pages/learn/alerts.mdx b/docs/pages/learn/alerts.mdx
index a79ee768..0f4e5159 100644
--- a/docs/pages/learn/alerts.mdx
+++ b/docs/pages/learn/alerts.mdx
@@ -1,12 +1,12 @@
# Alerts
-Alerts notify you when conditions in your application require attention — errors, latency spikes, metric thresholds, or missing data. Configure channels and rules to get notified through email, Slack, webhooks, GitHub Issues, Pushover, or Telegram.
+Alerts notify you when conditions in your application require attention — errors, latency spikes, metric thresholds, or missing data. Configure channels and rules to get notified through email, Slack, webhooks, GitHub Issues, Pushover, or Telegram, or to page whoever is [on call](/learn/on-call).
## How Alerts Work
The alerting system has two parts:
-- **Channels** define where notifications are delivered (email, Slack, webhook, GitHub, Pushover, Telegram)
+- **Channels** define where notifications are delivered (email, Slack, webhook, GitHub, Pushover, Telegram, or an escalation policy)
- **Rules** define what conditions trigger a notification
When a rule's condition is met, Traceway sends a notification through the rule's attached channel. Each rule is linked to exactly one channel, but a channel can be used by multiple rules.
@@ -23,8 +23,9 @@ A channel represents a notification destination. Traceway supports the following
| GitHub | Creates a GitHub issue in a repository | Personal access token, repository owner/name, optional labels |
| Pushover | Sends push notifications to your mobile devices | User Key, App Token |
| Telegram | Sends a message via a Telegram bot to a user or group chat | Bot Token, Chat ID |
+| Escalation policy | Opens an on-call [page](/learn/on-call) and runs the policy's escalation chain until somebody acknowledges, instead of sending a message | Escalation policy |
-Each channel has a **Test** button that sends a sample notification so you can verify the configuration before attaching rules.
+Each channel has a **Test** button that sends a sample notification so you can verify the configuration before attaching rules. Escalation-policy channels are the exception: their Test button opens a real [page](/learn/on-call) and notifies the on-call responder, so resolve the test page when you are done.
### Setting Up Email
diff --git a/docs/pages/learn/on-call.mdx b/docs/pages/learn/on-call.mdx
new file mode 100644
index 00000000..db99da13
--- /dev/null
+++ b/docs/pages/learn/on-call.mdx
@@ -0,0 +1,319 @@
+# On-Call
+
+On-Call turns an alert into a person's phone ringing. Alerts on their own post to a channel and hope somebody is watching. A page keeps escalating until a human acknowledges it.
+
+Traceway ships the full paging stack: teams, rotating schedules, escalation policies, per-responder contact methods, and a no-login acknowledge link. It lives under **On-Call** in the sidebar, and the badge there counts the pages still open on the current project.
+
+
+
+## How the pieces fit together
+
+An alert rule fires, and instead of sending a message it opens a **page**. The page walks an **escalation policy**. Each step of that policy names targets, and a target resolves to people through a **schedule**, a **team**, or a direct user. Each of those people is then reached through their own **contact methods**, in the order their **notification rules** define.
+
+
+
+Two ideas are worth holding onto:
+
+- **The page is the unit of work, not the message.** A noisy rule that fires two hundred times produces one page. Re-fires bump an event counter and never restart the escalation clock.
+- **Escalation stops on acknowledge, not on delivery.** Sending a message proves nothing. Traceway keeps climbing the policy until somebody acknowledges, or until the policy is exhausted.
+
+## Setting it up
+
+The pieces reference each other, so build them in this order. Steps 1 to 3 are organization scoped and need the **owner** or **admin** role. Step 4 is project scoped and needs write access on the project, not an admin role. Contact methods are personal, so every responder does step 5 for themselves.
+
+1. **Create a team.** It groups the responders and owns the projects they are responsible for.
+2. **Build a schedule** for that team, so there is always somebody on call.
+3. **Create an escalation policy** whose first step points at that schedule.
+4. **Add an escalation channel** on the Alerts page pointing at the policy, then attach your rules to it.
+5. **Tell each responder to add contact methods**, and optionally a notification rule chain.
+
+Step 5 is the one teams forget. It still works without it, because a responder with no contact methods is paged on their account email, but nobody wants to find that out during an incident.
+
+## Teams
+
+A team is a named group of people that owns projects. Ownership is one project to one team, so every project has at most one team answering for it. The issue detail page uses that link to show who is on call for the project you are looking at.
+
+
+
+Creating or editing a team sets its name, description, members, and owned projects in one dialog.
+
+
+
+Members are stored in order, but that order is only how the team is listed. A policy step targeting the whole team pages every member at once. Schedule layers do not inherit it either. Each layer keeps its own rotation order, picked from anyone in the organization.
+
+## Schedules
+
+A schedule answers one question: who is on call right now. It belongs to a team, carries its own timezone, and is built from **layers**.
+
+
+
+### Layers
+
+Each layer is an independent rotation over a list of people. Layers stack, and a layer later in the list takes precedence over the ones before it. At any instant at most one person is on call for the schedule, which is the highest-precedence layer covering that moment. If no layer covers it, nobody is on call and a policy step targeting the schedule reaches nobody.
+
+
+
+A layer has:
+
+| Field | Meaning |
+|-------|---------|
+| Rotation | `daily`, `weekly`, or `custom` (every N days) |
+| Handoff time | The wall-clock time the shift changes hands, in the schedule's timezone |
+| Handoff day | Which weekday the handoff happens on, for weekly rotations |
+| Rotation start | The anchor date the rotation counts from |
+| Members | The people to rotate through, in order |
+| Restrictions | Optional windows that limit when this layer covers anything |
+
+### Restrictions
+
+A restriction narrows a layer to certain hours or days. Without one, the layer covers every minute. Two shapes are supported:
+
+- **Daily**, for example 18:00 to 09:00. A window that ends earlier than it starts wraps past midnight.
+- **Weekly**, for example Saturday 00:00 to Monday 09:00. These can span several days.
+
+Restrictions are evaluated in the schedule's timezone, so daylight saving changes are handled for you. On a spring-forward day a wall-clock time inside the skipped hour resolves back an hour, so a window ending in that hour is simply an hour shorter that night. A short window that reaches into the gap can collapse to nothing, in which case the layer covers nothing that day and only that day. The days around it are unaffected.
+
+The screenshot above shows the common pattern: a business hours layer restricted to Monday 09:00 through Friday 18:00, and a nights and weekends layer below it restricted to the inverse. Layers are listed in precedence order, so the one further down wins where the two overlap. The bottom `Schedule` row of the timeline is the stacked result, which is who actually gets paged.
+
+### Timezone
+
+The schedule's timezone is what handoff times and restrictions mean. Set it to where the people live, not where the servers live. A rotation set to `Europe/Berlin` hands off at 09:00 Berlin time all year, through daylight saving on both sides.
+
+The timeline renders in the schedule's timezone and marks the current moment with a red line. It can span at most 62 days per request, which the Day, Week, 2 Weeks, and Month controls stay inside.
+
+### Overrides
+
+An override temporarily replaces whoever the layers would have picked. Use it for a doctor's appointment, a swapped weekend, or someone going on vacation.
+
+
+
+Overrides beat every layer. Where two overrides overlap, the one created most recently wins. They appear on the timeline as a dashed block labelled `OVERRIDE`.
+
+They are also the one on-call mutation any member can make, not just admins. Covering for a teammate should not need an administrator. An override can cover at most 30 days, and it can be deleted by whoever created it, the person it covers, or an admin.
+
+
+
+## Escalation policies
+
+A policy is the ordered list of who to wake, and how long to wait before giving up on them.
+
+
+
+Each step holds one or more targets and a delay. The delay is how long the page waits for an acknowledgement before moving to the next step.
+
+
+
+A target is one of:
+
+| Target | Resolves to |
+|--------|-------------|
+| Schedule | Whoever is on call for that schedule at the moment the step runs |
+| Team | Every member of the team |
+| User | That specific person |
+| Channel | A notification channel, for posting the page into Slack or a webhook alongside the human paging |
+
+A page snapshots its policy the moment it opens, so editing a policy never changes pages that are already escalating. Inside that snapshot, targets are still resolved to people when the step runs. A schedule target reaches whoever is on call at that moment, including an override that started five minutes ago.
+
+Because a policy that points at something deleted would page nobody, a schedule or team a live policy targets cannot be deleted. Remove the step first. The same guard already stops you deleting a policy a channel is using.
+
+Within a single step, each person is notified once. Someone who appears on two targets of the same step does not get two pages.
+
+### Repeat
+
+After the last step, `Repeat` sends the whole chain around again up to five more times. Use it when a page must not be allowed to go unanswered overnight. With repeat set to zero the page stays open after the last step, but nothing further is sent.
+
+### Urgency
+
+Urgency decides which of the responder's two notification rule chains runs.
+
+| Setting | Behaviour |
+|---------|-----------|
+| Auto (default) | Critical severity becomes high urgency, everything else becomes low |
+| Always high | Every page from this policy is high urgency |
+| Always low | Every page from this policy is low urgency |
+
+Urgency is resolved once when the page opens and then remembered. A low severity re-fire can never quietly downgrade a page that is already escalating at high urgency.
+
+## Connecting alerts to a policy
+
+Paging is wired in through the normal Alerts machinery. Create a channel with the type **Escalation policy** and point it at the policy you want.
+
+
+
+Then attach any rule to that channel, exactly as you would for Slack or email. When the rule fires, the channel opens a page instead of sending a message.
+
+The channel's **Test** button opens a real page and runs the real escalation. It is the honest way to find out whether your rotation and your policy work. Two things to know about it. The test page is created with `info` severity, so a policy left on **Auto** treats it as low urgency and runs your low urgency chain. And every test of the same channel shares a dedup key, so pressing Test again while the previous test page is still unresolved only bumps that page. Resolve the test page when you are done.
+
+The policy must belong to the same organization as the project. The dialog only lists policies you can actually use.
+
+## Pages
+
+A page is one incident. It has a status, an escalation level, and a delivery log.
+
+| Status | Meaning |
+|--------|---------|
+| Open | Nobody has taken it. Escalation is running. |
+| Acknowledged | Somebody is on it. Escalation has stopped, and queued deliveries are cancelled. |
+| Resolved | Finished. The dedup key is released, so the same condition can open a fresh page. |
+
+### Deduplication
+
+While a page is unresolved, the same rule and dedup token bump the existing page instead of opening a new one. The event counter goes up, the escalation clock is untouched, and nobody gets notified again.
+
+This is what keeps a flapping error from turning into a pager storm. A rule that fires sixty times over half an hour produces one page and exactly the notifications its policy calls for.
+
+Rules without a dedup token deduplicate at the rule level, so one noisy rule is one page.
+
+### Acknowledging and resolving
+
+Acknowledging stops escalation immediately and cancels anything still queued for delivery, including the later steps of a responder's own notification chain. Resolving closes the page.
+
+Neither action requires write access on the project. A responder with a read-only role who gets paged at 3am can still take the incident and close it out.
+
+
+
+The detail view shows the escalation chain with the current level highlighted, and every delivery attempt with its outcome. That log is the thing to read when somebody says they never got paged.
+
+### Acknowledging without logging in
+
+Every delivery addressed to a person carries its own single-purpose acknowledge link. Opening it shows a summary of the page. It does not require a session, so it works from a phone at 3am with no password manager in reach. Deliveries to a **Channel** target are the exception, since a shared Slack room has no single owner. Those carry the ordinary dashboard link instead.
+
+
+
+The link is scoped to acknowledging that one page and nothing else. It gives no dashboard access, cannot resolve, and stops working once the page is resolved. Opening the link is read-only, so an email scanner following links cannot acknowledge your incident by accident. The acknowledgement is attributed to whoever the delivery was addressed to, and the page records that it came in through a link rather than the dashboard.
+
+## Contact methods
+
+Contact methods are personal. Each responder manages their own from the **Account** page, and nobody else can read or change them. The one thing teammates do see is a page's delivery log, which names the destination each notification went to so an incident can be debugged. Personal destinations are masked there: a phone number shows its last four digits, an email override shows its first character and domain. Your account email is shown in full, since everyone in the organization can already see it.
+
+
+
+| Type | Configuration |
+|------|---------------|
+| Email | Optional address. Leave it blank to use your account email. |
+| Slack | Incoming webhook URL, with optional channel and username overrides |
+| Pushover | User key and app token |
+| Telegram | Bot token and chat ID |
+| SMS | Phone number in E.164 format, for example `+12025550123` |
+
+
+
+Every method has a **Test** button that sends a canned message, and an enable toggle so you can silence one without deleting it.
+
+If you configure nothing at all, pages still fall back to your account email, so a responder is never silently skipped.
+
+That fallback only leaves the building if the server can send mail. Set `SMTP_ENABLED=true` along with the rest of the SMTP settings. With SMTP off, Traceway runs the email adapter in log-only mode and writes the page to the server log instead of delivering it, which is fine for local development and a bad surprise in production. On a self-hosted instance, configure SMTP before you rely on on-call.
+
+### Slack webhooks and private addresses
+
+Slack is the one contact method type where you choose the destination host yourself. Every other type sends to a fixed service, so only the Slack webhook URL is checked: it must use http or https and resolve to a public address. A URL whose host does not resolve is rejected too, since it could never receive a page.
+
+The check exists because contact methods are personal. They need no project role, so without it any member of the organization, readonly included, could point a webhook at an internal address and use the **Test** button to send requests into the server's own network.
+
+If your paging destinations genuinely live on a private network, for example a self-hosted Mattermost behind the same firewall as Traceway, set `ALLOW_PRIVATE_NOTIFICATION_TARGETS=true` on the server to turn the check off.
+
+Project notification channels are not affected either way. The [webhook channel type](/learn/alerts) exists to post anywhere, and creating one requires write access.
+
+### SMS and Twilio
+
+SMS needs Twilio credentials on the server. Set `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, and one sender, either `TWILIO_FROM_NUMBER` or `TWILIO_MESSAGING_SERVICE_SID`.
+
+Without them SMS is not offered at all. The type picker hides it, and the server rejects attempts to create one. This is deliberate. A channel that silently drops your 3am page is worse than no channel.
+
+Phone numbers must be verified before they are paged. Adding one sends a six digit code, which expires after ten minutes and allows five attempts. Unverified numbers are never used for a real page, and the code send rate is capped per number so nobody can be flooded with codes.
+
+### Notification rules
+
+By default, a page notifies every enabled contact method at once. Notification rules replace that with an ordered chain per urgency.
+
+Each step names one contact method and a delay in minutes. A typical high urgency chain nudges you quietly first, then gets louder:
+
+| Step | Method | After |
+|------|--------|-------|
+| 1 | Slack | 0 minutes |
+| 2 | Push notification | 2 minutes |
+| 3 | SMS | 5 minutes |
+
+The whole chain is scheduled the moment the page reaches you. Acknowledging cancels every step that has not fired yet, so taking the incident in the first thirty seconds means your phone never rings.
+
+Low urgency usually deserves a shorter chain, often a single Slack message with no follow-up. Leave a chain empty to fall back to notifying every enabled method immediately.
+
+## Who is on call right now
+
+The **Overview** tab answers that at a glance, per team and per schedule, with who is up next.
+
+
+
+The issue detail page shows the same thing for the project you are looking at, so you know who to pull in without leaving the stack trace.
+
+## Common setups
+
+### One engineer, one rotation
+
+The smallest useful configuration. One team, one schedule with a single weekly layer over everybody, one policy with a single step pointing at the schedule and a delay of 15 minutes. Set repeat to 1 or 2 so an unanswered page comes back around.
+
+### Business hours and out of hours
+
+Two layers on one schedule. The first layer covers the working week and rotates over the whole team. The second covers nights and weekends and rotates over the smaller group that signed up for it. Restrictions on both keep them out of each other's way, and the second wins anywhere they overlap. This is the configuration in the screenshots above.
+
+### Primary and secondary
+
+Two schedules on the same team, and a policy with two steps. Step one targets the primary schedule with a delay of five minutes. Step two targets the secondary. A third step targeting the whole team makes a decent last resort.
+
+### Loud for critical, quiet for the rest
+
+Two policies. The critical one keeps urgency on Auto and escalates quickly. The other sets urgency to **Always low** with one step and a long delay, then each responder points their low urgency chain at Slack only. Attach noisy rules to the second policy and they will never ring a phone.
+
+### Vacation cover
+
+Add an override for the dates. It beats the rotation without editing it, and it disappears on its own when the dates pass. There is no need to reshuffle the layer and remember to put it back.
+
+## Permissions
+
+| Action | Required role |
+|--------|---------------|
+| View teams, schedules, policies, and who is on call | Any member of the organization |
+| Create or edit teams, schedules, and policies | Organization owner or admin |
+| Create a schedule override | Any member of the organization |
+| Delete a schedule override | Its creator, the person it covers, or an organization owner or admin |
+| View, acknowledge, and resolve pages | Project read access, no write access needed |
+| Manage your own contact methods and notification rules | Yourself only |
+
+## Delivery guarantees
+
+Nothing is sent from inside the request that triggered it. A firing rule only opens the page, in its own transaction. The escalator then claims that page and, in a single transaction, advances the escalation level and writes every delivery for that level into a durable outbox. A separate worker drains the outbox and does the sending.
+
+Because the level advance and the deliveries commit together, a crash mid-incident cannot lose a page or enqueue a level twice. Sending itself is at-least-once by design. A delivery interrupted between the send and the record of it is reclaimed about five minutes later and retried, so a crash at exactly the wrong moment can repeat one message. Paging twice is the right failure mode when the alternative is not paging at all.
+
+Deliveries that fail are retried on a backoff of 1, 5, 15, and 60 minutes, up to five attempts, before being recorded as permanently failed and reported as an exception. Acknowledging cancels anything still queued, and a cancelled delivery can never come back to life, though a send already in flight may still land once.
+
+The `outbox` block on `/api/health/deep` exposes queue depth and the oldest pending delivery, so you can alert on your alerting.
+
+## Configuration
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `ONCALL_POLL_SECONDS` | `30` | How often the escalator checks for pages that are due to escalate. Minimum 5. Kept separate from rule evaluation so raising that interval never delays paging. |
+| `OUTBOX_POLL_SECONDS` | `15` | How often the outbox drain worker sends queued notifications. Minimum 5. |
+| `APP_BASE_URL` | `http://localhost:5173` | The origin used to build acknowledge links. Set it. The escalator sends from a background worker with no request to derive an origin from, so if this is unset every page links to localhost. |
+| `TWILIO_ACCOUNT_SID` | unset | Twilio account SID, required for SMS |
+| `TWILIO_AUTH_TOKEN` | unset | Twilio auth token, required for SMS |
+| `TWILIO_FROM_NUMBER` | unset | Sending number. Use this or a messaging service. |
+| `TWILIO_MESSAGING_SERVICE_SID` | unset | Twilio messaging service, as an alternative to a from number |
+| `ALLOW_PRIVATE_NOTIFICATION_TARGETS` | unset | Set to `true` to let personal contact methods point at private or loopback addresses. Off by default so a contact method, which needs no project role, cannot be used to reach the server's own network. |
+| `SMTP_ENABLED` | unset | Must be `true` for email pages to be delivered. Anything else runs the email adapter in log-only mode. |
+
+A freshly opened page does not wait for the next poll. The escalator is woken directly, so the first level is notified within a second or so regardless of `ONCALL_POLL_SECONDS`.
+
+## Limits
+
+| Thing | Limit |
+|-------|-------|
+| Steps per escalation policy | 10 |
+| Targets per step | 10 |
+| Delay between steps | 1 to 1440 minutes |
+| Policy repeats | 5 |
+| Steps per notification rule chain | 10 |
+| Delay per notification rule step | 0 to 120 minutes |
+| Override duration | 30 days |
+| Timeline range per request | 62 days |
diff --git a/docs/public/on-call/how-the-pieces-fit-together.svg b/docs/public/on-call/how-the-pieces-fit-together.svg
new file mode 100644
index 00000000..6113d8fe
--- /dev/null
+++ b/docs/public/on-call/how-the-pieces-fit-together.svg
@@ -0,0 +1,132 @@
+
+
+
+
+
+
+oncall_pieces
+
+
+rule
+
+an alert rule fires
+error rate · latency · metric threshold · missing data
+
+
+
+channel
+
+escalation channel
+channel type “escalation”, pointing at one policy
+
+
+
+rule->channel
+
+
+
+
+
+page
+
+a page opens
+one incident: re-fires bump its count, nobody is notified twice
+
+
+
+channel->page
+
+
+
+
+
+step
+
+policy step
+level 1, level 2, … each with a delay
+
+
+
+page->step
+
+
+
+
+
+target
+
+target
+schedule · team · user · notification channel
+
+
+
+step->target
+
+
+
+
+
+person
+
+the person on call
+resolved when the step runs, overrides included
+
+
+
+target->person
+
+
+
+
+
+chain
+
+notification rule chain
+their own steps, for this page’s urgency
+
+
+
+person->chain
+
+
+
+
+
+method
+
+contact method
+email · Slack · Pushover · Telegram · SMS
+
+
+
+chain->method
+
+
+
+
+
+method:e->step:e
+
+
+ nobody acknowledged:
+ next level after
+ the step's delay
+
+
+
+ack
+
+acknowledged
+escalation stops, queued deliveries are cancelled
+
+
+
+method->ack
+
+
+
+
+
diff --git a/docs/public/on-call/oncall-ack-link.png b/docs/public/on-call/oncall-ack-link.png
new file mode 100644
index 00000000..adfc1323
Binary files /dev/null and b/docs/public/on-call/oncall-ack-link.png differ
diff --git a/docs/public/on-call/oncall-channel-dialog.png b/docs/public/on-call/oncall-channel-dialog.png
new file mode 100644
index 00000000..b174e71b
Binary files /dev/null and b/docs/public/on-call/oncall-channel-dialog.png differ
diff --git a/docs/public/on-call/oncall-contact-method-dialog.png b/docs/public/on-call/oncall-contact-method-dialog.png
new file mode 100644
index 00000000..7b206d2e
Binary files /dev/null and b/docs/public/on-call/oncall-contact-method-dialog.png differ
diff --git a/docs/public/on-call/oncall-contact-methods.png b/docs/public/on-call/oncall-contact-methods.png
new file mode 100644
index 00000000..bfc0e9d0
Binary files /dev/null and b/docs/public/on-call/oncall-contact-methods.png differ
diff --git a/docs/public/on-call/oncall-layer-editor.png b/docs/public/on-call/oncall-layer-editor.png
new file mode 100644
index 00000000..08c94518
Binary files /dev/null and b/docs/public/on-call/oncall-layer-editor.png differ
diff --git a/docs/public/on-call/oncall-override-dialog.png b/docs/public/on-call/oncall-override-dialog.png
new file mode 100644
index 00000000..f786339d
Binary files /dev/null and b/docs/public/on-call/oncall-override-dialog.png differ
diff --git a/docs/public/on-call/oncall-overview.png b/docs/public/on-call/oncall-overview.png
new file mode 100644
index 00000000..a1416d33
Binary files /dev/null and b/docs/public/on-call/oncall-overview.png differ
diff --git a/docs/public/on-call/oncall-page-detail.png b/docs/public/on-call/oncall-page-detail.png
new file mode 100644
index 00000000..d5d508ca
Binary files /dev/null and b/docs/public/on-call/oncall-page-detail.png differ
diff --git a/docs/public/on-call/oncall-pages.png b/docs/public/on-call/oncall-pages.png
new file mode 100644
index 00000000..26d3cd77
Binary files /dev/null and b/docs/public/on-call/oncall-pages.png differ
diff --git a/docs/public/on-call/oncall-policies.png b/docs/public/on-call/oncall-policies.png
new file mode 100644
index 00000000..be39f476
Binary files /dev/null and b/docs/public/on-call/oncall-policies.png differ
diff --git a/docs/public/on-call/oncall-policy-dialog.png b/docs/public/on-call/oncall-policy-dialog.png
new file mode 100644
index 00000000..55474f06
Binary files /dev/null and b/docs/public/on-call/oncall-policy-dialog.png differ
diff --git a/docs/public/on-call/oncall-schedule-detail.png b/docs/public/on-call/oncall-schedule-detail.png
new file mode 100644
index 00000000..9f24c35c
Binary files /dev/null and b/docs/public/on-call/oncall-schedule-detail.png differ
diff --git a/docs/public/on-call/oncall-schedules.png b/docs/public/on-call/oncall-schedules.png
new file mode 100644
index 00000000..3196ff19
Binary files /dev/null and b/docs/public/on-call/oncall-schedules.png differ
diff --git a/docs/public/on-call/oncall-team-dialog.png b/docs/public/on-call/oncall-team-dialog.png
new file mode 100644
index 00000000..c9c06ebf
Binary files /dev/null and b/docs/public/on-call/oncall-team-dialog.png differ
diff --git a/docs/public/on-call/oncall-teams.png b/docs/public/on-call/oncall-teams.png
new file mode 100644
index 00000000..31bc21f7
Binary files /dev/null and b/docs/public/on-call/oncall-teams.png differ
diff --git a/docs/styles/custom.css b/docs/styles/custom.css
index f27bb650..0dfe164b 100644
--- a/docs/styles/custom.css
+++ b/docs/styles/custom.css
@@ -853,19 +853,30 @@ body[data-page^="protocol"] .nx-mx-auto.nx-flex > div.nx-w-64.nx-shrink-0 {
flex-direction: column;
}
+/* Nextra wraps the nav list in a transition wrapper that is `overflow: hidden`.
+ Sizing it with a plain `flex: 1` pins it to the scroll container's height, so
+ a nav list taller than the viewport gets clipped instead of scrolled: the
+ scroll container sees a child that exactly fits and never grows a scrollbar,
+ and the wheel scrolls the page instead. `1 0 auto` still fills the container
+ when the list is short, but lets the wrapper grow past it when the list is
+ long, and `overflow: visible` hands the overflow to the scroll container. */
.nextra-sidebar-container > :nth-child(2) > :nth-child(1) {
- flex: 1;
+ flex: 1 0 auto;
+ min-height: 0;
+ overflow: visible;
}
/* Nextra's sticky-bottom panels (Edit-on-GitHub at the bottom of the TOC,
collapse toggle at the bottom of the sidebar) ship with an opaque #111
background and a 16px white-ish drop shadow meant to fade scrolling content
beneath them. On Traceway's darker palette that reads as a misaligned box
- with a glow. Drop the background + shadow, keep a hairline border-top so the
- region is still subtly delineated. */
+ with a glow. Drop the shadow and keep a hairline border-top so the region is
+ still subtly delineated. The background stays opaque, just the page colour
+ rather than #111: these panels sit on top of a scrolling list, and a
+ transparent one lets nav items slide visibly underneath them. */
nav.nextra-toc .nx-sticky.nx-bottom-0,
.nextra-sidebar-container .nx-sticky.nx-bottom-0 {
- background: transparent !important;
+ background: var(--ink-0) !important;
box-shadow: none !important;
border-top: 1px solid rgba(255, 255, 255, 0.06) !important;
}
diff --git a/frontend/src/lib/components/app-sidebar.svelte b/frontend/src/lib/components/app-sidebar.svelte
index ba5a69f0..77a1436d 100644
--- a/frontend/src/lib/components/app-sidebar.svelte
+++ b/frontend/src/lib/components/app-sidebar.svelte
@@ -1,6 +1,7 @@
+
+
+ {#each steps as step, i (i)}
+ {#if i > 0}
+
+
+
after {steps[i - 1].delayMinutes}m
+
+ {/if}
+
+
+ L{i + 1}
+
+
+ {#each step.targets as target (target.type + ':' + target.id)}
+ {@const Icon = typeIcons[target.type] ?? Bell}
+
+
+ {targetLabel(target)}
+
+ {/each}
+ {#if step.targets.length === 0}
+ No targets
+ {/if}
+
+
+ {/each}
+ {#if steps.length === 0}
+
No steps
+ {/if}
+ {#if repeatCount > 0}
+
+
+ repeats ×{repeatCount}
+
+ {/if}
+
diff --git a/frontend/src/lib/components/traceway/oncall-owner.svelte b/frontend/src/lib/components/traceway/oncall-owner.svelte
new file mode 100644
index 00000000..7a74c72e
--- /dev/null
+++ b/frontend/src/lib/components/traceway/oncall-owner.svelte
@@ -0,0 +1,64 @@
+
+
+
+
+{#if oncall?.team}
+
+
+
+ On-call: {oncall.team.name} {#if names}
+ {` — ${names}`} {/if}
+
+
+{/if}
diff --git a/frontend/src/lib/components/ui/switch/index.ts b/frontend/src/lib/components/ui/switch/index.ts
new file mode 100644
index 00000000..f5533db7
--- /dev/null
+++ b/frontend/src/lib/components/ui/switch/index.ts
@@ -0,0 +1,7 @@
+import Root from "./switch.svelte";
+
+export {
+ Root,
+ //
+ Root as Switch,
+};
diff --git a/frontend/src/lib/components/ui/switch/switch.svelte b/frontend/src/lib/components/ui/switch/switch.svelte
new file mode 100644
index 00000000..1bc165bf
--- /dev/null
+++ b/frontend/src/lib/components/ui/switch/switch.svelte
@@ -0,0 +1,29 @@
+
+
+
+
+
diff --git a/frontend/src/lib/state/oncall.svelte.ts b/frontend/src/lib/state/oncall.svelte.ts
new file mode 100644
index 00000000..28b5e2c1
--- /dev/null
+++ b/frontend/src/lib/state/oncall.svelte.ts
@@ -0,0 +1,469 @@
+import { api } from '$lib/api';
+import { authState } from './auth.svelte';
+import { projectsState } from './projects.svelte';
+
+export interface TeamMember {
+ teamId: number;
+ userId: number;
+ position: number;
+ name: string;
+ email: string;
+}
+
+export interface TeamProject {
+ projectId: string;
+ name: string;
+}
+
+export interface Team {
+ id: number;
+ organizationId: number;
+ name: string;
+ description: string;
+ createdAt: string;
+ updatedAt: string;
+ memberCount: number;
+ projectCount: number;
+ scheduleCount: number;
+ members: TeamMember[];
+ projects: TeamProject[];
+}
+
+export interface ScheduleRestriction {
+ type: 'daily' | 'weekly';
+ startTime: string;
+ endTime: string;
+ startDay?: number;
+ endDay?: number;
+}
+
+export interface ScheduleLayer {
+ id?: string;
+ name: string;
+ rotationType: 'daily' | 'weekly' | 'custom';
+ handoffTime: string;
+ handoffDay?: number;
+ intervalDays?: number;
+ rotationStart: string;
+ userIds: number[];
+ restrictions: ScheduleRestriction[];
+}
+
+export interface ScheduleDefinition {
+ schemaVersion: number;
+ layers: ScheduleLayer[];
+}
+
+export interface Schedule {
+ id: number;
+ organizationId: number;
+ teamId: number;
+ name: string;
+ description: string;
+ timezone: string;
+ definition: ScheduleDefinition;
+ createdBy: number;
+ createdAt: string;
+ updatedAt: string;
+}
+
+export interface ScheduleOverride {
+ id: number;
+ scheduleId: number;
+ userId: number;
+ startAt: string;
+ endAt: string;
+ createdBy: number;
+ createdAt: string;
+}
+
+export interface TimelineShift {
+ userId: number;
+ layerId: string;
+ start: string;
+ end: string;
+ isOverride: boolean;
+}
+
+export interface TimelineLayer {
+ id: string;
+ name: string;
+ shifts: TimelineShift[];
+}
+
+export interface TimelineUser {
+ name: string;
+ email: string;
+}
+
+export interface TimelineResponse {
+ schedule: { id: number; name: string; timezone: string; teamId: number };
+ from: string;
+ to: string;
+ layers: TimelineLayer[];
+ final: TimelineShift[];
+ users: Record;
+}
+
+export interface OncallUser {
+ userId: number;
+ name: string;
+ email: string;
+}
+
+export interface OverviewSchedule {
+ id: number;
+ name: string;
+ oncall: OncallUser[];
+ until: string | null;
+ nextUp: OncallUser | null;
+ nextAt: string | null;
+}
+
+export interface OverviewTeam {
+ team: {
+ id: number;
+ name: string;
+ description: string;
+ memberCount: number;
+ projectCount: number;
+ scheduleCount: number;
+ };
+ schedules: OverviewSchedule[];
+}
+
+export interface ProjectOncall {
+ team: { id: number; name: string } | null;
+ schedules: { id: number; name: string }[];
+ oncall: OncallUser[];
+}
+
+export interface OrgMember {
+ id: number;
+ email: string;
+ name: string;
+ role: string;
+}
+
+export type PolicyTargetType = 'schedule' | 'user' | 'team' | 'channel';
+
+export interface PolicyTarget {
+ type: PolicyTargetType;
+ id: number;
+}
+
+export interface PolicyStep {
+ targets: PolicyTarget[];
+ delayMinutes: number;
+}
+
+export interface PolicyDefinition {
+ schemaVersion: number;
+ steps: PolicyStep[];
+ repeatCount: number;
+ urgency?: 'auto' | 'high' | 'low' | '';
+}
+
+export interface EscalationPolicy {
+ id: number;
+ organizationId: number;
+ name: string;
+ definition: PolicyDefinition;
+ createdBy: number;
+ createdAt: string;
+ updatedAt: string;
+}
+
+export interface OncallPage {
+ id: number;
+ organizationId: number;
+ projectId: string;
+ policyId: number;
+ policySnapshot: PolicyDefinition;
+ ruleId: number;
+ ruleName: string;
+ ruleType: string;
+ subject: string;
+ body: string;
+ url: string;
+ severity: string;
+ urgency: '' | 'high' | 'low';
+ status: 'open' | 'acknowledged' | 'resolved';
+ eventCount: number;
+ lastEventAt: string;
+ escalationLevel: number;
+ repeatIteration: number;
+ nextEscalationAt: string | null;
+ lastEscalatedAt: string | null;
+ acknowledgedBy: number | null;
+ acknowledgedAt: string | null;
+ acknowledgedVia: '' | 'dashboard' | 'link';
+ resolvedBy: number | null;
+ resolvedAt: string | null;
+ createdAt: string;
+ updatedAt: string;
+}
+
+export interface PageNotification {
+ id: number;
+ pageId: number;
+ level: number;
+ iteration: number;
+ userId: number | null;
+ targetDesc: string;
+ methodType: string;
+ status: 'pending' | 'sent' | 'failed' | 'cancelled';
+ errorMsg: string;
+ createdAt: string;
+ sentAt: string | null;
+ scheduledFor: string | null;
+}
+
+export interface PageDetailResponse {
+ page: OncallPage;
+ notifications: PageNotification[];
+ users: Record;
+}
+
+export interface ContactMethod {
+ id: number;
+ userId: number;
+ methodType: 'email' | 'slack' | 'pushover' | 'telegram' | 'sms';
+ config: Record;
+ enabled: boolean;
+ verified: boolean;
+ createdAt: string;
+}
+
+export interface UserNotificationRuleStep {
+ id?: number;
+ contactMethodId: number;
+ delayMinutes: number;
+}
+
+export function getCurrentUserFromToken(): { userId: number; email: string } | null {
+ const token = authState.token;
+ if (!token) return null;
+ try {
+ const payload = JSON.parse(atob(token.split('.')[1].replace(/-/g, '+').replace(/_/g, '/')));
+ if (typeof payload.userId !== 'number') return null;
+ return { userId: payload.userId, email: payload.email ?? '' };
+ } catch {
+ return null;
+ }
+}
+
+class OncallState {
+ teams = $state([]);
+ teamsLoading = $state(false);
+ teamsError = $state('');
+ schedules = $state([]);
+ schedulesLoading = $state(false);
+ schedulesError = $state('');
+ overview = $state([]);
+ overviewLoading = $state(false);
+ overviewError = $state('');
+ policies = $state([]);
+ policiesLoading = $state(false);
+ policiesError = $state('');
+ openPagesCount = $state(0);
+
+ canManage(organizationId: number): boolean {
+ return authState.canManageOrganization(organizationId);
+ }
+
+ async refreshOpenCount() {
+ try {
+ const res = await api.get('/pages/open-count', {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ this.openPagesCount = res.count ?? 0;
+ } catch {
+ // badge stays at its last value
+ }
+ }
+
+ async loadTeams(organizationId: number) {
+ this.teamsLoading = true;
+ this.teamsError = '';
+ try {
+ const res = await api.get(`/organizations/${organizationId}/teams`);
+ this.teams = res.teams || [];
+ } catch (e: unknown) {
+ this.teamsError = e instanceof Error ? e.message : 'Failed to load teams';
+ this.teams = [];
+ } finally {
+ this.teamsLoading = false;
+ }
+ }
+
+ async loadSchedules(organizationId: number) {
+ this.schedulesLoading = true;
+ this.schedulesError = '';
+ try {
+ const res = await api.get(`/organizations/${organizationId}/schedules`);
+ this.schedules = res.schedules || [];
+ } catch (e: unknown) {
+ this.schedulesError = e instanceof Error ? e.message : 'Failed to load schedules';
+ this.schedules = [];
+ } finally {
+ this.schedulesLoading = false;
+ }
+ }
+
+ async loadOverview(organizationId: number) {
+ this.overviewLoading = true;
+ this.overviewError = '';
+ try {
+ const res = await api.get(`/organizations/${organizationId}/oncall/now`);
+ this.overview = res.teams || [];
+ } catch (e: unknown) {
+ this.overviewError = e instanceof Error ? e.message : 'Failed to load who is on call';
+ this.overview = [];
+ } finally {
+ this.overviewLoading = false;
+ }
+ }
+
+ async createTeam(
+ organizationId: number,
+ body: { name: string; description: string; memberUserIds: number[]; projectIds: string[] }
+ ) {
+ const team = await api.post(`/organizations/${organizationId}/teams`, body);
+ await this.loadTeams(organizationId);
+ return team;
+ }
+
+ async updateTeam(
+ organizationId: number,
+ teamId: number,
+ body: {
+ name: string;
+ description: string;
+ memberUserIds?: number[];
+ projectIds?: string[];
+ }
+ ) {
+ await api.put(`/organizations/${organizationId}/teams/${teamId}`, body);
+ }
+
+ async updateTeamMembers(organizationId: number, teamId: number, userIds: number[]) {
+ await api.put(`/organizations/${organizationId}/teams/${teamId}/members`, { userIds });
+ }
+
+ async updateTeamProjects(organizationId: number, teamId: number, projectIds: string[]) {
+ await api.put(`/organizations/${organizationId}/teams/${teamId}/projects`, { projectIds });
+ }
+
+ async deleteTeam(organizationId: number, teamId: number) {
+ await api.delete(`/organizations/${organizationId}/teams/${teamId}`);
+ await this.loadTeams(organizationId);
+ }
+
+ async createSchedule(
+ organizationId: number,
+ body: {
+ teamId: number;
+ name: string;
+ description: string;
+ timezone: string;
+ definition: ScheduleDefinition;
+ }
+ ) {
+ const schedule = await api.post(`/organizations/${organizationId}/schedules`, body);
+ await this.loadSchedules(organizationId);
+ return schedule;
+ }
+
+ async updateSchedule(
+ organizationId: number,
+ scheduleId: number,
+ body: {
+ teamId: number;
+ name: string;
+ description: string;
+ timezone: string;
+ definition: ScheduleDefinition;
+ }
+ ) {
+ await api.put(`/organizations/${organizationId}/schedules/${scheduleId}`, body);
+ await this.loadSchedules(organizationId);
+ }
+
+ async deleteSchedule(organizationId: number, scheduleId: number) {
+ await api.delete(`/organizations/${organizationId}/schedules/${scheduleId}`);
+ await this.loadSchedules(organizationId);
+ }
+
+ async getSchedule(
+ organizationId: number,
+ scheduleId: number
+ ): Promise<{ schedule: Schedule; overrides: ScheduleOverride[] }> {
+ return await api.get(`/organizations/${organizationId}/schedules/${scheduleId}`);
+ }
+
+ async getTimeline(
+ organizationId: number,
+ scheduleId: number,
+ from: string,
+ to: string
+ ): Promise {
+ return await api.get(
+ `/organizations/${organizationId}/schedules/${scheduleId}/timeline?from=${encodeURIComponent(from)}&to=${encodeURIComponent(to)}`
+ );
+ }
+
+ async createOverride(
+ organizationId: number,
+ scheduleId: number,
+ body: { userId: number; startAt: string; endAt: string }
+ ) {
+ return await api.post(`/organizations/${organizationId}/schedules/${scheduleId}/overrides`, body);
+ }
+
+ async deleteOverride(organizationId: number, scheduleId: number, overrideId: number) {
+ await api.delete(
+ `/organizations/${organizationId}/schedules/${scheduleId}/overrides/${overrideId}`
+ );
+ }
+
+ async getMembers(organizationId: number): Promise {
+ return await api.get(`/organizations/${organizationId}/members`);
+ }
+
+ async loadPolicies(organizationId: number) {
+ this.policiesLoading = true;
+ this.policiesError = '';
+ try {
+ const res = await api.get(`/organizations/${organizationId}/escalation-policies`);
+ this.policies = res.policies || [];
+ } catch (e: unknown) {
+ this.policiesError = e instanceof Error ? e.message : 'Failed to load escalation policies';
+ this.policies = [];
+ } finally {
+ this.policiesLoading = false;
+ }
+ }
+
+ async createPolicy(organizationId: number, body: { name: string; definition: PolicyDefinition }) {
+ const policy = await api.post(`/organizations/${organizationId}/escalation-policies`, body);
+ await this.loadPolicies(organizationId);
+ return policy;
+ }
+
+ async updatePolicy(
+ organizationId: number,
+ policyId: number,
+ body: { name: string; definition: PolicyDefinition }
+ ) {
+ await api.put(`/organizations/${organizationId}/escalation-policies/${policyId}`, body);
+ await this.loadPolicies(organizationId);
+ }
+
+ async deletePolicy(organizationId: number, policyId: number) {
+ await api.delete(`/organizations/${organizationId}/escalation-policies/${policyId}`);
+ await this.loadPolicies(organizationId);
+ }
+}
+
+export const oncallState = new OncallState();
diff --git a/frontend/src/lib/utils/formatters.ts b/frontend/src/lib/utils/formatters.ts
index c44334d5..854f28b0 100644
--- a/frontend/src/lib/utils/formatters.ts
+++ b/frontend/src/lib/utils/formatters.ts
@@ -63,6 +63,11 @@ export function formatRelativeTime(dateStr: string, timezone?: string): string {
return `${diffDays}d`;
}
+export function formatRelativeTimeAgo(dateStr: string, timezone?: string): string {
+ const relative = formatRelativeTime(dateStr, timezone);
+ return relative === 'just now' ? relative : `${relative} ago`;
+}
+
export type DateTimeFormat = 'full' | 'short' | 'date' | 'time' | 'datetime' | 'iso';
export function formatDateTime(
diff --git a/frontend/src/routes/+layout.svelte b/frontend/src/routes/+layout.svelte
index 7a4b2a84..c3084b65 100644
--- a/frontend/src/routes/+layout.svelte
+++ b/frontend/src/routes/+layout.svelte
@@ -87,7 +87,11 @@
]);
function isPublicPath(pathname: string): boolean {
- return PUBLIC_PATHS.has(pathname) || pathname.startsWith('/accept-invitation');
+ return (
+ PUBLIC_PATHS.has(pathname) ||
+ pathname.startsWith('/accept-invitation') ||
+ pathname.startsWith('/ack/')
+ );
}
// An unauthenticated visit to a protected path matches neither layout
diff --git a/frontend/src/routes/account/+page.svelte b/frontend/src/routes/account/+page.svelte
index 23c7d28e..dde46d95 100644
--- a/frontend/src/routes/account/+page.svelte
+++ b/frontend/src/routes/account/+page.svelte
@@ -1,5 +1,9 @@
@@ -7,4 +11,8 @@
Account
+ (methodsVersion += 1)} />
+ {#key methodsVersion}
+
+ {/key}
diff --git a/frontend/src/routes/account/contact-method-dialog.svelte b/frontend/src/routes/account/contact-method-dialog.svelte
new file mode 100644
index 00000000..65a60c3e
--- /dev/null
+++ b/frontend/src/routes/account/contact-method-dialog.svelte
@@ -0,0 +1,237 @@
+
+
+ (open = isOpen)}>
+
+
+ {isEditing ? 'Edit Contact Method' : 'New Contact Method'}
+
+ {isEditing
+ ? 'Change where Traceway reaches you when you\'re paged'
+ : "Add a way for Traceway to reach you when you're paged"}
+
+
+
+
+
+
+ Cancel
+
+ {#if isEditing}
+
+ {loading ? 'Updating...' : 'Update Contact Method'}
+ {:else}
+
+ {loading ? 'Creating...' : 'New Contact Method'}
+ {/if}
+
+
+
+
diff --git a/frontend/src/routes/account/contact-methods.svelte b/frontend/src/routes/account/contact-methods.svelte
new file mode 100644
index 00000000..e18811aa
--- /dev/null
+++ b/frontend/src/routes/account/contact-methods.svelte
@@ -0,0 +1,371 @@
+
+
+
+
Contact Methods
+
(showCreate = true)}>
+ New Contact Method
+
+
+
+
+
+
+ When you're paged, Traceway notifies every enabled method. With none configured, your account
+ email is used.
+
+
+
+{#if loading}
+
+{:else if loadError}
+
+
{loadError}
+
loadMethods()}>Retry
+
+{:else if methods.length === 0}
+
+
No contact methods yet. Pages fall back to your account email.
+
(showCreate = true)}>
+
+ Create your first Contact Method
+
+
+{:else}
+
+
+
+
+ Method
+ Destination
+ Enabled
+ Actions
+
+
+
+ {#each methods as method (method.id)}
+ {@const Icon = methodTypeIcons[method.methodType] ?? Bell}
+
+
+
+
+ {methodTypeLabels[method.methodType] ?? method.methodType}
+
+
+
+
+ {methodSummary(method)}
+ {#if isUnsendableSms(method)}
+
+ SMS unavailable
+
+ {:else if isUnverifiedSms(method)}
+ Unverified
+ {/if}
+
+
+
+ toggleEnabled(method, checked)}
+ />
+
+
+
+ {#if isUnverifiedSms(method) && !isUnsendableSms(method)}
+
openVerify(method)}
+ >
+
+
+ {/if}
+
+ testMethod(method)}
+ >
+
+
+
+
{
+ methodToEdit = method;
+ showEdit = true;
+ }}
+ >
+
+
+
{
+ deleteError = '';
+ methodToDelete = method;
+ }}
+ >
+
+
+
+
+
+ {/each}
+
+
+
+{/if}
+
+ {
+ showCreate = false;
+ loadMethods();
+ if (created && created.methodType === 'sms' && created.verified === false) {
+ openVerify(created, true);
+ }
+ }}
+/>
+
+ {
+ showEdit = false;
+ methodToEdit = null;
+ loadMethods();
+ if (updated && updated.methodType === 'sms' && updated.verified === false) {
+ openVerify(updated, true);
+ }
+ }}
+/>
+
+ loadMethods()}
+/>
+
+ {
+ if (!open) {
+ methodToDelete = null;
+ deleteError = '';
+ }
+ }}
+>
+
+
+ Delete Contact Method
+
+ Are you sure you want to delete this {methodTypeLabels[methodToDelete?.methodType ?? ''] ??
+ ''} contact method? You will no longer be paged through it, and any step of your
+ notification rules that uses it is removed too. To change where it points, edit it instead.
+
+
+
+
+ Cancel
+
+
+ {deleting ? 'Deleting...' : 'Delete Contact Method'}
+
+
+
+
diff --git a/frontend/src/routes/account/notification-rules.svelte b/frontend/src/routes/account/notification-rules.svelte
new file mode 100644
index 00000000..11ee0ea7
--- /dev/null
+++ b/frontend/src/routes/account/notification-rules.svelte
@@ -0,0 +1,264 @@
+
+
+{#snippet chain(title: string, steps: StepDraft[], setSteps: (next: StepDraft[]) => void)}
+
+
{title}
+ {#if steps.length === 0}
+
+ No steps — all your contact methods are notified immediately.
+
+ {/if}
+
+ {#each steps as step, index (index)}
+
+
{index + 1}.
+
Notify
+
{
+ if (val) {
+ step.contactMethodId = Number(val);
+ setSteps([...steps]);
+ }
+ }}
+ >
+
+ {selectedLabel(step)}
+
+
+ {#each methods as method (method.id)}
+ {@const unavailable = methodUnavailable(method)}
+
+ {methodLabel(method)}{unavailable ? ` ${unavailable}` : ''}
+
+ {/each}
+
+
+
after
+
+
minutes
+
+ setSteps(moveStep(steps, index, -1))}
+ >
+
+
+ setSteps(moveStep(steps, index, 1))}
+ >
+
+
+ setSteps(removeStep(steps, index))}
+ >
+
+
+
+
+ {/each}
+
+
= 10}
+ onclick={() => setSteps(addStep(steps))}
+ >
+ Add step
+
+
+{/snippet}
+
+
+
Notification Rules
+
+
+
+ When a page targets you, these steps run for its urgency until you acknowledge. With no steps,
+ all your contact methods are notified immediately.
+
+
+{#if loading}
+
+{:else if loadError}
+
+
{loadError}
+
load()}>Retry
+
+{:else}
+
+
+
+ {@render chain('High urgency', highSteps, (next) => (highSteps = next))}
+ {@render chain('Low urgency', lowSteps, (next) => (lowSteps = next))}
+
+
+
+
+ {saving ? 'Updating...' : 'Update Notification Rules'}
+
+
+
+{/if}
diff --git a/frontend/src/routes/account/verify-code-dialog.svelte b/frontend/src/routes/account/verify-code-dialog.svelte
new file mode 100644
index 00000000..03bc9033
--- /dev/null
+++ b/frontend/src/routes/account/verify-code-dialog.svelte
@@ -0,0 +1,147 @@
+
+
+ (open = isOpen)}>
+
+
+ Verify Phone Number
+
+ {codeJustSent
+ ? `Enter the 6-digit code we texted to ${phoneLabel}`
+ : `Enter the 6-digit code sent to ${phoneLabel}, or request a new one`}
+
+
+
+
+
+
+ Cancel
+
+
+ {loading ? 'Verifying...' : 'Verify Phone Number'}
+
+
+
+
diff --git a/frontend/src/routes/ack/[token]/+page.svelte b/frontend/src/routes/ack/[token]/+page.svelte
new file mode 100644
index 00000000..14776bea
--- /dev/null
+++ b/frontend/src/routes/ack/[token]/+page.svelte
@@ -0,0 +1,195 @@
+
+
+
+
+
+
+
+ {#if themeState.isDark}
+
+ {:else}
+
+ {/if}
+
+
+ On-call page
+
+
+ {#if status === 'loading'}
+
+ {:else if status === 'invalid'}
+
+ This link is no longer valid. The page may have been resolved.
+
+ {:else if status === 'justAcknowledged'}
+
+
+
+
+
Acknowledged
+
The on-call escalation has stopped.
+
+ {:else if (status === 'ready' || status === 'acknowledging') && info}
+
+
+
+
+
{info.subject || 'Page'}
+
+ {#if info.severity === 'critical'}
+ Critical
+ {:else if info.severity === 'warning'}
+ Warning
+ {:else if info.severity === 'info'}
+ Info
+ {/if}
+ {#if info.urgency === 'high'}
+ High urgency
+ {:else if info.urgency === 'low'}
+ Low urgency
+ {/if}
+
+
+
+
+
+
Project
+
{info.projectName || '—'}
+
+
+
Rule
+
{info.ruleName || '—'}
+
+
+
Events
+
{info.eventCount}
+
+
+
Opened
+
{formatTime(info.createdAt)}
+
+
+
+ {#if info.body}
+
+ {info.body}
+
+ {/if}
+
+ {#if info.status === 'acknowledged'}
+
+
+ Acknowledged{info.acknowledgedByName ? ` by ${info.acknowledgedByName}` : ''}
+ {info.acknowledgedAt ? ` at ${formatTime(info.acknowledgedAt)}` : ''}
+
+ {:else}
+
+
+ {status === 'acknowledging' ? 'Acknowledging...' : 'Acknowledge'}
+
+ {/if}
+
+ {:else}
+
+ {/if}
+
+
+
diff --git a/frontend/src/routes/ack/[token]/+page.ts b/frontend/src/routes/ack/[token]/+page.ts
new file mode 100644
index 00000000..d671dfe2
--- /dev/null
+++ b/frontend/src/routes/ack/[token]/+page.ts
@@ -0,0 +1,7 @@
+import type { PageLoad } from './$types';
+
+export const prerender = false;
+
+export const load: PageLoad = ({ params }) => {
+ return { token: params.token };
+};
diff --git a/frontend/src/routes/issues/[exceptionHash]/+page.svelte b/frontend/src/routes/issues/[exceptionHash]/+page.svelte
index d7769584..32c048fb 100644
--- a/frontend/src/routes/issues/[exceptionHash]/+page.svelte
+++ b/frontend/src/routes/issues/[exceptionHash]/+page.svelte
@@ -10,6 +10,7 @@
import { StackTraceCard, EventCard, EventsTable, PageHeader } from '$lib/components/issues';
import { toast } from 'svelte-sonner';
import ArchiveConfirmationDialog from '$lib/components/archive-confirmation-dialog.svelte';
+ import OncallOwner from '$lib/components/traceway/oncall-owner.svelte';
import Archive from '@lucide/svelte/icons/archive';
import type {
ExceptionGroup,
@@ -146,6 +147,7 @@
subtitle="Exception Hash: {exceptionHash}"
onBack={createSmartBackHandler({ fallbackPath: resolve('/issues') })}
/>
+
{#if loading && !group}
diff --git a/frontend/src/routes/layout.css b/frontend/src/routes/layout.css
index 1b55fc5f..39f62d17 100644
--- a/frontend/src/routes/layout.css
+++ b/frontend/src/routes/layout.css
@@ -194,6 +194,35 @@
font-family: var(--font-body);
}
+ /* Native date/time inputs put their picker button straight after the text,
+ so it floats in the middle of a wide field and gets clipped in a narrow
+ one. Pin it to the trailing edge and keep it out of the text's way. */
+ input[type='time']::-webkit-calendar-picker-indicator,
+ input[type='date']::-webkit-calendar-picker-indicator,
+ input[type='datetime-local']::-webkit-calendar-picker-indicator,
+ input[type='month']::-webkit-calendar-picker-indicator,
+ input[type='week']::-webkit-calendar-picker-indicator {
+ margin-inline-start: auto;
+ padding-inline-start: 0.5rem;
+ opacity: 0.5;
+ cursor: pointer;
+ }
+
+ input[type='time']:hover::-webkit-calendar-picker-indicator,
+ input[type='date']:hover::-webkit-calendar-picker-indicator,
+ input[type='datetime-local']:hover::-webkit-calendar-picker-indicator,
+ input[type='month']:hover::-webkit-calendar-picker-indicator,
+ input[type='week']:hover::-webkit-calendar-picker-indicator {
+ opacity: 0.85;
+ }
+
+ /* The text half must never be squeezed into an ellipsis by the button. */
+ input[type='time']::-webkit-datetime-edit,
+ input[type='date']::-webkit-datetime-edit,
+ input[type='datetime-local']::-webkit-datetime-edit {
+ flex: 0 0 auto;
+ }
+
body {
font-family: var(--font-body);
font-feature-settings: 'cv02', 'cv03', 'cv04', 'cv11', 'ss01', 'ss03';
diff --git a/frontend/src/routes/notifications/+page.svelte b/frontend/src/routes/notifications/+page.svelte
index 23cf52ec..997c0d73 100644
--- a/frontend/src/routes/notifications/+page.svelte
+++ b/frontend/src/routes/notifications/+page.svelte
@@ -82,7 +82,8 @@
slack: 'Slack',
github: 'GitHub',
pushover: 'Pushover',
- telegram: 'Telegram'
+ telegram: 'Telegram',
+ escalation: 'Escalation policy'
};
const tabDescriptions: Record = {
@@ -724,6 +725,10 @@
Sent
{:else if item.status === 'failed'}
Failed
+ {:else if item.status === 'deduped'}
+
+ Deduped
+
{:else}
Skipped
{/if}
@@ -842,7 +847,13 @@
Test Channel
- Send a test notification to "{testingChannel?.name}"? This will deliver a test message through the configured channel.
+ {#if testingChannel?.channelType === 'escalation'}
+ Testing "{testingChannel?.name}" opens a real page and notifies whoever is on call, exactly
+ as a firing rule would. Resolve the page from On-Call when you are done.
+ {:else}
+ Send a test notification to "{testingChannel?.name}"? This will deliver a test message
+ through the configured channel.
+ {/if}
{#if testError}
diff --git a/frontend/src/routes/notifications/channel-dialog.svelte b/frontend/src/routes/notifications/channel-dialog.svelte
index 662def63..825b6756 100644
--- a/frontend/src/routes/notifications/channel-dialog.svelte
+++ b/frontend/src/routes/notifications/channel-dialog.svelte
@@ -57,6 +57,9 @@
let pushoverTtl = $state(0);
let telegramBotToken = $state('');
let telegramChatId = $state('');
+ let escalationPolicyId = $state(null);
+ let escalationPolicies = $state<{ id: number; name: string }[]>([]);
+ let escalationPoliciesLoaded = $state(false);
const isEditing = $derived(channel !== null);
@@ -66,9 +69,29 @@
{ value: 'slack', label: 'Slack' },
{ value: 'github', label: 'GitHub' },
{ value: 'pushover', label: 'Pushover' },
- { value: 'telegram', label: 'Telegram' }
+ { value: 'telegram', label: 'Telegram' },
+ { value: 'escalation', label: 'Escalation policy' }
];
+ async function loadEscalationPolicies() {
+ try {
+ const res = await api.get('/escalation-policies', {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ escalationPolicies = res.policies || [];
+ } catch {
+ escalationPolicies = [];
+ } finally {
+ escalationPoliciesLoaded = true;
+ }
+ }
+
+ $effect(() => {
+ if (open && channelType === 'escalation' && !escalationPoliciesLoaded) {
+ loadEscalationPolicies();
+ }
+ });
+
function resetForm() {
name = '';
channelType = 'email';
@@ -97,6 +120,8 @@
pushoverTtl = 0;
telegramBotToken = '';
telegramChatId = '';
+ escalationPolicyId = null;
+ escalationPoliciesLoaded = false;
}
function populateFromChannel(ch: NotificationChannel) {
@@ -139,6 +164,8 @@
} else if (ch.channelType === 'telegram') {
telegramBotToken = config.botToken || '';
telegramChatId = config.chatId || '';
+ } else if (ch.channelType === 'escalation') {
+ escalationPolicyId = config.policyId ?? null;
}
}
@@ -193,6 +220,8 @@
botToken: telegramBotToken,
chatId: telegramChatId
};
+ } else if (channelType === 'escalation') {
+ return { policyId: escalationPolicyId };
}
return {};
}
@@ -526,6 +555,39 @@
required
/>
+ {:else if channelType === 'escalation'}
+
+
Escalation Policy
+ {#if escalationPoliciesLoaded && escalationPolicies.length === 0}
+
+ No escalation policies yet — create one on the
+ On-Call page .
+
+ {:else}
+
{
+ if (val) escalationPolicyId = Number(val);
+ }}
+ >
+
+ {escalationPolicies.find((p) => p.id === escalationPolicyId)?.name ??
+ 'Select policy'}
+
+
+ {#each escalationPolicies as policy (policy.id)}
+ {policy.name}
+ {/each}
+
+
+ {/if}
+
+ Testing an escalation channel opens a real page and notifies the on-call responder.
+
+
{/if}
diff --git a/frontend/src/routes/on-call/+page.svelte b/frontend/src/routes/on-call/+page.svelte
new file mode 100644
index 00000000..adf68029
--- /dev/null
+++ b/frontend/src/routes/on-call/+page.svelte
@@ -0,0 +1,134 @@
+
+
+
+
+
On-Call
+ {#if orgs.length > 1}
+
+ Organization
+ {
+ if (val) selectedOrgId = Number(val);
+ }}
+ >
+
+ {currentOrganizationName || 'Select organization'}
+
+
+ {#each orgs as org (org.id)}
+ {org.name}
+ {/each}
+
+
+
+ {/if}
+
+
+
{
+ if (v) setTab(v);
+ }}
+ >
+
+ Pages
+ Overview
+ Teams
+ Schedules
+ Policies
+
+
+
+ {#if currentOrganizationId === null}
+
+
You are not a member of any organization yet.
+
+ {:else if activeTab === 'pages'}
+
+ {:else if activeTab === 'overview'}
+
setTab('teams')} />
+ {:else if activeTab === 'teams'}
+
+ {:else if activeTab === 'schedules'}
+
+ {:else if activeTab === 'policies'}
+
+ {/if}
+
diff --git a/frontend/src/routes/on-call/layer-editor.svelte b/frontend/src/routes/on-call/layer-editor.svelte
new file mode 100644
index 00000000..ed0402c6
--- /dev/null
+++ b/frontend/src/routes/on-call/layer-editor.svelte
@@ -0,0 +1,417 @@
+
+
+
+
+
Layers
+
+ Later layers take precedence over earlier ones. If a handoff falls inside a restriction
+ window, the person changes mid-window.
+
+
+
+
+
+ {#each layers as layer, layerIndex (layerIndex)}
+
+
+
+
+
moveLayer(layerIndex, -1)}
+ >
+
+
+
moveLayer(layerIndex, 1)}
+ >
+
+
+
removeLayer(layerIndex)}
+ >
+
+
+
+
+
+
+ Rotation
+ {
+ if (val) layer.rotationType = val as ScheduleLayer['rotationType'];
+ }}
+ >
+
+ {rotationOptions.find((o) => o.value === layer.rotationType)?.label}
+
+
+ {#each rotationOptions as option (option.value)}
+ {option.label}
+ {/each}
+
+
+
+
+ Handoff time
+
+
+ {#if layer.rotationType === 'weekly'}
+
+ Handoff day
+ {
+ if (val) layer.handoffDay = Number(val);
+ }}
+ >
+ {dayLabel(layer.handoffDay)}
+
+ {#each dayOptions as option (option.value)}
+ {option.label}
+ {/each}
+
+
+
+ {:else if layer.rotationType === 'custom'}
+
+ Interval (days)
+
+
+ {/if}
+
+ Rotation start
+
+
+
+
+
+
Members (rotation order)
+ {#each layer.userIds as userId, memberIndex (memberIndex)}
+
+ {memberIndex + 1}.
+ {memberLabel(userId)}
+ moveLayerMember(layer, memberIndex, -1)}
+ >
+
+
+ moveLayerMember(layer, memberIndex, 1)}
+ >
+
+
+ removeLayerMember(layer, memberIndex)}
+ >
+
+
+
+ {:else}
+
No members in this layer yet.
+ {/each}
+ {#if members.filter((m) => !layer.userIds.includes(m.id)).length > 0}
+
{
+ if (val) addLayerMember(layer, Number(val));
+ }}
+ >
+ Add member...
+
+ {#each members.filter((m) => !layer.userIds.includes(m.id)) as member (member.id)}
+
+ {member.name || member.email}
+
+ {/each}
+
+
+ {/if}
+
+
+
+
Restrictions
+ {#each layer.restrictions as restriction, restrictionIndex (restrictionIndex)}
+
+ {
+ if (val) setRestrictionType(restriction, val as 'daily' | 'weekly');
+ }}
+ >
+
+ {restriction.type === 'daily' ? 'Daily' : 'Weekly'}
+
+
+ Daily
+ Weekly
+
+
+ {#if restriction.type === 'weekly'}
+ {
+ if (val) restriction.startDay = Number(val);
+ }}
+ >
+ {dayLabel(restriction.startDay)}
+
+ {#each dayOptions as option (option.value)}
+ {option.label}
+ {/each}
+
+
+ {/if}
+
+ to
+ {#if restriction.type === 'weekly'}
+ {
+ if (val) restriction.endDay = Number(val);
+ }}
+ >
+ {dayLabel(restriction.endDay)}
+
+ {#each dayOptions as option (option.value)}
+ {option.label}
+ {/each}
+
+
+ {/if}
+
+ removeRestriction(layer, restrictionIndex)}
+ >
+
+
+
+ {:else}
+
No restrictions. The layer covers all hours.
+ {/each}
+
addRestriction(layer)}>
+ Add Restriction
+
+
+
+ {/each}
+
+
+
+ Add Layer
+
+
+ Cancel
+
+
+ {loading ? 'Updating...' : 'Update Schedule'}
+
+
+
+
diff --git a/frontend/src/routes/on-call/override-dialog.svelte b/frontend/src/routes/on-call/override-dialog.svelte
new file mode 100644
index 00000000..5b993008
--- /dev/null
+++ b/frontend/src/routes/on-call/override-dialog.svelte
@@ -0,0 +1,146 @@
+
+
+ (open = isOpen)}>
+
+
+ Add Override
+
+ Temporarily put someone on call for "{schedule.name}". Overrides can cover up to 30 days.
+
+
+
+
+
+
+ Cancel
+
+
+ {loading ? 'Creating...' : 'New Override'}
+
+
+
+
diff --git a/frontend/src/routes/on-call/overview-tab.svelte b/frontend/src/routes/on-call/overview-tab.svelte
new file mode 100644
index 00000000..52f290e3
--- /dev/null
+++ b/frontend/src/routes/on-call/overview-tab.svelte
@@ -0,0 +1,127 @@
+
+
+{#if oncallState.overviewLoading}
+
+{:else if oncallState.overviewError}
+
+
{oncallState.overviewError}
+
oncallState.loadOverview(organizationId)}>
+ Retry
+
+
+{:else if overview.length === 0}
+
+
Create a team to get started.
+
Go to Teams
+
+{:else}
+
+ {#each overview as entry (entry.team.id)}
+
+
+ {entry.team.name}
+ {#if entry.team.description}
+ {entry.team.description}
+ {/if}
+
+
+ {#if entry.schedules.length === 0}
+ No schedules yet.
+ {:else}
+ {#each entry.schedules as schedule (schedule.id)}
+
+
+
+ {schedule.name}
+
+ {#if schedule.oncall.length === 0}
+
No one is on call right now.
+ {:else}
+
+ {#each schedule.oncall as user (user.userId)}
+
+
+ {initials(user)}
+
+
{user.name || user.email}
+
+ {/each}
+ {#if schedule.until}
+
+ until {formatDateTime(schedule.until, { format: 'short' })}
+
+ {/if}
+
+ {/if}
+ {#if schedule.nextUp}
+
+ Next: {schedule.nextUp.name || schedule.nextUp.email}{schedule.nextAt
+ ? ` at ${formatDateTime(schedule.nextAt, { format: 'short' })}`
+ : ''}
+
+ {/if}
+
+ {/each}
+ {/if}
+ {#if teamProjects(entry.team.id).length > 0}
+
+
+ {#each teamProjects(entry.team.id) as project (project.projectId)}
+ {project.name}
+ {/each}
+
+ {/if}
+
+
+ {/each}
+
+{/if}
diff --git a/frontend/src/routes/on-call/page-actions.ts b/frontend/src/routes/on-call/page-actions.ts
new file mode 100644
index 00000000..2d2558f7
--- /dev/null
+++ b/frontend/src/routes/on-call/page-actions.ts
@@ -0,0 +1,36 @@
+import { toast } from 'svelte-sonner';
+import { api } from '$lib/api';
+import { projectsState } from '$lib/state/projects.svelte';
+import { oncallState } from '$lib/state/oncall.svelte';
+
+async function conflictStatus(id: number): Promise {
+ try {
+ const res = await api.get(`/pages/${id}`, {
+ projectId: projectsState.currentProjectId ?? undefined
+ });
+ return res?.page?.status ?? 'unknown';
+ } catch {
+ return 'unknown';
+ }
+}
+
+// Runs an acknowledge/resolve call with the shared toast handling and sidebar
+// badge refresh; callers refresh their own views afterwards.
+export async function runPageAction(id: number, action: 'acknowledge' | 'resolve'): Promise {
+ const done = action === 'acknowledge' ? 'acknowledged' : 'resolved';
+ try {
+ await api.post(
+ `/pages/${id}/${action}`,
+ {},
+ { projectId: projectsState.currentProjectId ?? undefined }
+ );
+ toast.success(`Page ${done}`, { position: 'top-center' });
+ } catch (e: unknown) {
+ if ((e as { status?: number }).status === 409) {
+ toast.error(`Page is already ${await conflictStatus(id)}`, { position: 'top-center' });
+ } else {
+ toast.error(`Failed to ${action} page`, { position: 'top-center' });
+ }
+ }
+ oncallState.refreshOpenCount();
+}
diff --git a/frontend/src/routes/on-call/page-badges.svelte b/frontend/src/routes/on-call/page-badges.svelte
new file mode 100644
index 00000000..9c3323d7
--- /dev/null
+++ b/frontend/src/routes/on-call/page-badges.svelte
@@ -0,0 +1,37 @@
+
+
+{#if severity === 'critical'}
+ Critical
+{:else if severity === 'warning'}
+ Warning
+{:else if severity === 'info'}
+ Info
+{:else if fallback}
+ —
+{/if}
+
+{#if urgency === 'high'}
+ {longUrgency ? 'High urgency' : 'High'}
+{:else if urgency === 'low'}
+ {longUrgency ? 'Low urgency' : 'Low'}
+{/if}
+
+{#if status === 'open'}
+ Open
+{:else if status === 'acknowledged'}
+ Acknowledged
+{:else if status === 'resolved'}
+ Resolved
+{/if}
diff --git a/frontend/src/routes/on-call/page-detail-sheet.svelte b/frontend/src/routes/on-call/page-detail-sheet.svelte
new file mode 100644
index 00000000..36f18bce
--- /dev/null
+++ b/frontend/src/routes/on-call/page-detail-sheet.svelte
@@ -0,0 +1,291 @@
+
+
+
+
+ {#if loading && !detail}
+
+
+
+ {:else if pageData}
+
+ {pageData.subject || 'Page'}
+
+
+
+
+
+
+
Rule
+
{pageData.ruleName || '—'}
+
+
+
Rule type
+
{pageData.ruleType || '—'}
+
+
+
Events
+
{pageData.eventCount}
+
+
+
Last event
+
{pageData.lastEventAt ? formatRelativeTimeAgo(pageData.lastEventAt) : '—'}
+
+
+
+ {#if pageData.url}
+
+
+ View in dashboard
+
+ {/if}
+
+ {#if pageData.body}
+
+
Details
+
{pageData.body}
+ {#if pageData.body.length > 300 || pageData.body.split('\n').length > 6}
+
(bodyExpanded = !bodyExpanded)}
+ >
+ {bodyExpanded ? 'Show less' : 'Show more'}
+
+ {/if}
+
+ {/if}
+
+
+
Escalation chain
+
= 0 ? pageData.escalationLevel : null}
+ />
+ {#if pageData.status === 'open' && pageData.nextEscalationAt === null && pageData.escalationLevel >= 0}
+
+ Escalation exhausted — no further steps will be notified.
+
+ {:else if pageData.status === 'open' && pageData.nextEscalationAt}
+
+ Next escalation at {formatTime(pageData.nextEscalationAt)}
+
+ {/if}
+
+
+
+
Delivery timeline
+ {#if (detail?.notifications ?? []).length === 0}
+
No notifications sent yet.
+ {:else}
+
+ {#each detail?.notifications ?? [] as notification (notification.id)}
+ {@const scheduled = isScheduled(notification)}
+ {@const cancelled = notification.status === 'cancelled'}
+
+
+
+
+
+ L{notification.level + 1}
+ →
+
+ {notification.targetDesc}
+ {#if !notification.targetDesc.includes(`(${notification.methodType})`)}
+ ({notification.methodType})
+ {/if}
+
+ {#if !scheduled}
+ —
+ {notification.status}
+ {/if}
+
+ {#if scheduled && notification.scheduledFor}
+
+ scheduled for {formatClock(notification.scheduledFor)}
+
+ {/if}
+
+ {#if notification.status === 'failed' && notification.errorMsg}
+
{notification.errorMsg}
+ {/if}
+ {#if !scheduled}
+
+ {formatTime(notification.sentAt ?? notification.createdAt)}
+
+ {/if}
+
+
+ {/each}
+
+ {/if}
+ {#if pageData.acknowledgedAt}
+
+
+
+ {#if pageData.acknowledgedVia === 'link'}
+ {#if pageData.acknowledgedBy !== null}
+ Acknowledged by {userName(pageData.acknowledgedBy)} (via link)
+ {:else}
+ Acknowledged via ack link
+ {/if}
+ {:else}
+ Acknowledged by {userName(pageData.acknowledgedBy)}
+ {/if}
+
{formatTime(pageData.acknowledgedAt)}
+
+
+ {/if}
+ {#if pageData.resolvedAt}
+
+
+
+ Resolved by {userName(pageData.resolvedBy)}
+
{formatTime(pageData.resolvedAt)}
+
+
+ {/if}
+
+
+
+ {#if pageData.status !== 'resolved'}
+
+ {#if pageData.status === 'open'}
+ performAction('acknowledge')} disabled={actionLoading}>
+ Acknowledge
+
+ {/if}
+ performAction('resolve')} disabled={actionLoading}>
+ Resolve
+
+
+ {/if}
+ {/if}
+
+
diff --git a/frontend/src/routes/on-call/pages-tab.svelte b/frontend/src/routes/on-call/pages-tab.svelte
new file mode 100644
index 00000000..43a28634
--- /dev/null
+++ b/frontend/src/routes/on-call/pages-tab.svelte
@@ -0,0 +1,258 @@
+
+
+
+
+ {#each filters as filter (filter.value)}
+ setFilter(filter.value)}
+ >
+ {filter.label}
+
+ {/each}
+
+
+
+
+ {#if loading}
+
+
+
+
+
+
+
+
+
+ {:else if error}
+
+
+
+
+
{error}
+
loadPages()}>Retry
+
+
+
+
+ {:else if pages.length === 0}
+
+
+
+ {:else}
+
+
+ Severity
+ Subject
+ Level
+ Age
+ Events
+ Status
+ Actions
+
+
+
+ {#each pages as item (item.id)}
+ openDetail(item)}>
+
+
+
+
+ {item.subject || '—'}
+ {#if item.ruleName}
+ {item.ruleName}
+ {/if}
+
+ {levelLabel(item)}
+ {formatRelativeTimeAgo(item.createdAt)}
+ {item.eventCount}
+
+
+
+
+
+ {#if item.status === 'open'}
+ acknowledge(item, e)}
+ >
+
+
+ {/if}
+ {#if item.status !== 'resolved'}
+ resolve(item, e)}
+ >
+
+
+ {/if}
+
+
+
+ {/each}
+
+ {/if}
+
+
+
+
+
+
+
diff --git a/frontend/src/routes/on-call/policies-tab.svelte b/frontend/src/routes/on-call/policies-tab.svelte
new file mode 100644
index 00000000..045f4389
--- /dev/null
+++ b/frontend/src/routes/on-call/policies-tab.svelte
@@ -0,0 +1,246 @@
+
+
+
+ {#if canManage}
+
+ {/if}
+
+ {#if oncallState.policiesLoading}
+
+ {:else if oncallState.policiesError}
+
+
{oncallState.policiesError}
+
oncallState.loadPolicies(organizationId)}>
+ Retry
+
+
+{:else if oncallState.policies.length === 0}
+
+
No escalation policies yet. Define who gets paged, and in what order.
+ {#if canManage}
+
+
+ Create your first Policy
+
+ {/if}
+
+ {:else}
+
+ {#each oncallState.policies as policy (policy.id)}
+
+
+
+
+ {policy.name}
+
+ {policy.definition?.steps?.length ?? 0} step{(policy.definition?.steps?.length ??
+ 0) === 1
+ ? ''
+ : 's'}
+
+
+ {#if canManage}
+
+
openEdit(policy)}>
+
+
+
{
+ deleteError = '';
+ deletingPolicy = policy;
+ }}
+ >
+
+
+
+ {/if}
+
+
+
+
+
+
+ {/each}
+
+ {/if}
+
+
+ {
+ policyDialogOpen = false;
+ }}
+/>
+
+ {
+ if (!open) {
+ deletingPolicy = null;
+ deleteError = '';
+ }
+ }}
+>
+
+
+ Delete Policy
+
+ Are you sure you want to delete "{deletingPolicy?.name}"? This action cannot be undone.
+
+
+
+
+ Cancel
+
+
+ {deleting ? 'Deleting...' : 'Delete Policy'}
+
+
+
+
diff --git a/frontend/src/routes/on-call/policy-dialog.svelte b/frontend/src/routes/on-call/policy-dialog.svelte
new file mode 100644
index 00000000..fc3e8739
--- /dev/null
+++ b/frontend/src/routes/on-call/policy-dialog.svelte
@@ -0,0 +1,365 @@
+
+
+ (open = isOpen)}>
+
+
+ {isEditing ? 'Edit Policy' : 'New Policy'}
+
+ {isEditing
+ ? 'Update the escalation policy steps and targets'
+ : 'Define who gets paged, and in what order'}
+
+
+
+
+
+
+ Cancel
+
+ {#if isEditing}
+
+ {loading ? 'Updating...' : 'Update Policy'}
+ {:else}
+
+ {loading ? 'Creating...' : 'New Policy'}
+ {/if}
+
+
+
+
diff --git a/frontend/src/routes/on-call/schedule-detail.svelte b/frontend/src/routes/on-call/schedule-detail.svelte
new file mode 100644
index 00000000..56c73318
--- /dev/null
+++ b/frontend/src/routes/on-call/schedule-detail.svelte
@@ -0,0 +1,269 @@
+
+
+{#if loading}
+
+{:else if error}
+ {error}
+{:else if schedule}
+
+
+
+
+ {schedule.name}
+ {#if team}
+ {team.name}
+ {/if}
+ {schedule.timezone}
+
+ {#if schedule.description}
+ {schedule.description}
+ {/if}
+
+ {#if canManage}
+
+
(showLayerEditor = !showLayerEditor)}
+ >
+ Edit Layers
+
+
(editDialogOpen = true)}>
+
+
+
(showDeleteDialog = true)}
+ >
+
+
+
+ {/if}
+
+
+ {#if showLayerEditor && canManage}
+ {
+ showLayerEditor = false;
+ timelineRefreshKey += 1;
+ loadDetail();
+ onChanged();
+ }}
+ onCancel={() => (showLayerEditor = false)}
+ />
+ {/if}
+
+ {#key timelineRefreshKey}
+
+ {/key}
+
+
+
+
Overrides (next 30 days) · {schedule.timezone}
+
(overrideDialogOpen = true)}>
+ Add Override
+
+
+ {#if overrides.length === 0}
+
No upcoming overrides.
+ {:else}
+
+ {#each overrides as override (override.id)}
+
+
+ {userLabel(override.userId)}
+
+ {formatDateTime(override.startAt, {
+ format: 'short',
+ timezone: schedule.timezone
+ })} — {formatDateTime(override.endAt, {
+ format: 'short',
+ timezone: schedule.timezone
+ })}
+
+
+
(overrideToDelete = override)}
+ >
+
+
+
+ {/each}
+
+ {/if}
+
+
+
+
+ {
+ editDialogOpen = false;
+ loadDetail();
+ onChanged();
+ }}
+ />
+
+ {
+ overrideDialogOpen = false;
+ timelineRefreshKey += 1;
+ loadDetail();
+ }}
+ />
+
+
+
+
+ Delete Schedule
+
+ Are you sure you want to delete "{schedule.name}"? This action cannot be undone.
+
+
+
+ Cancel
+
+
+ Delete Schedule
+
+
+
+
+
+ {
+ if (!open) overrideToDelete = null;
+ }}
+ >
+
+
+ Delete Override
+
+ Are you sure you want to delete the override for {overrideToDelete
+ ? userLabel(overrideToDelete.userId)
+ : ''}?
+
+
+
+ Cancel
+
+
+ Delete Override
+
+
+
+
+{/if}
diff --git a/frontend/src/routes/on-call/schedule-dialog.svelte b/frontend/src/routes/on-call/schedule-dialog.svelte
new file mode 100644
index 00000000..3a59dfed
--- /dev/null
+++ b/frontend/src/routes/on-call/schedule-dialog.svelte
@@ -0,0 +1,182 @@
+
+
+ (open = isOpen)}>
+
+
+ {isEditing ? 'Edit Schedule' : 'New Schedule'}
+
+ {isEditing
+ ? 'Update the schedule details'
+ : 'Create an on-call schedule for a team'}
+
+
+
+
+
+
+ Cancel
+
+ {#if isEditing}
+
+ {loading ? 'Updating...' : 'Update Schedule'}
+ {:else}
+
+ {loading ? 'Creating...' : 'New Schedule'}
+ {/if}
+
+
+
+
diff --git a/frontend/src/routes/on-call/schedule-timeline.svelte b/frontend/src/routes/on-call/schedule-timeline.svelte
new file mode 100644
index 00000000..d23ab461
--- /dev/null
+++ b/frontend/src/routes/on-call/schedule-timeline.svelte
@@ -0,0 +1,309 @@
+
+
+
+
+
+ {#each presets as p (p.value)}
+ setPreset(p.value)}
+ >
+ {p.label}
+
+ {/each}
+
+
+ shiftRange(-1)}>
+
+
+ {rangeLabel}
+ shiftRange(1)}>
+
+
+ Today
+
+
+
+ {#if loading && !timeline}
+
+ {:else if error}
+
{error}
+ {:else if timeline}
+
+
+
+ {tz}
+
+
+ {#each ticks as tick (tick.pos)}
+
+ {tick.label}
+
+ {/each}
+
+
+ {#each rows as row (row.id)}
+
+ {row.name}
+
+
+ {#each ticks as tick (tick.pos)}
+
+ {/each}
+ {#each positionShifts(row.shifts) as { shift, left, width }, i (i)}
+
+ {#if width > 6}
+
+ {userName(shift.userId)}
+
+ {#if shift.isOverride && width > 14}
+
+ Override
+
+ {/if}
+ {/if}
+
+ {/each}
+ {#if nowPos !== null}
+
+ {/if}
+
+ {/each}
+
+
+ {#if timeline.layers.length === 0}
+
+ This schedule has no layers yet. Add layers to generate shifts.
+
+ {/if}
+ {/if}
+
diff --git a/frontend/src/routes/on-call/schedules-tab.svelte b/frontend/src/routes/on-call/schedules-tab.svelte
new file mode 100644
index 00000000..8f528cf6
--- /dev/null
+++ b/frontend/src/routes/on-call/schedules-tab.svelte
@@ -0,0 +1,126 @@
+
+
+
+ {#if canManage}
+
+
(scheduleDialogOpen = true)}>
+ New Schedule
+
+
+ {/if}
+
+ {#if oncallState.schedulesLoading}
+
+ {:else if oncallState.schedulesError}
+
+
{oncallState.schedulesError}
+
oncallState.loadSchedules(organizationId)}>
+ Retry
+
+
+{:else if schedules.length === 0}
+
+
No schedules yet. Create one to get started.
+ {#if canManage}
+
(scheduleDialogOpen = true)}>
+
+ Create your first Schedule
+
+ {/if}
+
+ {:else}
+
+
+
+
+ Name
+ Team
+ Timezone
+ Layers
+
+
+
+ {#each schedules as schedule (schedule.id)}
+ selectSchedule(schedule)}
+ >
+ {schedule.name}
+
+ {teamName(schedule.teamId)}
+
+ {schedule.timezone}
+ {schedule.definition?.layers?.length ?? 0}
+
+ {/each}
+
+
+
+ {/if}
+
+ {#if selectedScheduleId !== null}
+ {#key `${selectedScheduleId}-${detailRefreshKey}`}
+
{
+ selectedScheduleId = null;
+ }}
+ onChanged={() => {
+ detailRefreshKey += 1;
+ }}
+ />
+ {/key}
+ {/if}
+
+
+ {
+ scheduleDialogOpen = false;
+ if (scheduleId) selectedScheduleId = scheduleId;
+ }}
+/>
diff --git a/frontend/src/routes/on-call/team-dialog.svelte b/frontend/src/routes/on-call/team-dialog.svelte
new file mode 100644
index 00000000..1ee33bc1
--- /dev/null
+++ b/frontend/src/routes/on-call/team-dialog.svelte
@@ -0,0 +1,275 @@
+
+
+
+
+
+ {isEditing ? 'Edit Team' : 'New Team'}
+
+ {isEditing
+ ? 'Update the team, its members and owned projects'
+ : 'Create a team with members and owned projects'}
+
+
+
+
+
+
+ Cancel
+
+ {#if isEditing}
+
+ {loading ? 'Updating...' : 'Update Team'}
+ {:else}
+
+ {loading ? 'Creating...' : 'New Team'}
+ {/if}
+
+
+
+
diff --git a/frontend/src/routes/on-call/teams-tab.svelte b/frontend/src/routes/on-call/teams-tab.svelte
new file mode 100644
index 00000000..af68ae11
--- /dev/null
+++ b/frontend/src/routes/on-call/teams-tab.svelte
@@ -0,0 +1,206 @@
+
+
+
+ {#if canManage}
+
+ {/if}
+
+ {#if oncallState.teamsLoading}
+
+ {:else if oncallState.teamsError}
+
+
{oncallState.teamsError}
+
oncallState.loadTeams(organizationId)}>
+ Retry
+
+
+{:else if teams.length === 0}
+
+
No teams yet. Create a team to get started.
+ {#if canManage}
+
+
+ Create your first Team
+
+ {/if}
+
+ {:else}
+
+
+
+
+ Name
+ Members
+ Projects
+ Schedules
+ {#if canManage}
+ Actions
+ {/if}
+
+
+
+ {#each teams as team (team.id)}
+
+
+ {team.name}
+ {#if team.description}
+ {team.description}
+ {/if}
+
+
+
+
+ {#each team.members.slice(0, 5) as member (member.userId)}
+
+
+ {initials(member.name, member.email)}
+
+
+ {/each}
+
+
{team.memberCount}
+
+
+
+
+ {#each team.projects as project (project.projectId)}
+ {project.name}
+ {:else}
+ —
+ {/each}
+
+
+ {team.scheduleCount}
+ {#if canManage}
+
+
+
openEditTeam(team)}
+ >
+
+
+
(teamToDelete = team)}
+ >
+
+
+
+
+ {/if}
+
+ {/each}
+
+
+
+ {/if}
+
+
+ {
+ teamDialogOpen = false;
+ oncallState.loadTeams(organizationId);
+ }}
+/>
+
+ {
+ if (!open) teamToDelete = null;
+ }}
+>
+
+
+ Delete Team
+
+ Are you sure you want to delete "{teamToDelete?.name}"? Its schedules will be deleted as
+ well. This action cannot be undone.
+
+
+
+ Cancel
+
+
+ Delete Team
+
+
+
+
diff --git a/website/app/product/on-call/page.tsx b/website/app/product/on-call/page.tsx
new file mode 100644
index 00000000..6f4f2141
--- /dev/null
+++ b/website/app/product/on-call/page.tsx
@@ -0,0 +1,336 @@
+import Link from "next/link";
+import type { Metadata } from "next";
+import { ArrowRight, PhoneCall } from "lucide-react";
+
+import { Chip } from "@/components/chip";
+import { SectionHead } from "@/components/section-head";
+import { FeatureRow } from "@/components/feature-row";
+import { FaqList } from "@/components/faq-list";
+import { FinalCTA } from "@/components/final-cta";
+import { AuroraBackground } from "@/components/aurora-background";
+
+export const metadata: Metadata = {
+ title: "On-Call · Traceway",
+ description:
+ "PagerDuty-style paging built into your observability stack. Rotating schedules, escalation policies, per-responder notification rules, and a no-login acknowledge link. Your alerts already know something is broken, so let them wake the right person.",
+};
+
+export default function OnCallPage() {
+ return (
+
+
+
+
+
+
+ On-Call & Incident Paging
+
+
+ Alerts post to a channel. Pages wake people up.
+
+
+ Traceway already knows your error rate spiked. On-Call turns that
+ into somebody's phone ringing, then keeps escalating until a
+ human acknowledges. Rotating schedules, escalation policies, and
+ per-responder notification rules, in the same tool that holds the
+ stack trace.
+
+
+
+ Read the Docs
+
+
+ Try Traceway Cloud
+
+
+
+
+
+ {/* WHITE BAND: feature sections render on white */}
+
+
+
+ A flapping error is one page
+ >
+ }
+ description="A page is the unit of work, not the message. While it is unresolved, the same condition bumps an event counter instead of opening a new incident, and it never restarts the escalation clock. Sixty firings over half an hour produce one page and exactly the notifications the policy calls for."
+ bullets={[
+ "Re-fires bump the event count and notify nobody again",
+ "The escalation clock is never reset by a duplicate",
+ "Open, acknowledged, and resolved, with the level always visible",
+ "Resolving releases the key so the next occurrence pages fresh",
+ ]}
+ image={{
+ src: "/images/oncall-pages.png",
+ alt: "The Traceway on-call incident queue",
+ }}
+ />
+
+
+
+
+ Rotations that survive real life
+ >
+ }
+ description="Stack layers to describe how your team actually works. A business hours layer over the whole team, a nights and weekends layer over the smaller group who signed up for it, each restricted to its own window. Later layers take precedence, so the schedule row at the bottom is who actually gets paged."
+ bullets={[
+ "Daily, weekly, or every N days, with a handoff time you choose",
+ "Time-of-day and day-of-week restrictions per layer",
+ "Every schedule carries its own timezone, daylight saving included",
+ "The timeline renders the stack, so you can see it before it pages",
+ ]}
+ image={{
+ src: "/images/oncall-layer-editor.png",
+ alt: "Editing on-call schedule layers and restrictions",
+ }}
+ />
+
+
+
+
+ Escalation stops on acknowledge, not on send
+ >
+ }
+ description="Delivering a message proves nothing. A policy climbs level by level until a human takes the incident: primary schedule first, secondary after five minutes, the whole team after ten. Targets resolve to people when the step runs, so a schedule step always reaches whoever is on call right now, including an override that started a minute ago."
+ bullets={[
+ "Target a schedule, a team, a specific person, or a Slack channel",
+ "Per-step delay, and a repeat that loops the whole chain",
+ "Urgency picks which of the responder's rule chains runs",
+ "Each page snapshots its policy, so edits never disturb live incidents",
+ ]}
+ image={{
+ src: "/images/oncall-policy-dialog.png",
+ alt: "A three level escalation policy",
+ }}
+ />
+
+
+
+
+ Nudge first, then get loud
+ >
+ }
+ description="Each responder owns their own chain per urgency. Slack immediately, a push after two minutes, SMS after five. The whole chain is scheduled the moment the page reaches you, and acknowledging cancels every step that has not fired yet. Take the incident in the first thirty seconds and your phone never rings."
+ bullets={[
+ "Email, Slack, Pushover, Telegram, and SMS",
+ "Separate chains for high and low urgency",
+ "Acknowledging cancels the tail of your own chain",
+ "No contact methods configured falls back to your account email",
+ ]}
+ image={{
+ src: "/images/oncall-contact-methods.png",
+ alt: "Contact methods and per-urgency notification rules",
+ }}
+ />
+
+
+
+
+ Acknowledge without logging in
+ >
+ }
+ description="Every delivery to a person carries its own acknowledge link. Tap it from the notification and the escalation stops. No session, no password manager, no SSO round trip at 3am. The link is scoped to acknowledging that one page and nothing else, and it stops working once the page is resolved."
+ bullets={[
+ "One single-purpose token per delivery, stored hashed",
+ "Opening the link is read-only, so email scanners cannot ack for you",
+ "Attributed to the person the delivery was addressed to",
+ "Acknowledging never requires write access on the project",
+ ]}
+ image={{
+ src: "/images/oncall-ack-link.png",
+ alt: "The no-login acknowledge page",
+ }}
+ />
+
+
+
+
+ The rules you already have, pointed at a human
+ >
+ }
+ description="Paging is a channel type. Create an escalation channel, point it at a policy, and attach any existing alert rule to it. The rule fires and opens a page instead of sending a message. The Test button opens a real page and runs the real escalation, which is the only honest way to find out whether your rotation and your phone both work."
+ bullets={[
+ "No separate alerting config to keep in sync",
+ "Any rule can page: error rate, latency, metric threshold, missing data",
+ "Teams own projects, so an issue shows who is on call for it",
+ "Test end to end before you rely on it",
+ ]}
+ image={{
+ src: "/images/oncall-channel-dialog.png",
+ alt: "Creating an escalation channel from an alert rule",
+ }}
+ />
+
+
+
+
+ Never ask “who is on call?” in Slack again
+ >
+ }
+ description="One screen shows the current responder and the next one up, per team and per schedule. The same answer appears on the issue itself, so when you are staring at a stack trace you already know who owns it and who to pull in."
+ bullets={[
+ "Current and next on-call, per schedule",
+ "Shown on the issue page for the project you are looking at",
+ "Teams own projects, so ownership is never ambiguous",
+ ]}
+ image={{
+ src: "/images/oncall-overview.png",
+ alt: "Current and next on-call per team and schedule",
+ }}
+ />
+
+
+
+
+ Every attempt, on the record
+ >
+ }
+ description="When someone says they never got paged, the delivery log settles it. Each page shows its escalation chain with the current level, and every delivery attempt with its destination and outcome. Nothing sends from inside the request that triggered it, so a crash mid-incident cannot lose a page."
+ bullets={[
+ "Durable outbox, with the level advance committed alongside it",
+ "Retries on a 1, 5, 15, 60 minute backoff before failing terminally",
+ "Acknowledging cancels queued deliveries, permanently",
+ "Queue depth and oldest pending delivery on the health endpoint",
+ ]}
+ image={{
+ src: "/images/oncall-page-detail.png",
+ alt: "Page detail with escalation chain and delivery log",
+ }}
+ />
+
+
+
+
+ Stop paging the whole team
+ >
+ }
+ description="Schedules, escalation policies, and per-responder rules, in the same tool that already holds your traces. Included on every plan, and open source if you self-host."
+ primary={{
+ label: "Read the On-Call docs",
+ href: "https://docs.tracewayapp.com/learn/on-call",
+ }}
+ secondary={{
+ label: "Start Free",
+ href: "https://cloud.tracewayapp.com/register",
+ }}
+ />
+
+
+
+
+
+
+
+ Not for the paging loop itself. Traceway covers rotating
+ schedules with stacked layers and restrictions,
+ overrides, multi-level escalation policies with repeat,
+ per-responder notification rules per urgency,
+ deduplicated incidents, and acknowledge links that work
+ without logging in. The difference is that the alert,
+ the stack trace, the trace, and the page all live in one
+ place, so the person you woke up lands on the evidence
+ instead of a link to another tool.
+
+ >
+ ),
+ },
+ {
+ q: "What stops a noisy alert from paging me sixty times?",
+ a: "Deduplication. While a page is unresolved, the same rule and dedup token bump the existing incident rather than opening a new one. The event counter goes up, the escalation clock is untouched, and nobody is notified again. The number of notifications is decided by your escalation policy and your own rule chain, never by how many times the condition fired.",
+ },
+ {
+ q: "How do I handle a vacation or a swapped shift?",
+ a: "Add an override for the dates. It beats every layer for that window and disappears on its own when the dates pass, so there is no rotation to reshuffle and nothing to remember to put back. Any member of the organization can create one, because covering for a teammate should not need an administrator.",
+ },
+ {
+ q: "What happens if nobody acknowledges?",
+ a: (
+ <>
+
+ The policy keeps climbing. Each step waits its
+ configured delay, then notifies the next set of targets.
+ After the last step you can set{" "}
+ Repeat to send the whole chain around again
+ up to five more times. Once the policy is exhausted the
+ page stays open and visible in the queue, but nothing
+ further is sent.
+
+ >
+ ),
+ },
+ {
+ q: "Does SMS work on a self-hosted instance?",
+ a: (
+ <>
+
+ It needs Twilio credentials on your server:{" "}
+ TWILIO_ACCOUNT_SID,{" "}
+ TWILIO_AUTH_TOKEN, and one sender. Without
+ them SMS is not offered at all, rather than silently
+ accepted and dropped. Slack, Pushover, and Telegram need
+ nothing beyond the webhook or token each responder adds
+ to their own contact method. Email is the one that
+ depends on the server, so configure SMTP before you rely
+ on it: with SMTP off, Traceway logs the message instead
+ of sending it.
+
+ >
+ ),
+ },
+ {
+ q: "Can someone with read-only access acknowledge a page?",
+ a: "Yes, deliberately. Acknowledging and resolving require project read access and nothing more. A responder who gets paged at 3am is never blocked from taking the incident by a permissions check, and the acknowledge link in the notification works without a session at all.",
+ },
+ ]}
+ />
+
+
+
+
+ );
+}
diff --git a/website/components/site-footer.tsx b/website/components/site-footer.tsx
index 4391720a..15d37f97 100644
--- a/website/components/site-footer.tsx
+++ b/website/components/site-footer.tsx
@@ -27,6 +27,7 @@ const COLUMNS: Column[] = [
{ label: "MCP Server", href: "/product/mcp" },
{ label: "AI Tracing", href: "/product/ai-tracing" },
{ label: "Dashboards as Code", href: "/product/dashboards" },
+ { label: "On-Call", href: "/product/on-call" },
{ label: "Performance", href: "/product/performance" },
{ label: "Flutter Session Replay", href: "/product/flutter-session-replay" },
{ label: "Symbolicator", href: "/product/symbolication" },
diff --git a/website/components/site-header.tsx b/website/components/site-header.tsx
index 277115aa..0d994e29 100644
--- a/website/components/site-header.tsx
+++ b/website/components/site-header.tsx
@@ -18,6 +18,7 @@ import {
Braces,
Plug,
LayoutDashboard,
+ PhoneCall,
} from "lucide-react";
import { MobileNav } from "@/components/mobile-nav";
import { DiscordIcon } from "@/components/discord-icon";
@@ -90,6 +91,12 @@ const SPECIALIZED: NavItem[] = [
href: "/product/dashboards",
icon: LayoutDashboard,
},
+ {
+ title: "On-Call",
+ description: "Rotating schedules, escalation policies, real paging.",
+ href: "/product/on-call",
+ icon: PhoneCall,
+ },
{
title: "Performance",
description: "P50/P95/P99 percentiles, waterfall traces.",
diff --git a/website/public/images/oncall-ack-link.png b/website/public/images/oncall-ack-link.png
new file mode 100644
index 00000000..adfc1323
Binary files /dev/null and b/website/public/images/oncall-ack-link.png differ
diff --git a/website/public/images/oncall-channel-dialog.png b/website/public/images/oncall-channel-dialog.png
new file mode 100644
index 00000000..b174e71b
Binary files /dev/null and b/website/public/images/oncall-channel-dialog.png differ
diff --git a/website/public/images/oncall-contact-methods.png b/website/public/images/oncall-contact-methods.png
new file mode 100644
index 00000000..bfc0e9d0
Binary files /dev/null and b/website/public/images/oncall-contact-methods.png differ
diff --git a/website/public/images/oncall-layer-editor.png b/website/public/images/oncall-layer-editor.png
new file mode 100644
index 00000000..08c94518
Binary files /dev/null and b/website/public/images/oncall-layer-editor.png differ
diff --git a/website/public/images/oncall-overview.png b/website/public/images/oncall-overview.png
new file mode 100644
index 00000000..a1416d33
Binary files /dev/null and b/website/public/images/oncall-overview.png differ
diff --git a/website/public/images/oncall-page-detail.png b/website/public/images/oncall-page-detail.png
new file mode 100644
index 00000000..d5d508ca
Binary files /dev/null and b/website/public/images/oncall-page-detail.png differ
diff --git a/website/public/images/oncall-pages.png b/website/public/images/oncall-pages.png
new file mode 100644
index 00000000..26d3cd77
Binary files /dev/null and b/website/public/images/oncall-pages.png differ
diff --git a/website/public/images/oncall-policy-dialog.png b/website/public/images/oncall-policy-dialog.png
new file mode 100644
index 00000000..55474f06
Binary files /dev/null and b/website/public/images/oncall-policy-dialog.png differ