Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions internal/connector/admission/admission_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1138,7 +1138,8 @@ func TestTheGateReadsTheLiveServedSetToo(t *testing.T) {
// and post the public holding reply — telling the person on the card that
// their project is not served, when the project is there and the file is
// what is broken. no_route also has no timed retry, so repairing the file
// would not reconsider those records.
// would not reconsider those records — where config_unreadable is retried on
// a timer with no window at all, so it does.
func TestAnUnreadableConfigIsHeldAsOneRatherThanAnsweredAsUnserved(t *testing.T) {
broken := errors.New("connect.json cannot be read")
fail := true
Expand All @@ -1157,14 +1158,17 @@ func TestAnUnreadableConfigIsHeldAsOneRatherThanAnsweredAsUnserved(t *testing.T)
assert.Equal(t, ReasonConfigUnreadable, v.Reason, "not no_route: nothing read the file, so nothing can say the project is unserved")
assert.False(t, v.Served)

// It waits for a person, as every blocked record does. An automatic
// sweep was built for this and taken back out: offering due blocked
// rows is a scheduler with its own claiming, and it re-decided five
// other blocked reasons besides this one. Carded, so that nothing here
// promises a timer that does not run.
// It comes round on its own, every ten minutes, for as long as the file
// is unreadable. The window the other blocked reasons stop at is right
// for a server that has failed for a day; it is wrong for a local file,
// where an operator away for a week is ordinary and giving up would
// strand the work silently.
blockedAt := time.Date(2026, 9, 18, 12, 0, 0, 0, time.UTC)
_, retried := NextBlockedRetry(ReasonConfigUnreadable, blockedAt, blockedAt, time.Time{})
assert.False(t, retried, "no schedule claims this record, and nothing would act on one if it did")
next, retried := NextBlockedRetry(ReasonConfigUnreadable, blockedAt, blockedAt, time.Time{})
require.True(t, retried, "the sweep re-offers this one without anybody asking")
assert.Equal(t, blockedAt.Add(BlockedRetryInterval), next)
_, retried = NextBlockedRetry(ReasonConfigUnreadable, blockedAt, blockedAt.Add(8*24*time.Hour), time.Time{})
assert.True(t, retried, "a week of it is an operator on holiday, not a verdict")

// And once the file is readable the record decides normally.
fail = false
Expand Down
58 changes: 48 additions & 10 deletions internal/connector/admission/commit.go
Original file line number Diff line number Diff line change
Expand Up @@ -135,33 +135,71 @@ func (k *keyedMutex) lock(ctx context.Context, key string) (func(), error) {
// unverified assignment delta is retried every ten minutes for a day after it
// was first blocked, and on redispatch at any time after that. A throttled
// record (throttled) is on the same schedule, never before the server's
// deadline. bucket_mismatch
// and unroutable are not timed: the pointer's bucket and type never change, so
// only a person's redispatch re-runs them. A blocked record is never
// discarded for having failed: the checkpoint may already be past the event,
// and a tombstone would turn an outage into a permanent loss.
// deadline. A record blocked because connect.json could not be read
// (config_unreadable) is on the same interval with no window at all: that is
// a local file a person repairs, and an operator away for a week is
// ordinary, where a day of a failing server means the server is not coming
// back on its own. bucket_mismatch and unroutable are not timed: the
// pointer's bucket and type never change, so only a person's redispatch
// re-runs them. no_route is not timed either — it waits for the operator to
// serve the project, which is a decision, not a delay. A blocked record is
// never discarded for having failed: the checkpoint may already be past the
// event, and a tombstone would turn an outage into a permanent loss.
//
// The intake sweep is what runs this schedule (internal/connector,
// Intake.sweepBlockedRetries): it asks the ledger for the records this
// function calls due and offers them back to admission.
const (
BlockedRetryInterval = 10 * time.Minute
BlockedRetryWindow = 24 * time.Hour
)

// timedBlockedReasons is every blocked reason the schedule re-runs, and for
// each whether its retries stop at BlockedRetryWindow.
//
// NextBlockedRetry is its only reader, and that is the whole of how a reason
// reaches the sweep: the answer is computed once, when the verdict is
// written, and stored on the row (events.next_retry_at). The ledger's queries
// never see a reason — they compare that stored moment. A second filter by
// reason down there could only ever repeat this one, and would be a copy of
// the schedule that could fall out of step with it (Copilot on #770).
var timedBlockedReasons = map[Reason]bool{
ReasonReadFailed: true,
ReasonReadUnresolved: true,
ReasonDeltaUnverified: true,
ReasonTrustUnverified: true,
ReasonThrottled: true,
ReasonConfigUnreadable: false,
Comment thread
jorgemanrubia marked this conversation as resolved.
}

// NextBlockedRetry returns when a blocked record should next be re-run, and
// false when it waits for something other than time: the operator serving the
// project (no_route), or a person's redispatch once the window has passed.
// notBefore is a throttled record's Verdict.RetryAt; no retry is scheduled
// before it, and a deadline past the window hands the record to redispatch
// rather than asking early.
func NextBlockedRetry(reason Reason, blockedAt, lastAttempt, notBefore time.Time) (time.Time, bool) {
switch reason {
case ReasonReadFailed, ReasonReadUnresolved, ReasonDeltaUnverified, ReasonTrustUnverified, ReasonThrottled:
default:
//
// since is when the record entered the reason it is blocked on NOW, which is
// not when it entered blocked. The reasons carry different windows, so a
// record that spent a week as the unbounded config_unreadable and then blocks
// read_failed must get read_failed's twenty-four hours from the moment
// read_failed began; measured from the older stamp it would get none, which
// is the retry stranding the work it is there to recover (Copilot on #770).
// The ledger keeps that stamp per reason (events.retry_since).
//
// The returned time may be in the past — a connector that was not running
// when a retry came due is late, not excused — so a caller asks whether next
// is at or before now, never whether it is in the future.
func NextBlockedRetry(reason Reason, since, lastAttempt, notBefore time.Time) (time.Time, bool) {
bounded, timed := timedBlockedReasons[reason]
if !timed {
return time.Time{}, false
}
next := lastAttempt.Add(BlockedRetryInterval)
if notBefore.After(next) {
next = notBefore
}
if next.After(blockedAt.Add(BlockedRetryWindow)) {
if bounded && next.After(since.Add(BlockedRetryWindow)) {
return time.Time{}, false
}
return next, true
Expand Down
38 changes: 38 additions & 0 deletions internal/connector/admission/commit_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"io"
"slices"
"strings"
"sync"
"sync/atomic"
Expand Down Expand Up @@ -246,6 +247,43 @@ func TestNextBlockedRetry(t *testing.T) {

_, ok = NextBlockedRetry(ReasonNoRoute, blockedAt, blockedAt, time.Time{})
assert.False(t, ok, "no_route waits for connect.json, not for time")

// config_unreadable is the one reason with no window. A day of a failing
// server means the server is not coming back on its own; a broken
// connect.json is a local file, and an operator away for a week is
// ordinary.
next, ok = NextBlockedRetry(ReasonConfigUnreadable, blockedAt, blockedAt, time.Time{})
require.True(t, ok)
assert.Equal(t, blockedAt.Add(10*time.Minute), next)
next, ok = NextBlockedRetry(ReasonConfigUnreadable, blockedAt, blockedAt.Add(30*24*time.Hour), time.Time{})
require.True(t, ok, "a month later it is still asking")
assert.Equal(t, blockedAt.Add(30*24*time.Hour+10*time.Minute), next)
}

// Which reasons automatic re-decision is turned on for, stated as a set
// rather than read off the implementation: this is the feature, and a reason
// joining or leaving it is a decision somebody makes, not a diff nobody
// notices.
func TestTheScheduleRunsExactlySixReasons(t *testing.T) {
every := []Reason{
ReasonInvalidPointer, ReasonNotInMatrix, ReasonAgentAuthored, ReasonDelegated,
ReasonOutOfScope, ReasonUntrustedPerformer, ReasonAssignmentNotOperator,
ReasonUntrustedAuthor, ReasonStale, ReasonNotAddressed,
ReasonReadFailed, ReasonReadUnresolved, ReasonThrottled, ReasonTrustUnverified,
ReasonDeltaUnverified, ReasonBucketMismatch, ReasonUnroutable, ReasonNoRoute,
ReasonConfigUnreadable,
}
var scheduled []Reason
for _, reason := range every {
if _, ok := NextBlockedRetry(reason, testNow, testNow, time.Time{}); ok {
scheduled = append(scheduled, reason)
}
}
slices.Sort(scheduled)
assert.Equal(t, []Reason{
ReasonConfigUnreadable, ReasonDeltaUnverified, ReasonReadFailed,
ReasonReadUnresolved, ReasonThrottled, ReasonTrustUnverified,
}, scheduled, "the six reasons automatic re-decision is turned on for")
}

type sliceSource struct {
Expand Down
11 changes: 6 additions & 5 deletions internal/connector/admission/matrix.go
Original file line number Diff line number Diff line change
Expand Up @@ -152,10 +152,11 @@ const (
// would be a false thing to say — and to post a holding reply about —
// when the truth is that nothing could read the file.
//
// Like every other blocked reason, it waits for a person: repairing the
// file does not by itself decide these records, and `basecamp connect
// redispatch <id>` is what runs them. Nothing in the connector re-offers
// a blocked record on a timer — NextBlockedRetry describes a schedule
// no production code asks for — so this comment does not promise one.
// It is the one blocked reason with no window: the intake sweep re-offers
// it every BlockedRetryInterval for as long as it stands (NextBlockedRetry,
// internal/connector Intake.sweepBlockedRetries). A day is the right bound
// for a failing server, which is not coming back on its own after one; a
// broken connect.json is a local file, and an operator away for a week is
// ordinary. `basecamp connect redispatch <id>` still runs one at once.
ReasonConfigUnreadable Reason = "config_unreadable"
)
6 changes: 3 additions & 3 deletions internal/connector/admission/verdict.go
Original file line number Diff line number Diff line change
Expand Up @@ -263,9 +263,9 @@ func (a *Admitter) Decide(ctx context.Context, ev Event) (out Verdict, err error
// could read which projects are served. Held rather than
// discarded: a discard is the one outcome repairing the file
// cannot reverse, and a blocked record can still be run. It
// waits for a person — `basecamp connect redispatch <id>` once
// the file is back — as every blocked record does. Nothing
// re-offers one on a timer.
// comes round on the blocked schedule until the file is back,
// with no window (NextBlockedRetry), and a person may run
// `basecamp connect redispatch <id>` sooner.
return v.end(StateBlocked, ReasonConfigUnreadable), nil
}
// Every other gate discard turns on the trust set, the matrix or the
Expand Down
Loading
Loading